S3 contents library is not working. Unable to see S3 default directory on notebook. I have S3 bucket in aws and I have already created service account and role which grants permission to the S3 bucket

JupyterHub Zero to JupyterHub on Kubernetes
,
1

hub:
baseUrl: /notebooks
nodeSelector:
role: general
image:
name: public.ecr.aws/eodh/k2jh/jupyter
tag: 0.1.0
extraConfig:
s3Config: |
from s3contents import S3ContentsManager
from traitlets.config import get_config
import boto3
import logging
import sys

  # Enhanced logging configuration
  logging.basicConfig(
      level=logging.INFO,  # Changed from DEBUG to INFO for production
      format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
      handlers=[
          logging.StreamHandler(sys.stdout),
          logging.FileHandler('/tmp/jupyter_s3.log')
      ]
  )
  logger = logging.getLogger('s3contents')
  logger.setLevel(logging.INFO)  # Changed from DEBUG to INFO

  c = get_config()
  session = boto3.Session()
  credentials = session.get_credentials()

  # Basic S3 Configuration
  c.ServerApp.contents_manager_class = S3ContentsManager
  c.S3ContentsManager.bucket = "workspaces-jupyter-test-bucket"
  c.S3ContentsManager.prefix = "workspaces"

  # Set all directory configurations to ensure S3 is used
  c.ServerApp.root_dir = '/workspaces'
  c.NotebookApp.notebook_dir = '/workspaces'
  c.ServerApp.preferred_dir = '/workspaces'
  
  # Safely configure credentials
  c.S3ContentsManager.access_key_id = credentials.access_key
  c.S3ContentsManager.secret_access_key = credentials.secret_key
  c.S3ContentsManager.session_token = credentials.token

  # S3 settings
  c.S3ContentsManager.region_name = "eu-west-2"
  c.S3ContentsManager.endpoint_url = "https://s3.eu-west-2.amazonaws.com"
  c.S3ContentsManager.sse = "AES256"
  c.S3ContentsManager.signature_version = "s3v4"

  # Ensure working directory is set
  c.ServerApp.working_dir = '/workspaces'
  c.ServerApp.default_url = '/lab/tree/workspaces'  # Point directly to workspaces
  logger.info(f"S3 Configuration initialized for bucket: {c.S3ContentsManager.bucket}")

proxy:
service:
type: ClusterIP
chp:
nodeSelector:
role: general
singleuser:
image:
name: “public.ecr.aws/eodh/jupyter/notebooks”
tag: “0.1.0”
storage:
type: dynamic
homeMountPath: /home/{username}
dynamic:
storageClass: block-storage
pvcNameTemplate: claim-{username}–{servername}
serviceAccountName: jupyter-user
profileList:
- display_name: “Data Science (Python)”
description: “Standard Python DS environment.”
default: true
- display_name: “Jupyter/Base-notebook”
description: “Base Notebook environment.”
kubespawner_override:
image: “Quay”
nodeSelector:
role: processing
scheduling:
userScheduler:
nodeSelector:
role: general
prePuller:
hook:
nodeSelector:
role: processing
debug:
enabled: true

1 Like
2

hub.extraConfig only modifies the JupyterHub configuration. Your ContentsManager configuration needs to go into your singleuser server container, for example you could build it into your singleuser image, or you could mount it into the container

If you go down this route make sure you use singleuser.extraFiles since the example is generic.

2 Likes
3

Perfect! Thanks it worked for me

singleuser:
image:
name: “my modified image with s3contents pre-installed”
tag: “0.1.0”
extraFiles:
jupyter_server_config.py:
mountPath: /etc/jupyter/jupyter_server_config.py
stringData: |
from s3contents import S3ContentsManager
from traitlets.config import get_config
import os
import boto3
import logging
import sys
import stat

    c = get_config()

    WORKSPACE_DIR = "/home/jovyan/workspaces"

    # ✅ Ensure directory exists
    if not os.path.exists(WORKSPACE_DIR):
      os.makedirs(WORKSPACE_DIR, exist_ok=True)

    # ✅ Ensure permissions allow writing
    os.chmod(WORKSPACE_DIR, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)

    # Set up logging
    logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        stream=sys.stdout
    )
    logger = logging.getLogger('s3contents')
    logger.info("Initializing S3ContentsManager for singleuser server.")

    # Create a boto3 session (credentials are provided by the IAM role)
    session = boto3.Session()
    credentials = session.get_credentials()

    # Configure S3ContentsManager as the default contents manager
    c.ServerApp.contents_manager_class = S3ContentsManager

    # Basic S3 settings:
    c.S3ContentsManager.bucket = "workspaces-jupyter-test-bucket"
    c.S3ContentsManager.prefix = "workspaces"

    # Ensure directory exists and set working directory:
    os.makedirs(WORKSPACE_DIR, exist_ok=True)
    c.ServerApp.root_dir = WORKSPACE_DIR
    c.ServerApp.default_url = f"/lab/tree{WORKSPACE_DIR}"

    # If credentials are available, set them (they’ll be None when using IAM roles)
    if credentials:
        c.S3ContentsManager.access_key_id = credentials.access_key
        c.S3ContentsManager.secret_access_key = credentials.secret_key
        c.S3ContentsManager.session_token = credentials.token

    # Additional S3 parameters:
    c.S3ContentsManager.region_name = "eu-west-2"
    c.S3ContentsManager.endpoint_url = "https://s3.eu-west-2.amazonaws.com"
    c.S3ContentsManager.sse = "AES256"
    c.S3ContentsManager.signature_version = "s3v4"

    logger.info("S3 configuration loaded successfully.")
1 Like