Z2JH Hot to change NB_UID/NB_GID using authenticator_class?

hiroyuki-sato · October 6, 2023, 12:21pm

Hello members.

1. My Goal.

When a user logs in to the Z2JH, I want to change the user’s UID/GID and Home.
All of the user’s data is stored in the LDAP server.
I found this page. And I’m trying to this setting.

Store notebook data in the specific directory (User’s home) in the NFS directory.
Use the user’s UID/GID

2. Problem.

When the user bob logs in to the Z2JH, the server doesn’t respond in 30 seconds. and failed to launch the cluster.

2023-10-06T11:44:26.947238Z [Normal] Successfully assigned ns-z2jh/jupyter-bob to k8node2
2023-10-06T11:44:27Z [Normal] Container image "jupyterhub/k8s-network-tools:3.0.3" already present on machine
2023-10-06T11:44:27Z [Normal] Created container block-cloud-metadata
2023-10-06T11:44:27Z [Normal] Started container block-cloud-metadata
2023-10-06T11:44:27Z [Normal] Container image "myregistory/jhtest:latest" already present on machine
2023-10-06T11:44:27Z [Normal] Created container notebook
2023-10-06T11:44:27Z [Normal] Started container notebook
Spawn failed: Server at http://10.244.4.110:8888/user/bob/ didn't respond in 30 seconds

and I found the message Running as root is not recommended. Use --allow-root to bypass in the kubectl logs -f -n ns-z2jh jupyter-bob

3. Questions.

IIUC, the cluster needs to run root privileges. Do I need to set singleuser.uid set to 0?
Do I need to use extraEnv: GRANT_SUDO: "yes" and --allow-root options, If so, How do set it?
Is it possible to output the log in more detail? (I didn’t know the reason in the kubectl logs -f -n ns-z2jh jupyter-bob output)

4. Setting

hub:
  db:
    pvc:
      storageClassName: nfs-client
  config:
    Authenticator:
      enable_auth_state: true
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    # https://github.com/manics/zero-to-jupyterhub-k8s-examples/blob/5212ad2440e9ccf3831b5957b80775e58581b232/ldap-singleuser/jupyterhub.yml
    LDAPAuthenticator:
      server_address: ldap.server
      lookup_dn: True
      bind_dn_template:
        - uid={username},ou=users,dc=example,dc=com
      user_search_base: "dc=example,dc=com"
      user_attribute: uid
      lookup_dn_user_dn_attribute: uid
      escape_userdn: True
      use_ssl: true
      auth_state_attributes: ["uid", "uidNumber", "gidNumber", "homeDirectory"]
      use_lookup_dn_username: False
  extraConfig:
    SpawnerCustomConfig: |
      from ldapauthenticator import LDAPAuthenticator
      from hashlib import md5

      class LDAPAuthenticatorInfo(LDAPAuthenticator):
          async def pre_spawn_start(self, user, spawner):
              auth_state = await user.get_auth_state()

              if not auth_state:
                  return

              # https://github.com/jupyter/docker-stacks/blob/fde8f1eec3e70edf67560b5430545a22382334bf/base-notebook/start.sh#L41-L102
              spawner.environment["NB_UID"]  = str(auth_state["uidNumber"][0])
              spawner.environment["NB_USER"] = auth_state["uid"][0]
              spawner.environment["NB_GID"]  = str(auth_state["gidNumber"][0])


      c.JupyterHub.authenticator_class = LDAPAuthenticatorInfo

    CustomHubConfig: |
      c.JupyterHub.cleanup_servers = True


debug:
  enabled: true
singleuser:
  uid: 0
  #extraEnv:
  #  GRANT_SUDO: "yes"
  memory:
    limit: 2G
    guarantee: 2G
  storage:
    capacity: 100Mi
    #homeMountPath: /home/jovyan
    dynamic:
      pvcNameTemplate: claim-{username}{servername}
      storageAccessModes:
      - ReadWriteMany
      storageClass: nfs-client
      volumeNameTemplate: volume-{username}{servername}
    type: dynamic
  image:
    name: 'myrepository/jhtest'
    tag: 'latest'

5. log

% kubectl logs -f -n ns-z2jh jupyter-bob
Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[I 2023-10-06 11:49:28.543 ServerApp] Package jupyterhub took 0.0000s to import
[I 2023-10-06 11:49:28.549 ServerApp] Package jupyter_lsp took 0.0057s to import
[W 2023-10-06 11:49:28.549 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-10-06 11:49:28.552 ServerApp] Package jupyter_server_terminals took 0.0035s to import
[I 2023-10-06 11:49:28.604 ServerApp] Package jupyterlab took 0.0514s to import
[I 2023-10-06 11:49:28.631 ServerApp] Package nbclassic took 0.0009s to import
[W 2023-10-06 11:49:28.632 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-10-06 11:49:28.632 ServerApp] Package nbgitpuller took 0.0005s to import
[I 2023-10-06 11:49:28.633 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-10-06 11:49:28.633 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-10-06 11:49:28.633 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2023-10-06 11:49:28.635 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2023-10-06 11:49:28.635 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
[I 2023-10-06 11:49:28.635 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2023-10-06 11:49:28.637 ServerApp] jupyterhub | extension was successfully linked.
[W 2023-10-06 11:49:28.638 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-10-06 11:49:28.640 ServerApp] jupyterlab | extension was successfully linked.
[W 2023-10-06 11:49:28.641 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-10-06 11:49:28.642 ServerApp] nbclassic | extension was successfully linked.
[I 2023-10-06 11:49:28.643 ServerApp] nbgitpuller | extension was successfully linked.
[I 2023-10-06 11:49:28.816 ServerApp] notebook_shim | extension was successfully linked.
[I 2023-10-06 11:49:28.827 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-10-06 11:49:28.829 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2023-10-06 11:49:28.829 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2023-10-06 11:49:28.840 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2023-10-06 11:49:28.840 ServerApp] jupyterhub | extension was successfully loaded.
[I 2023-10-06 11:49:28.841 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.11/site-packages/jupyterlab
[I 2023-10-06 11:49:28.841 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2023-10-06 11:49:28.841 LabApp] Extension Manager is 'pypi'.
[I 2023-10-06 11:49:28.843 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-10-06 11:49:28.844 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2023-10-06 11:49:28.846 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-10-06 11:49:28.851 ServerApp] nbgitpuller | extension was successfully loaded.
[C 2023-10-06 11:49:28.852 ServerApp] Running as root is not recommended. Use --allow-root to bypass.

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

7. Environment

Z2JH: 3.0.3
K8S: v1.27.4
OS: Ubuntu 22.04

Best regards.

manics · October 6, 2023, 12:50pm

It looks like you’re using a custom singleuser image. Does it work if you use one of the official Jupyter images from
https://jupyter-docker-stacks.readthedocs.io/

hiroyuki-sato · October 6, 2023, 1:08pm

Hello, @manics Thank you for your comment.

I commented out this part.

#  image:
#    name: 'myregistory/jhtest'
#    tag: 'latest'

Server requested
2023-10-06T13:03:06.283978Z [Normal] Successfully assigned ns-z2jh/jupyter-bob to k8node3
2023-10-06T13:03:06Z [Normal] Container image "jupyterhub/k8s-network-tools:3.0.3" already present on machine
2023-10-06T13:03:06Z [Normal] Created container block-cloud-metadata
2023-10-06T13:03:06Z [Normal] Started container block-cloud-metadata
2023-10-06T13:03:07Z [Normal] Container image "jupyterhub/k8s-singleuser-sample:3.0.3" already present on machine
2023-10-06T13:03:07Z [Normal] Created container notebook
2023-10-06T13:03:07Z [Normal] Started container notebook
Spawn failed: Server at http://10.244.5.153:8888/user/bob/ didn't respond in 30 seconds

I found the below messages in the kubectl logs -f -n ns-z2jh hub-57b6977448-zdpfj outputs

[D 2023-10-06 13:03:42.695 JupyterHub user:1005] Finished stopping bob
[E 2023-10-06 13:03:42.702 JupyterHub gen:630] Exception in Future <Task finished name='Task-33' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://10.244.5.153:8888/user/bob/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 914, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 958, in _wait_up
        raise e
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 928, in _wait_up
        resp = await server.wait_up(
               ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
        re = await exponential_backoff(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    TimeoutError: Server at http://10.244.5.153:8888/user/bob/ didn't respond in 30 seconds

manics · October 6, 2023, 5:01pm

This is a minimal image mostly to allow a quick test without downloading a multi-GB image. The official Jupyter images contain a lot of extra options, including to handle alternative users and user ids:
https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html

hiroyuki-sato · October 8, 2023, 2:51pm

Hello, @manics. Thank you for your advice.

I changed the image to jupyter/scipy-notebook like the below.
But, It still not working.

  image:
    name: 'jupyter/scipy-notebook'
    tag: 'latest'

Could you tell me how to set NB_XXX parameters and -w "/home/<username>"?
Does those parameters set using spawner.environment in the hub.extraConfig configuratin?
And It it possible to output the current environment in the container execution? (execute env command for example)

Best regards.

1. The part of `authenticator_class`.

hub:
  extraConfig:
    SpawnerCustomConfig: |
      from ldapauthenticator import LDAPAuthenticator
      from hashlib import md5

      class LDAPAuthenticatorInfo(LDAPAuthenticator):
          async def pre_spawn_start(self, user, spawner):
              auth_state = await user.get_auth_state()

              if not auth_state:
                  return

              # https://github.com/jupyter/docker-stacks/blob/fde8f1eec3e70edf67560b5430545a22382334bf/base-notebook/start.sh#L41-L102
              spawner.environment["NB_UID"]  = str(auth_state["uidNumber"][0])
              spawner.environment["NB_USER"] = auth_state["uid"][0]
              spawner.environment["NB_GID"]  = str(auth_state["gidNumber"][0])
              spawner.environment["CHOWN_HOME"]  = "yes"
              spawner.environment["GRANT_SUDO"]  = "yes"


      c.JupyterHub.authenticator_class = LDAPAuthenticatorInfo

    CustomHubConfig: |
      c.JupyterHub.cleanup_servers = True

2. log file

kubectl logs -f -n ns-z2jh jupyter-bob
Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
[I 2023-10-08 14:21:48.416 ServerApp] Package jupyterhub took 0.0000s to import
[I 2023-10-08 14:21:48.422 ServerApp] Package jupyter_lsp took 0.0055s to import
[W 2023-10-08 14:21:48.422 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-10-08 14:21:48.422 ServerApp] Package jupyter_server_mathjax took 0.0005s to import
[I 2023-10-08 14:21:48.425 ServerApp] Package jupyter_server_terminals took 0.0025s to import
[I 2023-10-08 14:21:48.474 ServerApp] Package jupyterlab took 0.0488s to import
[I 2023-10-08 14:21:48.869 ServerApp] Package jupyterlab_git took 0.0082s to import
[I 2023-10-08 14:21:48.870 ServerApp] Package nbclassic took 0.0011s to import
[W 2023-10-08 14:21:48.874 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-10-08 14:21:48.874 ServerApp] Package nbdime took 0.0000s to import
[I 2023-10-08 14:21:48.875 ServerApp] Package notebook took 0.0000s to import
[I 2023-10-08 14:21:48.877 ServerApp] Package notebook_shim took 0.0000s to import
[W 2023-10-08 14:21:48.877 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-10-08 14:21:48.878 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2023-10-08 14:21:48.881 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2023-10-08 14:21:48.884 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2023-10-08 14:21:48.884 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
[I 2023-10-08 14:21:48.884 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2023-10-08 14:21:48.887 ServerApp] jupyterhub | extension was successfully linked.
[W 2023-10-08 14:21:48.888 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-10-08 14:21:48.891 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-10-08 14:21:48.891 ServerApp] jupyterlab_git | extension was successfully linked.
[W 2023-10-08 14:21:48.892 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-10-08 14:21:48.893 ServerApp] nbclassic | extension was successfully linked.
[I 2023-10-08 14:21:48.893 ServerApp] nbdime | extension was successfully linked.
[W 2023-10-08 14:21:48.894 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2023-10-08 14:21:48.896 ServerApp] notebook | extension was successfully linked.
[I 2023-10-08 14:21:49.010 ServerApp] notebook_shim | extension was successfully linked.
[I 2023-10-08 14:21:49.019 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-10-08 14:21:49.021 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2023-10-08 14:21:49.021 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2023-10-08 14:21:49.021 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2023-10-08 14:21:49.036 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2023-10-08 14:21:49.036 ServerApp] jupyterhub | extension was successfully loaded.
[I 2023-10-08 14:21:49.037 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2023-10-08 14:21:49.038 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-10-08 14:21:49.038 LabApp] Extension Manager is 'pypi'.
[I 2023-10-08 14:21:49.043 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-10-08 14:21:49.046 ServerApp] jupyterlab_git | extension was successfully loaded.
[I 2023-10-08 14:21:49.048 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-10-08 14:21:49.111 ServerApp] nbdime | extension was successfully loaded.
[I 2023-10-08 14:21:49.114 ServerApp] notebook | extension was successfully loaded.
[C 2023-10-08 14:21:49.119 ServerApp] Running as root is not recommended. Use --allow-root to bypass.

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2023-10-08 14:21:49.480 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

Server requested
2023-10-08T14:24:07.221969Z [Normal] Successfully assigned ns-z2jh/jupyter-bob to k8node3
2023-10-08T14:24:07Z [Normal] Container image "jupyterhub/k8s-network-tools:3.0.3" already present on machine
2023-10-08T14:24:07Z [Normal] Created container block-cloud-metadata
2023-10-08T14:24:07Z [Normal] Started container block-cloud-metadata
2023-10-08T14:24:08Z [Normal] Container image "jupyter/scipy-notebook:latest" already present on machine
2023-10-08T14:24:08Z [Normal] Created container notebook
2023-10-08T14:24:08Z [Normal] Started container notebook
Spawn failed: Server at http://10.244.5.163:8888/user/bob/ didn't respond in 30 seconds

3. complete configuration.

hub:
  db:
    pvc:
      storageClassName: nfs-client
  config:
    Authenticator:
      enable_auth_state: true
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    # https://github.com/manics/zero-to-jupyterhub-k8s-examples/blob/5212ad2440e9ccf3831b5957b80775e58581b232/ldap-singleuser/jupyterhub.yml
    LDAPAuthenticator:
      server_address: nfs.example.com
      lookup_dn: True
      bind_dn_template:
        - uid={username},ou=users,dc=example,dc=com
      user_search_base: "dc=example,dc=com"
      user_attribute: uid
      lookup_dn_user_dn_attribute: uid
      escape_userdn: True
      use_ssl: true
        #server_port: 689
      auth_state_attributes: ["uid", "uidNumber", "gidNumber", "homeDirectory"]
      use_lookup_dn_username: False
  extraConfig:
    SpawnerCustomConfig: |
      from ldapauthenticator import LDAPAuthenticator
      from hashlib import md5

      class LDAPAuthenticatorInfo(LDAPAuthenticator):
          async def pre_spawn_start(self, user, spawner):
              auth_state = await user.get_auth_state()

              if not auth_state:
                  return

              # https://github.com/jupyter/docker-stacks/blob/fde8f1eec3e70edf67560b5430545a22382334bf/base-notebook/start.sh#L41-L102
              spawner.environment["NB_UID"]  = str(auth_state["uidNumber"][0])
              spawner.environment["NB_USER"] = auth_state["uid"][0]
              spawner.environment["NB_GID"]  = str(auth_state["gidNumber"][0])
              spawner.environment["CHOWN_HOME"]  = "yes"
              spawner.environment["GRANT_SUDO"]  = "yes"


      c.JupyterHub.authenticator_class = LDAPAuthenticatorInfo

    CustomHubConfig: |
      c.JupyterHub.cleanup_servers = True


debug:
  enabled: true
singleuser:
  uid: 0
  memory:
    limit: 2G
    guarantee: 2G
  storage:
    capacity: 100Mi
    dynamic:
      pvcNameTemplate: claim-{username}{servername}
      storageAccessModes:
      - ReadWriteMany
      storageClass: nfs-client
      volumeNameTemplate: volume-{username}{servername}
    type: dynamic
  image:
    name: 'jupyter/scipy-notebook'
    tag: 'latest'

manics · October 8, 2023, 7:31pm

I can’t see any obvious problems with your latest config. Can you try adding some log statements to pre_spawn_start()?
E.g.

self.log.info(f"pre_spawn_start auth_state: {auth_state}")
self.log.info(f"environment: {spawner.environment}")

These should appear in your hub logs.

mahendrapaipuri · October 8, 2023, 9:53pm

From the logs and config you posted, it seems you are starting single user notebooks as root. In that case, I think you will need to pass c.Spawner.args = ['--allow-root'] in the JupyterHub config. Could you try with that config?

From your logs, I have a feeling that notebook server never started due to absence of --allow-root in spawner args.

manics · October 9, 2023, 8:28am

Assuming the singleuser configuration is correct, --allow-root shouldn’t be necessary as the startup scripts will modify the built-in user to match NB_USER/NB_UID/NB_GID, and switch to that user before executing the singleuser server:

github.com

jupyter/docker-stacks/blob/fbbf24585dd63f0d6a6709aed1640f03782bf21f/images/docker-stacks-foundation/start.sh#L41-L55


      
          # If the container started as the root user, then we have permission to refit
          # the jovyan user, and ensure file permissions, grant sudo rights, and such
          # things before we run the command passed to start.sh as the desired user
          # (NB_USER).
          #
          if [ "$(id -u)" == 0 ] ; then
              # Environment variables:
              # - NB_USER: the desired username and associated home folder
              # - NB_UID: the desired user id
              # - NB_GID: a group id we want our user to belong to
              # - NB_GROUP: a group name we want for the group
              # - GRANT_SUDO: a boolean ("1" or "yes") to grant the user sudo rights
              # - CHOWN_HOME: a boolean ("1" or "yes") to chown the user's home folder
              # - CHOWN_EXTRA: a comma separated list of paths to chown
              # - CHOWN_HOME_OPTS / CHOWN_EXTRA_OPTS: arguments to the chown commands

mahendrapaipuri · October 9, 2023, 8:42am

I think this log line is emitted only when we attempt to start jupyter server as root, right?! So, I thought something in single user config is not being taken into account and notebook is being launched as root.

hiroyuki-sato · October 9, 2023, 3:44pm

Hello, @mahendrapaipuri and @manics. Thank you for your reply.

My question: Does z2jh singleuser.cmd is set start-singleuser.sh by default?

I’m still debuging… but I have some progressed.

After setting c.Spawner.args = ['--allow-root'](I’ll try to remove this later), JupyterHub shows console. (At least not crashed).
But, It doesn’t change owner as I expected. The file onwer is root user not jovyan.

I’m connect the server using kubectl exec --stdin --tty --namespace=ns-z2jh jupyter-bob -- /bin/bash
and debug it.

After investigate, I seems that the image doesn’t execute /usr/local/bin/start-singleuser.sh.
So I added singleuser.cmd like the below
And finally, I found the entry bob:x:2001:2001::/home/bob:/bin/bash in /etc/passwd file.

singleuser:
  cmd: start-singleuser.sh

This comment start-singleuser.sh is z2jh default. It it true?

# jupyter_server / ServerApp will be used i think
singleuser:
  cmd: start-singleuser.sh # z2jh default!

helm list --all-namespaces
NAME                           	NAMESPACE           	REVISION	UPDATED                                	STATUS  	CHART                                     	APP VERSION
jhub                           	ns-z2jh             	48      	2023-10-09 17:23:01.677595372 +0900 JST	deployed	jupyterhub-3.0.3                          	4.0.2

Best regards

manics · October 9, 2023, 6:50pm

Good point… I’d forgotten about that!

singleuser.cmd defaults to jupyterhb-singleuser

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/8ccf2609ba5a42a9430f6940c04b2d8fab9bfc1d/jupyterhub/values.yaml#L413


      
            startTimeout: 300
            cpu:
              limit:
              guarantee:
            memory:
              limit:
              guarantee: 1G
            extraResource:
              limits: {}
              guarantees: {}
            cmd: jupyterhub-singleuser
            defaultUrl:
            extraPodConfig: {}
            profileList: []
          
          # scheduling relates to the user-scheduler pods and user-placeholder pods.
          scheduling:
            userScheduler:
              enabled: true
              revisionHistoryLimit:
              replicas: 2

If you unset it

singleuser:
  cmd:

it’ll use the image’s default command which is what you need

hiroyuki-sato · October 10, 2023, 12:19am

Hello, @manics. Thank you for your reply.

I had not been set singleuser.cmd.
It seems that Dockerfile use start-singleuser.sh instead of start-singleuser.

github.com

jupyter/docker-stacks/blob/fbbf24585dd63f0d6a6709aed1640f03782bf21f/images/base-notebook/Dockerfile#L58


      
              fix-permissions "${CONDA_DIR}" && \
              fix-permissions "/home/${NB_USER}"
          
          ENV JUPYTER_PORT=8888
          EXPOSE $JUPYTER_PORT
          
          # Configure container startup
          CMD ["start-notebook.sh"]
          
          # Copy local files as late as possible to avoid cache busting
          COPY start-notebook.sh start-singleuser.sh /usr/local/bin/
          COPY jupyter_server_config.py docker_healthcheck.py /etc/jupyter/
          
          # Fix permissions on /etc/jupyter as root
          USER root
          RUN fix-permissions /etc/jupyter/
          
          # HEALTHCHECK documentation: https://docs.docker.com/engine/reference/builder/#healthcheck
          # This healtcheck works well for `lab`, `notebook`, `nbclassic`, `server` and `retro` jupyter commands
          # https://github.com/jupyter/docker-stacks/issues/915#issuecomment-1068528799
          HEALTHCHECK --interval=5s --timeout=3s --start-period=5s --retries=3 \

Run start-singleuser(No .sh suffix) returns command not found.
This is jupyter/scipy-notebook image.

Does this behavior is expected?

ls -1 /usr/local/bin
before-notebook.d
fix-permissions
run-hooks.sh
start-notebook.d
start-notebook.sh
start.sh
start-singleuser.sh

start-singleuser
bash: start-singleuser: command not found

Best regards

manics · October 10, 2023, 3:13pm

What’s your current configuration? I’d expect the snippet I shared earlier where cmd is unset

singleuser:
  cmd:

to work

hiroyuki-sato · October 11, 2023, 5:37am

Hello, @manics. Thank you for your reply.

It worked as I expected when I set belows:

singieluser.cmd: (empty)
don’t set NB_UID explicitly
Remove: c.Spawner.args = ['--allow-root']

singleuser.cmd worked as I expected if I set below.

`singleuser.cmd`	Result	NG/OK
no entry	UID never changed	NG
`cmd:` (empty)	UID changed	OK
`cmd: start-singleuser.sh`	UID changed	OK

Removing `NB_UID`

I expect /home/bob links to /home/jovyan, but It create the new directory /home/bob. NFS mount point is /home/jovyan. so I can’t access via NFS.
After removing NB_UID setting, It worked as I expected.

Configuration (Only LDAP part)

hub:
  extraConfig:
    SpawnerCustomConfig: |
      from ldapauthenticator import LDAPAuthenticator
      from hashlib import md5

      class LDAPAuthenticatorInfo(LDAPAuthenticator):
          async def pre_spawn_start(self, user, spawner):
              auth_state = await user.get_auth_state()

              if not auth_state:
                  return

              self.log.debug(f"pre_spawn_start hogehogehoge")
              # https://github.com/jupyter/docker-stacks/blob/fde8f1eec3e70edf67560b5430545a22382334bf/base-notebook/start.sh#L41-L102
              spawner.environment["NB_UID"]  = str(auth_state["uidNumber"][0])
              #spawner.environment["NB_USER"] = auth_state["uid"][0]
              spawner.environment["NB_GID"]  = str(auth_state["gidNumber"][0])
              spawner.environment["CHOWN_HOME"]  = "yes"
              #spawner.environment["GRANT_SUDO"]  = "yes"
              self.log.info(f"environment: {spawner.environment}")

Configuration

hub:
  db:
    pvc:
      storageClassName: nfs-client
  config:
    Authenticator:
      enable_auth_state: true
    JupyterHub:
      authenticator_class: ldapauthenticator.LDAPAuthenticator
    # https://github.com/manics/zero-to-jupyterhub-k8s-examples/blob/5212ad2440e9ccf3831b5957b80775e58581b232/ldap-singleuser/jupyterhub.yml
    LDAPAuthenticator:
      server_address: nfs.example.com
      lookup_dn: True
      bind_dn_template:
        - uid={username},ou=users,dc=example,dc=com
      user_search_base: "dc=example,dc=com"
      user_attribute: uid
      lookup_dn_user_dn_attribute: uid
      escape_userdn: True
      use_ssl: true
        #server_port: 689
      auth_state_attributes: ["uid", "uidNumber", "gidNumber", "homeDirectory"]
      use_lookup_dn_username: False
  extraConfig:
    SpawnerCustomConfig: |
      from ldapauthenticator import LDAPAuthenticator
      from hashlib import md5

      class LDAPAuthenticatorInfo(LDAPAuthenticator):
          async def pre_spawn_start(self, user, spawner):
              auth_state = await user.get_auth_state()

              if not auth_state:
                  return

              # https://github.com/jupyter/docker-stacks/blob/fde8f1eec3e70edf67560b5430545a22382334bf/base-notebook/start.sh#L41-L102
              spawner.environment["NB_UID"]  = str(auth_state["uidNumber"][0])
              #spawner.environment["NB_USER"] = auth_state["uid"][0]
              spawner.environment["NB_GID"]  = str(auth_state["gidNumber"][0])
              spawner.environment["CHOWN_HOME"]  = "yes"
              #spawner.environment["GRANT_SUDO"]  = "yes"
              self.log.info(f"environment: {spawner.environment}")


      c.JupyterHub.authenticator_class = LDAPAuthenticatorInfo

    CustomHubConfig: |
      c.JupyterHub.cleanup_servers = True
      #c.Spawner.args = ['--allow-root']


debug:
  enabled: true
singleuser:
  cmd:
  uid: 0
  memory:
    limit: 2G
    guarantee: 2G
  storage:
    type: static
    static:
      pvcName: static-pvc
      subPath: "{username}"
  image:
    name: 'jupyter/scipy-notebook'
    tag: 'latest'

manics · October 11, 2023, 8:30am

If NB_USER is the same as your JupyterHub username I think

singleuser:
  storage:
    homeMountPath: "{username}"

may work

hiroyuki-sato · October 11, 2023, 11:33am

Hello, @manics. Thank you for your reply.

I’ll try homeMountPath.

Please tell me the configuration detail.

If I don’t set singleuser.cmd: (No entry like comment out # singleuser.cmd:) What command execute? (jupyterhub-singleuser?)
It seems that jupyterhub-singleuser is missing .sh suffix. Is this correct? It is necessary to fix in the future version?

singleuser.cmd	command
`singleuser.cmd:`(empty, unset)	Execute image deafult command
no entry	`jupyterhub-singleuser`

Best regards.

manics · October 11, 2023, 12:00pm

It uses the default command that’s defined to the image

jupyterhub-singleuser is an executable that’s installed by the jupyterhub Python package. It doesn’t need a file extension.

hiroyuki-sato · October 11, 2023, 12:51pm

Hello, @manics

Thank you for you reply.
I found the file /opt/conda/bin/jupyterhub-singleuser in the offical jupyter/scipy-notebook.

I understood like the following.

A

singleuser:
  #cmd: start-singleuser # z2jh default!

B

singleuser:
  cmd:

No.	config	execute script
A	no `cmd:` entry	`/opt/conda/bin/jupyterhub-singleuser`(z2jh default)
B	empty `singleuser.cmd`	`CMD ["start-notebook.sh"]` (the default command that’s defined to the image)

github.com

jupyterhub/zero-to-jupyterhub-k8s/blob/8ccf2609ba5a42a9430f6940c04b2d8fab9bfc1d/jupyterhub/values.yaml#L413


      
            startTimeout: 300
            cpu:
              limit:
              guarantee:
            memory:
              limit:
              guarantee: 1G
            extraResource:
              limits: {}
              guarantees: {}
            cmd: jupyterhub-singleuser
            defaultUrl:
            extraPodConfig: {}
            profileList: []
          
          # scheduling relates to the user-scheduler pods and user-placeholder pods.
          scheduling:
            userScheduler:
              enabled: true
              revisionHistoryLimit:
              replicas: 2

github.com

jupyter/docker-stacks/blob/4c0c0aa1715f8f5f964192462228a84bb540ed69/images/base-notebook/Dockerfile#L55


      
              npm cache clean --force && \
              jupyter lab clean && \
              rm -rf "/home/${NB_USER}/.cache/yarn" && \
              fix-permissions "${CONDA_DIR}" && \
              fix-permissions "/home/${NB_USER}"
          
          ENV JUPYTER_PORT=8888
          EXPOSE $JUPYTER_PORT
          
          # Configure container startup
          CMD ["start-notebook.sh"]
          
          # Copy local files as late as possible to avoid cache busting
          COPY start-notebook.sh start-singleuser.sh /usr/local/bin/
          COPY jupyter_server_config.py docker_healthcheck.py /etc/jupyter/
          
          # Fix permissions on /etc/jupyter as root
          USER root
          RUN fix-permissions /etc/jupyter/
          
          # HEALTHCHECK documentation: https://docs.docker.com/engine/reference/builder/#healthcheck

In the jupyter/scipy-notebook image.

find / -name 'jupyterhub-singleuser' -print 2> /dev/null
/opt/conda/bin/jupyterhub-singleuser

manics · October 11, 2023, 1:18pm

Where did you find this?

hiroyuki-sato · October 11, 2023, 1:36pm

Oh, Sorry. I missed remove comment.

singleuser:
  #cmd: jupyterhub-singleuser # z2jh default

I read this. But I think this setting already changed.

Topic		Replies	Views
KubeSpawner and LDAPauthentication run under users LDAP UID Zero to JupyterHub on Kubernetes help-wanted	8	3399	May 31, 2023
LDAP uid/gid integration with jupyterhub / zero-to-jupyterhub-k8s:1.1.3 Zero to JupyterHub on Kubernetes jupyterhub , help-wanted	0	669	October 26, 2021
Starting single-user notebook with our custom ldap docker image JupyterHub how-to	9	7357	March 22, 2023
Can't Change jupyter/datascience-notebook UID and GID JupyterHub	2	324	March 21, 2024
LDAP uid/gid integration with jupyterhub/k8s-hub:0.9.0-beta.2 JupyterHub	1	1463	January 21, 2020

1. My Goal.

2. Problem.

3. Questions.

4. Setting

5. log

7. Environment

1. The part of authenticator_class.

2. log file

3. complete configuration.

Removing NB_UID

Configuration (Only LDAP part)

Configuration

Related topics

1. The part of `authenticator_class`.

Removing `NB_UID`