I am trying to deploy Jupyterhub 2.0.0 helm chart on k8s version 1.27.2 inside minikube VM. When trying to deploy the helm chart locally, I get the following error on hub pod (after this pod keeps on restarting with the same error):
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: hub_config
[I 2023-05-31 03:35:16.707 JupyterHub app:2775] Running JupyterHub version 3.0.0
[I 2023-05-31 03:35:16.707 JupyterHub app:2805] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.0.0
[I 2023-05-31 03:35:16.796 alembic.runtime.migration migration:204] Context impl SQLiteImpl.
[I 2023-05-31 03:35:16.796 alembic.runtime.migration migration:207] Will assume non-transactional DDL.
[I 2023-05-31 03:35:16.810 alembic.runtime.migration migration:618] Running stamp_revision -> 651f5419b74d
[I 2023-05-31 03:35:16.823 alembic.runtime.migration migration:204] Context impl SQLiteImpl.
[I 2023-05-31 03:35:16.823 alembic.runtime.migration migration:207] Will assume non-transactional DDL.
[I 2023-05-31 03:35:17.228 JupyterHub roles:173] Role jupyterhub-idle-culler added to database
[I 2023-05-31 03:35:17.243 JupyterHub roles:238] Adding role admin for User: zeus
[I 2023-05-31 03:35:17.252 JupyterHub roles:238] Adding role user for User: zeus
[I 2023-05-31 03:35:17.259 JupyterHub app:1934] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-05-31 03:35:17.496 JupyterHub app:2844] Initialized 0 spawners in 0.075 seconds
[I 2023-05-31 03:35:17.502 JupyterHub app:3057] Not starting proxy
[W 2023-05-31 03:35:37.518 JupyterHub proxy:903] api_request to the proxy failed with status code 599, retrying...
[W 2023-05-31 03:35:57.655 JupyterHub proxy:903] api_request to the proxy failed with status code 599, retrying...
[E 2023-05-31 03:35:57.656 JupyterHub app:3297]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3295, in launch_instance_async
await self.start()
File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3061, in start
await self.proxy.get_all_routes()
File "/usr/local/lib/python3.9/site-packages/jupyterhub/proxy.py", line 950, in get_all_routes
resp = await self.api_request('', client=client)
File "/usr/local/lib/python3.9/site-packages/jupyterhub/proxy.py", line 914, in api_request
result = await exponential_backoff(
File "/usr/local/lib/python3.9/site-packages/jupyterhub/utils.py", line 236, in exponential_backoff
raise asyncio.TimeoutError(fail_message)
asyncio.exceptions.TimeoutError: Repeated api_request to proxy path "" failed.
Is jupyterhub tried/tested on k8s 1.27.2? If yes, can someone help me with the above error? Any inputs will be appreciated!
Thanks again for quick response @manics! My local setup of kubernetes was messed up. After properly setting up local k8s with 1.27.2 version, jupyterhub pods comes up fine. However, when trying to spawn singleuser container, userscheduler pod fails with:
E0531 18:27:29.478156 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
The latest dev version should support 1.27, if it doesn’t please open a bug report with as much information as possible.
There are ~80 changed files between 2.0.0 and the main branch, but most of those are docs or related to CI. If you compare the diff for just the jupyterhub/ subdirectory you’ll hopefully figure out the relevant changes.
However, I tried overriding the userscheduler image tag to v.1.24.0 in my helm chart, but bringing up jupyterhub fails and userscheduler pod has the following error:
You’ve done more investigation than I have into this issue, and what you say sounds plausible!
There’s no firm release date for 3.0.0, but if you’ve tested a recent dev release and confirmed everything is working I’d pin that version and run with that. You can also try and modify 2.0.0 to work with k8s 1.27, but since that’s not something we’ve tested you’ll have to test it all yourself too.
As with all production systems ensure you’ve got ongoing backups of any user data regardless of which option you choose.
Appreciate your response here! If you could provide a rough estimate (I know it is difficult to give exact timelines) for the next major release, that would be really helpful!
Which version of the Z2JH chart are you using? Can you show us your full configuration, including any customisations you’ve made to the chart? How was your K8s cluster setup, and do you have any proxies/firewalls?
In jupyterhub 4, the _xsrf cookie set must have been set in a browser during a visit to a jupyterhub website, which then should pass back if its making a post request.
What software makes the post request? That may need to be made compatible with jupyterhub 4.
@consideRatio that’s what I’m surprised, the browser request does have the cookie but jupyterhub is not reading it somehow. The cookie was added to the browser when accessing the jupyterhub url
[I 2023-06-13 14:04:14.205 JupyterHub app:2889] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.0.0
[I 2023-06-13 14:04:14.205 JupyterHub app:2889] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-4.0.0
but I only see it for the Proxy.
When submitting forms, the xsrf token must be sent twice (that’s how xsrf tokens work), both in the cookie and in the form. This is how it is used to verify that the form was sent from a page with access to the cookie.
If the Authenticator is using a custom login form (or the default login form is customized by other means), it must include the xsrf token as a hidden input, as seen here.
