# Z2JH : Invalid client\_id parameter value

**URL:** <https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591>\
**Category:** Zero to JupyterHub on Kubernetes\
**Tags:** jupyterhub, help-wanted\
**Created:** [November 27, 2023, 12:37pm UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591 "2023-11-27T12:37:40Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [November 27, 2023, 12:37pm UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/1 "2023-11-27T12:37:40Z")

</div>

Hi All,

This issue is observed intermittently for a few users. After successful OAuth2 authentication, this error log is seen on the hub pod:

```auto
[D 2023-11-26 10:47:41.771 JupyterHub base:342] Refreshing auth for <username>
[D 2023-11-26 10:47:41.772 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[W 2023-11-26 10:47:41.776 JupyterHub web:1869] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjJmZjQ...hYiJ9 (10.2.136.231): Invalid client_id parameter value.
[W 2023-11-26 10:47:41.777 JupyterHub log:191] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=[secret] (<username>@10.2.136.231) 18.87ms
[I 2023-11-26 10:47:50.399 JupyterHub log:191] 302 GET /hub/ -> /user/<username>/ (<username>@10.2.136.231) 3.06ms
[D 2023-11-26 10:47:51.192 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[W 2023-11-26 10:47:51.193 JupyterHub web:1869] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=eyJ1dWlkIj...wYmcifQ (10.2.136.231): Invalid client_id parameter value.
[W 2023-11-26 10:47:51.195 JupyterHub log:191] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=[secret] (<username>@10.2.136.231) 5.54ms
[D 2023-11-26 10:50:29.937 JupyterHub scopes:690] Argument-based access to /hub/api/users/<username>/activity via users:activity
[D 2023-11-26 10:50:29.942 JupyterHub users:879] Not updating activity for <User(<username> 1/1 running)>: 2023-11-26T06:58:09.623259Z < 2023-11-26T10:47:41.766690Z
[D 2023-11-26 10:50:29.942 JupyterHub users:900] Not updating server activity on <username>/: 2023-11-26T06:58:09.623259Z < 2023-11-26T10:47:41.766690Z
[I 2023-11-26 10:50:29.943 JupyterHub log:191] 200 POST /hub/api/users/<username>/activity (<username>@172.16.2.82) 16.99ms
[D 2023-11-26 10:55:37.672 JupyterHub base:342] Refreshing auth for <username>

```

The user is locked out of their lab. Also on a hub pod restart, the user is able to work with their lab pod for sometime.

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [November 28, 2023, 9:39am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/2 "2023-11-28T09:39:24Z")

</div>

> [@yashali](#):
>
> `Invalid client_id parameter value`

This error means that there is [no oauth client registered with that client id](https://github.com/jupyterhub/jupyterhub/blob/2ecb31b1adadb869311185ff23f95f26a12de011/jupyterhub/oauth/provider.py#L420-L429) in the database. For example, this could happen if the user’s server has been shutdown. Can you find any other earlier logs for that user’s server and confirm whether it’s running or not? Can you also find any information about how long the server was running?

Check for an earlier logs like:

- `Removed client jupyterhub-user-<username>`

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [November 28, 2023, 10:14am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/3 "2023-11-28T10:14:06Z")

</div>

Hi @minrk , thank you for taking time to answer this.

Here are the logs for the “username”:

```auto

[I 2023-11-16 08:37:33.633 JupyterHub base:835] User logged in: <username>
[I 2023-11-16 08:37:33.886 JupyterHub log:191] 302 GET /hub/ -> /user/<username>/ (<username>@10.2.136.168) 10.14ms
[D 2023-11-16 08:37:34.674 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[D 2023-11-16 08:37:34.675 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /user/<username>/oauth_callback for client jupyterhub-user-<username>.
[D 2023-11-16 08:37:34.675 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /user/<username>/oauth_callback
[D 2023-11-16 08:37:34.675 JupyterHub provider:495] validate_redirect_uri: client_id=jupyterhub-user-<username>, redirect_uri=/user/<username>/oauth_callback
[D 2023-11-16 08:37:34.678 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes ['read:users:name!user', 'read:users:groups!user', 'access:servers!server=<username>/'] for client 'jupyterhub-user-<username>' (<OAuthClient(identifier='jupyterhub-user-<username>')>).
[D 2023-11-16 08:37:34.679 JupyterHub provider:622] Allowing request for scope(s) for jupyterhub-user-<username>: access:servers!server=<username>/,read:users:name!user,read:users:groups!user
[D 2023-11-16 08:37:34.680 JupyterHub auth:320] Skipping oauth confirmation for <User(<username> 1/1 running)> accessing Server at /user/<username>/
[D 2023-11-16 08:37:34.681 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[D 2023-11-16 08:37:34.682 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /user/<username>/oauth_callback for client jupyterhub-user-<username>.
[D 2023-11-16 08:37:34.682 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /user/<username>/oauth_callback
[D 2023-11-16 08:37:34.682 JupyterHub provider:495] validate_redirect_uri: client_id=jupyterhub-user-<username>, redirect_uri=/user/<username>/oauth_callback
[D 2023-11-16 08:37:34.683 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes {'read:users:name!user', 'read:users:groups!user', 'access:servers!server=<username>/'} for client 'jupyterhub-user-<username>' (<OAuthClient(identifier='jupyterhub-user-<username>')>).
[D 2023-11-16 08:37:34.684 JupyterHub provider:622] Allowing request for scope(s) for jupyterhub-user-<username>: access:servers!server=<username>/,read:users:name!user,read:users:groups!user
[D 2023-11-16 08:37:34.685 JupyterHub provider:246] Saving authorization code jupyterhub-user-<username>, TbG..., (), {}
[I 2023-11-16 08:37:34.692 JupyterHub log:191] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=[secret] -> /user/<username>/oauth_callback?code=[secret]&state=[secret] (<username>@10.2.136.168) 20.58ms
[D 2023-11-16 08:37:35.149 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:533] Using provided redirect_uri /user/<username>/oauth_callback
[D 2023-11-16 08:37:35.149 JupyterHub provider:116] confirm_redirect_uri: client_id=jupyterhub-user-<username>, redirect_uri=/user/<username>/oauth_callback
[D 2023-11-16 08:37:35.150 JupyterHub provider:344] Saving bearer token {'access_token': 'REDACTED', 'expires_in': 1209600, 'token_type': 'Bearer', 'scope': 'access:servers!server=<username>/ read:users:name!user read:users:groups!user', 'refresh_token': 'REDACTED'}
[D 2023-11-16 08:37:35.158 JupyterHub provider:204] Deleting oauth code TbG... for jupyterhub-user-<username>
[I 2023-11-16 08:37:35.168 JupyterHub log:191] 200 POST /hub/api/oauth2/token (<username>@172.16.2.8) 36.43ms
[D 2023-11-16 08:37:35.177 JupyterHub base:297] Recording first activity for <APIToken('Vve1...', user='<username>', client_id='jupyterhub-user-<username>')>
[I 2023-11-16 09:05:08.143 JupyterHub log:191] 200 GET /hub/api/user (<username>@172.16.2.8) 12.10ms
[D 2023-11-16 09:20:21.088 JupyterHub base:342] Refreshing auth for <username>
[I 2023-11-16 09:20:21.089 JupyterHub log:191] 302 GET /hub/ -> /user/<username>/ (<username>@10.2.136.168) 12.35ms
[D 2023-11-16 09:20:21.869 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[W 2023-11-16 09:20:21.870 JupyterHub web:1869] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImU5...9zaW5nc2lkL2xhYj8ifQ (10.2.136.168): Invalid client_id parameter value.
[W 2023-11-16 09:20:21.872 JupyterHub log:191] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=[secret] (<username>@10.2.136.168) 5.69ms

```

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [November 28, 2023, 1:26pm UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/4 "2023-11-28T13:26:26Z")

</div>

There are no other log lines containing `<username>` between 08:37 and 09:20?

Had JupyterHub restarted during that time? What Spawner are you using?

Can you share `pip list` or `conda list` for your hub environment?

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [November 28, 2023, 1:51pm UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/5 "2023-11-28T13:51:30Z")

</div>

Hi @minrk Yes there are more log lines for the “username” :

```auto

[I 2023-11-16 08:37:33.633 JupyterHub base:835] User logged in: <username>
[I 2023-11-16 08:37:33.886 JupyterHub log:191] 302 GET /hub/ -> /user/<username>/ (<username>@10.2.136.168) 10.14ms
[D 2023-11-16 08:37:34.674 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[D 2023-11-16 08:37:34.675 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /user/<username>/oauth_callback for client jupyterhub-user-<username>.
[D 2023-11-16 08:37:34.675 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /user/<username>/oauth_callback
[D 2023-11-16 08:37:34.675 JupyterHub provider:495] validate_redirect_uri: client_id=jupyterhub-user-<username>, redirect_uri=/user/<username>/oauth_callback
[D 2023-11-16 08:37:34.678 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes ['read:users:name!user', 'read:users:groups!user', 'access:servers!server=<username>/'] for client 'jupyterhub-user-<username>' (<OAuthClient(identifier='jupyterhub-user-<username>')>).
[D 2023-11-16 08:37:34.679 JupyterHub provider:622] Allowing request for scope(s) for jupyterhub-user-<username>: access:servers!server=<username>/,read:users:name!user,read:users:groups!user
[D 2023-11-16 08:37:34.680 JupyterHub auth:320] Skipping oauth confirmation for <User(<username> 1/1 running)> accessing Server at /user/<username>/
[D 2023-11-16 08:37:34.681 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[D 2023-11-16 08:37:34.682 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /user/<username>/oauth_callback for client jupyterhub-user-<username>.
[D 2023-11-16 08:37:34.682 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /user/<username>/oauth_callback
[D 2023-11-16 08:37:34.682 JupyterHub provider:495] validate_redirect_uri: client_id=jupyterhub-user-<username>, redirect_uri=/user/<username>/oauth_callback
[D 2023-11-16 08:37:34.683 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes {'read:users:name!user', 'read:users:groups!user', 'access:servers!server=<username>/'} for client 'jupyterhub-user-<username>' (<OAuthClient(identifier='jupyterhub-user-<username>')>).
[D 2023-11-16 08:37:34.684 JupyterHub provider:622] Allowing request for scope(s) for jupyterhub-user-<username>: access:servers!server=<username>/,read:users:name!user,read:users:groups!user
[D 2023-11-16 08:37:34.685 JupyterHub provider:246] Saving authorization code jupyterhub-user-<username>, TbG..., (), {}
[I 2023-11-16 08:37:34.692 JupyterHub log:191] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=[secret] -> /user/<username>/oauth_callback?code=[secret]&state=[secret] (<username>@10.2.136.168) 20.58ms
[D 2023-11-16 08:37:35.149 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:533] Using provided redirect_uri /user/<username>/oauth_callback
[D 2023-11-16 08:37:35.149 JupyterHub provider:116] confirm_redirect_uri: client_id=jupyterhub-user-<username>, redirect_uri=/user/<username>/oauth_callback
[D 2023-11-16 08:37:35.150 JupyterHub provider:344] Saving bearer token {'access_token': 'REDACTED', 'expires_in': 1209600, 'token_type': 'Bearer', 'scope': 'access:servers!server=<username>/ read:users:name!user read:users:groups!user', 'refresh_token': 'REDACTED'}
[D 2023-11-16 08:37:35.158 JupyterHub provider:204] Deleting oauth code TbG... for jupyterhub-user-<username>
[I 2023-11-16 08:37:35.168 JupyterHub log:191] 200 POST /hub/api/oauth2/token (<username>@172.16.2.8) 36.43ms
[D 2023-11-16 08:37:35.177 JupyterHub base:297] Recording first activity for <APIToken('Vve1...', user='<username>', client_id='jupyterhub-user-<username>')>
[I 2023-11-16 08:37:35.187 JupyterHub log:191] 200 GET /hub/api/user (<username>@172.16.2.8) 14.73ms
[D 2023-11-16 08:42:40.140 JupyterHub base:342] Refreshing auth for <username>
[I 2023-11-16 08:42:40.141 JupyterHub log:191] 200 GET /hub/api/user (<username>@172.16.2.8) 11.48ms
[D 2023-11-16 08:47:41.403 JupyterHub base:342] Refreshing auth for <username>
[I 2023-11-16 08:47:41.403 JupyterHub log:191] 200 GET /hub/api/user (<username>@172.16.2.8) 11.62ms
[D 2023-11-16 08:50:12.499 JupyterHub scopes:677] Unrestricted access to /hub/api/users/<username> via read:users
[I 2023-11-16 08:50:12.501 JupyterHub log:191] 200 GET /hub/api/users/<username> (refresher@172.16.4.6) 9.64ms
[D 2023-11-16 08:52:45.609 JupyterHub base:342] Refreshing auth for <username>
[I 2023-11-16 08:52:45.609 JupyterHub log:191] 200 GET /hub/api/user (<username>@172.16.2.8) 13.50ms
[D 2023-11-16 09:05:08.142 JupyterHub base:342] Refreshing auth for <username>
[I 2023-11-16 09:05:08.143 JupyterHub log:191] 200 GET /hub/api/user (<username>@172.16.2.8) 12.10ms
[D 2023-11-16 09:20:21.088 JupyterHub base:342] Refreshing auth for <username>
[I 2023-11-16 09:20:21.089 JupyterHub log:191] 302 GET /hub/ -> /user/<username>/ (<username>@10.2.136.168) 12.35ms
[D 2023-11-16 09:20:21.869 JupyterHub provider:420] Validating client id jupyterhub-user-<username>
[W 2023-11-16 09:20:21.870 JupyterHub web:1869] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImU5MDY4Mm...9zaW5nc2lkL2xhYj8ifQ (10.2.136.168): Invalid client_id parameter value.
[W 2023-11-16 09:20:21.872 JupyterHub log:191] 400 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-<username>&redirect_uri=%2Fuser%2F<username>%2Foauth_callback&response_type=code&state=[secret] (<username>@10.2.136.168) 5.69ms

```

We are using KubeSpawner. And Jhub didnt restart

pip list:

```auto
Package Version
-------------------------------- ------------
aiohttp 3.8.1
aiosignal 1.2.0
alembic 1.9.1
async-generator 1.10
async-timeout 4.0.2
attrs 22.2.0
azure-core 1.23.1
azure-identity 1.9.0
bcrypt 4.0.0
cachetools 5.3.2
certifi 2022.12.7
certipy 0.1.3
cffi 1.15.1
charset-normalizer 2.1.1
cryptography 39.0.0
cx-Oracle 8.3.0
escapism 1.0.1
frozenlist 1.3.1
google-auth 2.23.3
greenlet 2.0.1
idna 3.4
importlib-metadata 6.0.0
importlib-resources 6.1.0
Jinja2 3.1.2
jsonschema 4.17.3
jupyter-telemetry 0.1.0
jupyterhub 4.0.0
jupyterhub-firstuseauthenticator 1.0.0
jupyterhub-hmacauthenticator 1.0
jupyterhub-idle-culler 1.2.1
jupyterhub-kubespawner 4.2.0
jupyterhub-ldapauthenticator 1.3.2
jupyterhub-ltiauthenticator 1.2.0
jupyterhub-nativeauthenticator 1.1.0
jupyterhub-tmpauthenticator 0.6
kubernetes 28.1.0
kubernetes-asyncio 24.2.2
ldap3 2.9.1
Mako 1.2.4
MarkupSafe 2.1.1
msal 1.24.1
msal-extensions 0.3.1
our_pvt_auth_class 0.1.4
multidict 6.0.4
mwoauth 0.3.8
nullauthenticator 1.0.0
numpy 1.24.4
oauthenticator 15.1.0
oauthlib 3.2.2
onetimepass 1.0.1
packaging 23.0
pamela 1.0.0
pandas 2.0.3
pip 23.3.1
pkgutil_resolve_name 1.3.10
portalocker 2.8.2
prometheus-client 0.15.0
psycopg2-binary 2.9.3
py-spy 0.3.14
pyasn1 0.4.8
pyasn1-modules 0.3.0
pycparser 2.21
pycurl 7.45.2
PyJWT 2.4.0
PyMySQL 1.0.2
pyOpenSSL 23.0.0
pyparsing 3.0.9
pyrsistent 0.19.3
python-dateutil 2.8.2
python-json-logger 2.0.4
python-slugify 8.0.1
pytz 2023.3.post1
PyYAML 6.0
requests 2.28.1
requests-oauthlib 1.3.1
rsa 4.9
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.7
setuptools 56.0.0
six 1.16.0
SQLAlchemy 1.4.46
sqlalchemy-cockroachdb 1.4.4
statsd 3.3.0
text-unidecode 1.3
tornado 6.3.3
traitlets 5.8.1
typing_extensions 4.8.0
tzdata 2023.3
urllib3 1.26.14
websocket-client 1.6.4
yarl 1.9.2
zipp 3.11.0

```

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [November 29, 2023, 7:29am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/6 "2023-11-29T07:29:42Z")

</div>

I’m afraid I don’t have an idea for what to check next. You could try upgrading, since some packages (especially kubespawner and jupyterhub) are a bit out of date, but I don’t currently see how the changes since your versions would affect this.

Somehow the oauth client is getting deleted, but it’s strange that that would happen without shutting down the user’s server.

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [November 29, 2023, 9:32am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/7 "2023-11-29T09:32:17Z")

</div>

Sure let me try upgrading everything. Thanks @minrk for your help! Will post here again soon with further updates 🙂

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [December 7, 2023, 6:47am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/8 "2023-12-07T06:47:59Z")

</div>

@minrk Do you think this error might occur on one of the deployments if we have registered two urls (for 2 different K8s deployments of Z2JH) with Auth0 under the same auth0\_client\_id and a user logs in and out of both the deployments at the same time in some order that might break it? Both deployments have separate Hub DBs.

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [December 7, 2023, 10:25am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/9 "2023-12-07T10:25:04Z")

</div>

That is conceivable, I’ll have to think more about how it might happen. This is specifically the oauth between the Hub and the single-user server, which is not connected to the oauth between the hub and your oauth provider (more info [in the docs](https://jupyterhub.readthedocs.io/en/latest/explanation/oauth.html)), but having two sessions for hubs on sufficiently similar domains _could_ enter into it.

Do the two hubs run on the same or similar domains? Can you share either their actual URLs or two URLs that have the same relationship (e.g. `hub1.example.com` and `hub2.example.com` vs `hubs.example.com/hub1` and `hubs.example.com/hub2`)?

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [December 7, 2023, 11:38am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/10 "2023-12-07T11:38:57Z")

</div>

@minrk , the domains are sufficiently different : `hub1.example.com` and `hub2.example.com`. But have the same `auth0_client_id`. Some points to note:

- error occurs on hub2 only. hub1 is on K8s 1.24 and hub2 on K8s 1.27. Both deployments use all same images
- restarting the hub or the user server fixes the issue for some time before it happens again
- hub1 and hub2 have separate hub dbs

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [December 18, 2023, 5:18am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/11 "2023-12-18T05:18:14Z")

</div>

Hi @minrk , any thoughts on this?

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [December 20, 2023, 10:15am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/12 "2023-12-20T10:15:50Z")

</div>

If they both have the same auth0 client, does that mean they both have the same redirect uri with auth0?

I don’t have any great ideas right now, but if you can send complete debug-level logs containing `username` (possibly even from both the hub and the single-user server), I might be able to make sense of it.

It really should mean that the oauth client is being deleted, which should generally mean that the server has been stopped. But I haven’t seen any log lines to that effect.

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [February 16, 2024, 8:22am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/13 "2024-02-16T08:22:42Z")

</div>

Hi @minrk,  
We have a few new observations with respect to this issue.

Under what circumstances inactive user-pods should be culled, but the user-pods still be up and the database looks as follows???

spawners table :-  
12 12 “{”“pod\_name”“: ““jupyter-username””}” “2024-02-06 09:45:30.758641” “2024-02-06 15:18:32.638” “{”“profile”“: ““my-profile””}”

servers table is empty

oauth\_clients table also is empty

But the user-pod is up 😯 and here are the logs:

```auto
[D 2024-02-15 12:44:50.869 SingleUserLabApp mixins:491] Notifying Hub of activity 2024-02-06T10:51:15.413727Z
[E 2024-02-15 12:44:50.873 SingleUserLabApp mixins:511] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 509, in notify
        await client.fetch(req)
      File "/opt/conda/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
        future.result()
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2024-02-15 12:44:50.873 SingleUserLabApp mixins:536] Error notifying Hub of activity
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 534, in keep_activity_updated
        await self.notify_activity()
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 516, in notify_activity
        await exponential_backoff(
      File "/opt/conda/lib/python3.8/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Failed to notify Hub of activity
[D 2024-02-15 12:49:20.982 SingleUserLabApp mixins:491] Notifying Hub of activity 2024-02-06T10:51:15.413727Z
[D 2024-02-15 12:53:55.058 SingleUserLabApp mixins:491] Notifying Hub of activity 2024-02-06T10:51:15.413727Z
[D 2024-02-15 12:58:39.312 SingleUserLabApp mixins:491] Notifying Hub of activity 2024-02-06T10:51:15.413727Z
[D 2024-02-15 13:03:30.615 SingleUserLabApp mixins:491] Notifying Hub of activity 2024-02-06T10:51:15.413727Z

```

Here are the hub logs for this user-pod:

```auto
[D 2024-02-15 12:43:14.183 JupyterHub proxy:880] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2024-02-15 12:43:14.184 JupyterHub proxy:953] Omitting non-jupyterhub route '/'
[D 2024-02-15 12:43:14.184 JupyterHub proxy:392] Checking routes
[W 2024-02-15 12:43:14.185 JupyterHub proxy:468] Deleting stale route /user/username/
[D 2024-02-15 12:43:14.185 JupyterHub proxy:880] Proxy: Fetching DELETE http://proxy-api:8001/api/routes/user/username
[I 2024-02-15 12:43:14.187 JupyterHub app:3242] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[D 2024-02-15 12:43:14.188 JupyterHub app:2847] It took 3.102 seconds for the Hub to start
[D 2024-02-15 12:43:14.279 JupyterHub base:297] Recording first activity for <APIToken('66bd...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2024-02-15 12:43:14.286 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 9.60ms
[D 2024-02-15 12:43:14.292 JupyterHub scopes:863] Checking access via scope list:users
--
[D 2024-02-15 12:45:04.323 JupyterHub reflector:362] events watcher timeout
[D 2024-02-15 12:45:04.323 JupyterHub reflector:281] Connecting events watcher
[D 2024-02-15 12:45:04.457 JupyterHub reflector:362] pods watcher timeout
[D 2024-02-15 12:45:04.457 JupyterHub reflector:281] Connecting pods watcher
[D 2024-02-15 12:45:06.604 JupyterHub log:191] 200 GET /hub/health (@172.16.4.1) 0.62ms
[D 2024-02-15 12:45:08.310 JupyterHub base:342] Refreshing auth for username
[D 2024-02-15 12:45:08.317 JupyterHub scopes:863] Checking access via scope users:activity
[D 2024-02-15 12:45:08.317 JupyterHub scopes:690] Argument-based access to /hub/api/users/username/activity via users:activity
[D 2024-02-15 12:45:08.319 JupyterHub users:879] Not updating activity for <User(username 0/2 running)>: 2024-02-06T14:27:15.018783Z < 2024-02-14T06:57:21.708712Z
[D 2024-02-15 12:45:08.319 JupyterHub users:900] Not updating server activity on username/: 2024-02-06T14:27:15.018783Z < 2024-02-14T06:57:21.708712Z
[I 2024-02-15 12:45:08.320 JupyterHub log:191] 200 POST /hub/api/users/username/activity (username@172.16.3.35) 28.14ms
[D 2024-02-15 12:45:14.188 JupyterHub proxy:880] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2024-02-15 12:45:14.191 JupyterHub proxy:953] Omitting non-jupyterhub route '/'
[D 2024-02-15 12:45:14.199 JupyterHub proxy:392] Checking routes
[D 2024-02-15 12:45:14.200 JupyterHub utils:278] Server at http://refresher:8081/services/refresher/ responded with 404
[D 2024-02-15 12:45:14.200 JupyterHub app:2493] External service refresher running at http://refresher:8081/

```

Culler configuration in the deployment-values file looks as follows:

```auto
cull:
  adminUsers: false # --cull-admin-users
  removeNamedServers: true # --remove-named-servers
  timeout: 72000
  every: 600
  concurrency: 10 # --concurrency
  maxAge: 0 # --max-age

```

The culler defined is as follows:

```auto
if get_config("cull.enabled", False):
    jupyterhub_idle_culler_role = {
        "name": "jupyterhub-idle-culler",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "delete:servers",
            # "admin:users", # dynamically added if --cull-users is passed
        ],
        # assign the role to a jupyterhub service, so it gains these permissions
        "services": ["jupyterhub-idle-culler"],
    }

    cull_cmd = ["python3", "-m", "jupyterhub_idle_culler"]
    base_url = c.JupyterHub.get("base_url", "/")
    cull_cmd.append("--url=http://localhost:8081" + url_path_join(base_url, "hub/api"))

    cull_timeout = get_config("cull.timeout")
    if cull_timeout:
        cull_cmd.append(f"--timeout={cull_timeout}")

    cull_every = get_config("cull.every")
    if cull_every:
        cull_cmd.append(f"--cull-every={cull_every}")

    cull_concurrency = get_config("cull.concurrency")
    if cull_concurrency:
        cull_cmd.append(f"--concurrency={cull_concurrency}")

    if get_config("cull.users"):
        cull_cmd.append("--cull-users")
        jupyterhub_idle_culler_role["scopes"].append("admin:users")

    if not get_config("cull.adminUsers"):
        cull_cmd.append("--cull-admin-users=false")

    if get_config("cull.removeNamedServers"):
        cull_cmd.append("--remove-named-servers")

    cull_max_age = get_config("cull.maxAge")
    if cull_max_age:
        cull_cmd.append(f"--max-age={cull_max_age}")

    c.JupyterHub.services.append(
        {
            "name": "jupyterhub-idle-culler",
            "command": cull_cmd,
        }
    )
    c.JupyterHub.load_roles.append(jupyterhub_idle_culler_role)

```

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [February 20, 2024, 8:23am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/14 "2024-02-20T08:23:40Z")

</div>

> [@yashali](#):
>
> Under what circumstances inactive user-pods should be culled, but the user-pods still be up

This should not happen. This is an “orphaned pod” situation, where JupyterHub _believes_ a user’s server has stopped when it hasn’t. Since you have kubespawner 4.2, upgrading z2jh and kubespawner may fix it. We have had [some issues related to orphaned pods recently](https://discourse.jupyter.org/t/how-to-cleanup-orphaned-user-pods-after-bug-in-z2jh-3-0-and-kubespawner-6-0/21677).

The key bit of logs to find is when JupyterHub is deciding to stop the server, or deciding that the server has stopped. Do you still have the logs of jupyterhub startup _before_ `[D 2024-02-15 12:43:14.183 JupyterHub proxy:880]`?

There might be something like `username appears to have stopped while the Hub was down`. That would be the bug, or a similar one, that has been fixed in the latest releases of kubespawner/z2jh.

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [February 21, 2024, 5:36am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/15 "2024-02-21T05:36:38Z")

</div>

Thanks @minrk , we have upgraded all versions of the pkgs

```auto
Package Version
-------------------------------- ------------
aiohttp 3.9.1
aiosignal 1.3.1
alembic 1.13.1
async-generator 1.10
async-timeout 4.0.3
attrs 23.2.0
azure-core 1.29.7
azure-identity 1.15.0
bcrypt 4.1.2
cachetools 5.3.2
certifi 2023.11.17
certipy 0.1.3
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
cryptography 42.0.1
cx-Oracle 8.3.0
dask 2024.1.0
dask-gateway 2022.11.0
distributed 2024.1.0
docstring-parser 0.15
escapism 1.0.1
frozenlist 1.4.1
fsspec 2023.12.2
future 0.18.3
google-auth 2.27.0
greenlet 3.0.3
idna 3.6
importlib-metadata 7.0.1
Jinja2 3.1.3
JPype1 1.5.0
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter-telemetry 0.1.0
jupyterhub 4.0.0
jupyterhub-firstuseauthenticator 1.0.0
jupyterhub-hmacauthenticator 1.0
jupyterhub-idle-culler 1.2.1
jupyterhub-kubespawner 6.2.0
jupyterhub-ldapauthenticator 1.3.2
jupyterhub-ltiauthenticator 1.6.2
jupyterhub-nativeauthenticator 1.2.0
jupyterhub-tmpauthenticator 1.0.0
kazoo 2.9.0
kubernetes 29.0.0
kubernetes_asyncio 29.0.0
ldap3 2.9.1
locket 1.0.0
lxml 5.1.0
Mako 1.3.1
MarkupSafe 2.1.4
msal 1.26.0
msal-extensions 1.1.0
msgpack 1.0.7
multidict 6.0.4
mwoauth 0.4.0
NamedAtomicLock 1.1.3
nullauthenticator 1.0.0
numpy 1.26.3
oauthenticator 15.1.0
oauthlib 3.2.2
onetimepass 1.0.1
packaging 23.2
pamela 1.1.0
pandas 2.2.0
partd 1.4.1
pip 23.3.2
portalocker 2.8.2
prometheus-client 0.19.0
psutil 5.9.8
psycopg2-binary 2.9.9
py-spy 0.3.14
pyasn1 0.5.1
pyasn1-modules 0.3.0
pycparser 2.21
pycryptodome 3.20.0
pycurl 7.45.2
PyJWT 2.8.0
PyMySQL 1.1.0
pyOpenSSL 24.0.0
pyparsing 3.1.1
pyrsistent 0.20.0
python-dateutil 2.8.2
python-json-logger 2.0.7
python-slugify 8.0.1
pytz 2023.3.post1
PyYAML 6.0.1
referencing 0.32.1
requests 2.31.0
requests-oauthlib 1.3.1
rpds-py 0.17.1
rsa 4.9
ruamel.yaml 0.18.5
ruamel.yaml.clib 0.2.8
scipy 1.12.0
setuptools 58.1.0
six 1.16.0
sortedcontainers 2.4.0
SQLAlchemy 2.0.25
sqlalchemy-cockroachdb 2.0.2
statsd 4.0.1
tblib 3.0.0
text-unidecode 1.3
toolz 0.12.1
tornado 6.4
traitlets 5.14.1
typing_extensions 4.9.0
tzdata 2023.4
urllib3 2.1.0
websocket-client 1.7.0
yarl 1.9.4
zict 3.0.0
zipp 3.17.0

```

Will keep monitoring and post here when I have more updates 🙂

PS: I wonder if the orphaned pods issue as described in [the post](https://discourse.jupyter.org/t/how-to-cleanup-orphaned-user-pods-after-bug-in-z2jh-3-0-and-kubespawner-6-0/21677) is exactly the same as our issue or not. It looks like it’s the same, but we are still on z2jh version 2 and the post talks about the bug introduced in version 3.

---

<div class="post-metadata">

**Author:** ![consideRatio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/consideratio/32/1033_2.png) [@consideRatio](https://discourse.jupyter.org/u/consideRatio)\
**Post date:** [February 21, 2024, 6:48am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/16 "2024-02-21T06:48:50Z")

</div>

If you’ve never touched z2jh 3.0.0-alpha.1 or later, I’m fairly certain its not that bug - it was quite unique.

---

<div class="post-metadata">

**Author:** ![yashali](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/yashali/32/8696_2.png) [@yashali](https://discourse.jupyter.org/u/yashali)\
**Post date:** [March 18, 2024, 9:27am UTC](https://discourse.jupyter.org/t/z2jh-invalid-client-id-parameter-value/22591/17 "2024-03-18T09:27:18Z")

</div>

Yes, you are correct. Unfortunately, it was an oversight at our end. Two hubs were sharing the same postgres db 😕 so one hub cleaned up the other’s user pods and this error was a result of that. Thanks for your help! 🙂 @consideRatio@minrk