After login in JupyterHub, I get the error “Forbidden. You don’t have permission to access this resource.”
The hub is running from a Docker and I have no modifications to the configuration file, i.e. I am using the default configuration.
The server is available in two ways, via a public IP and via a website with a gateway configured to reach that IP.
If I access the hub by IP, everything works fine.
I only get this error when I access it from the website and after logging in. However, after deleting the cookies, I can access the login page again.
To configure the gateway, I followed the instructions in the official documentation.
I would like to point out that in the JupyterHub output, there is no indication of the error.
After running this command docker logs -f --details jupyterhub I get
(...)
User logged in: admin
302 POST /hub/login?next=%2Fhub%2F -> /hub/
Because deleting the cookies made the site work again, I initially thought it was a problem with JupyterHub and not Apache.
However, the error message did not have the formatting of the error messages in JuyterHub and there was no indication of an error in the JupyterHub output. So, the issue was probably with Apache.
Investigating the Apache log file, I found that the ModSecurity module was blocking access to the hub after logging in. ModSecurity thinks the request of some cookie is an SQL Injection attack.
By the way, ModSecurity is an open-source web application firewall (WAF) module for Apache HTTP Server. It provides a way to enhance the security of web applications by monitoring and analyzing HTTP traffic in real-time.
Below you can find the error message in the Apache log file:
ModSecurity: Access denied with code 403 (phase 2).
Pattern match "([\\\\~\\\\!\\\\@\\\\#\\\\$\\\\%\\\\^\\\\&\\\\*\\\\(\\\\)\\\\-\\\\+\\\\=\\\\{\\\\}\\\\[\\\\]\\\\|\\\\:\\\\;\\"\\\\'\\\\\\xc2\\xb4\\\\\\xe2\\x80\\x99\\\\\\xe2\\x80\\x98\\\\`\\\\<\\\\>].*?){8,}"
at REQUEST_COOKIES:jupyterhub-hub-login.
[file "/usr/local/apache/modsecurity-owasp-old/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"]
[line "157"] [id "981172"] [rev "2"]
[msg "Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded"]
[data "Matched Data: - found within REQUEST_COOKIES:jupyterhub-hub-login: \\x222|1:0|10:1689510357|20:jupyterhub-hub-login|44:ZGM1ODNjYWNkZmE4NDcxMWFjNzg0MDU1MzUwZjFkMDE=|2eef798f2f1f98778b4402384fd4f8a43dff65d47345925db73767ea84460d0e\\x22"]
[ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"]
[tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"]
[hostname "jupyter.(...).com"] [uri "/hub/"]
[unique_id "ZLPh1YrbTlIO_BE7gHFCQAAAAMs"], referer: https://jupyter.(...).com/hub/login?next=%2Fhub%2F
Fortunately, it is possible to disable ModSecurity per host.
<IfModule mod_security2.c>
SecRuleEngine Off
</IfModule>
Perhaps this issue should be considered a bug, since it is good practice to have ModSecurity enabled on the Apache server.
The Core Rule Set comments acknowledge that this rule is expected to have a large false-positive rate, and will need to be disabled for specific cookies:
# Expect a lot of false positives with this rule.
# The most likely false positive instances will be complex session ids.
# This will make it necessary to disable the rule for certain known cookies.
# The following directive is an example to switch off the rule globally for
# the cookie foo_id. Place this instruction in your configuration after
# the include directive for the Core Rules Set.
#
# SecRuleUpdateTargetById 942420 "!REQUEST_COOKIES:foo_id"
#
There are also CRS docs on tuning false positives. So you don’t need to disable mod_security to fix this, you need to allow specific cookies through (or disable this one rule altogether, since it seems a pretty silly one):