Hello I have jupyterhub installed on k8s. If i look at the pods logs i see only
[I 2023-09-26 10:18:53.530 SingleUserLabApp log:186] 200 GET /user/mcostach/api/terminals?1695723533496 (email@example.com) 1.36ms
[I 2023-09-26 10:18:53.848 SingleUserLabApp log:186] 200 GET /user/mcostach/api/metrics/v1?1695723533822 (firstname.lastname@example.org) 1.76ms
[I 2023-09-26 10:18:58.864 SingleUserLabApp log:186] 200 GET /user/mcostach/api/metrics/v1?1695723538841 (email@example.com) 1.75ms
My security team is concerned that they cannot see the user activity inside the pod. Is there a way to do this for example to see what cammands were executed from kernel or what notes were triggered?
The logs you posted are the logs from JupyterHub. So, if you look into the logs of the pods where single user servers are running, you will have more details on what users are doing with their servers.
Thank you for reply, those were in fact from the singleuser pod
kubectl -n jupyter logs jupyter-mcostach | tail
[I 2023-09-26 11:55:29.439 SingleUserLabApp kernelmanager:318] Starting buffering for 81daf51c-13d7-469a-a884-aeea2dfc231c:281d1c08-2f58-4b7a-83c9-76f19f7e03ab
[I 2023-09-26 11:55:29.465 SingleUserLabApp log:186] 101 GET /user/mcostach/api/kernels/81daf51c-13d7-469a-a884-aeea2dfc231c/channels?session_id=e9feebf6-a696-4d4a-8187-07a4cb238ddb (firstname.lastname@example.org) 2.29ms
[I 2023-09-26 11:55:29.471 SingleUserLabApp log:186] 200 GET /user/mcostach/api/sessions?1695729329435 (email@example.com) 2.40ms
[I 2023-09-26 11:55:29.471 SingleUserLabApp log:186] 200 GET /user/mcostach/api/kernels?1695729329445 (firstname.lastname@example.org) 2.08ms
[I 2023-09-26 11:55:29.511 SingleUserLabApp log:186] 101 GET /user/mcostach/api/kernels/81daf51c-13d7-469a-a884-aeea2dfc231c/channels?session_id=62bec626-aad3-4949-896a-4d8944b743d5 (email@example.com) 3.00ms
[I 2023-09-26 11:55:33.018 SingleUserLabApp log:186] 204 PUT /user/mcostach/lab/api/workspaces/default?1695729332990 (firstname.lastname@example.org) 2.08ms
[I 2023-09-26 11:55:38.320 SingleUserLabApp log:186] 200 GET /user/mcostach/api/terminals?1695729338295 (email@example.com) 1.68ms
[I 2023-09-26 11:55:38.377 SingleUserLabApp log:186] 200 GET /user/mcostach/api/contents?content=1&1695729338354 (firstname.lastname@example.org) 2.21ms
[I 2023-09-26 11:55:39.498 SingleUserLabApp log:186] 200 GET /user/mcostach/api/sessions?1695729339474 (email@example.com) 2.03ms
[I 2023-09-26 11:55:39.501 SingleUserLabApp log:186] 200 GET /user/mcostach/api/kernels?1695729339475 (firstname.lastname@example.org) 1.54ms
Fair enough, you are right! So, if you need more verbose logs, you can enable debug logging. I dont think you can log what commands were executed from kernel or terminal.
Yeah that’s what my security team wants …
I don’t know how to do it in Jupyter or Kubernetes, but at the operating system level there’s support for logging executed commands and arguments:
I gave this a try, and the logging works fine if you run commands in the bash shell:
docker run -it jupyterhub/singleuser /bin/bash
But it doesn’t log anything from the jupyter terminal. There must be something else to log instead of execve?
As @manics says, you will need to do this at the operating system level, rather than at the Jupyter level. Even if Jupyter Server (or similar) logged it, it would not be suitable for auditing purposes, as the end user may as well modify the code inside to emit false info or redact info. Doing it at the os level (while not allowing end users the ability to get root) is the only way to get audit-quality information.
On kubernetes, Falco seems like something that should do what you need. You can set it up and configure it to emit auditing events to a secure output sink (probably whatever you are already using for logging). This can also assure you that auditing events actually came from Falco and not end users / attackers trying to fake it.
Sorry to not be clear – I added auditctl to my jupyter docker image. It logs commands just fine if I start the container with docker run. But when I use Jupyterhub (swarm spawner) to start the container, or if I use docker service create to start the container, then there is no logging of commands. I wonder what the difference between using docker and docker swarm is.
Have you tried configuring auditd on the host operating system, and looking at the host audit logs?
Yes, that works – but it doesn’t give any information on which container the command came from (as far as I can tell). I thought that if I could have each container logging the commands, then I could know who did what.
It’s just weird that I get different behavior from running the container vs starting it as a service in a swarm. I wonder if there’s some difference in the logging between the two methods.
There are other tools you can try, for instance I just found Monitoring Command Execution In Containers With Sysdig - keefer.io which claims to have good support for containers.
Thanks! I’ll give it a try.
Just to follow up: Sysdig gives tons of info (including terminal cli execution), and it does specify container names. I am still figuring out how to continuously log and monitor that data though.
I assumed that it would log the shell commands from within a notebook (! commands), but it doesn’t. What does the notebook use to execute them? (I’ve tried clone and execv).
Jupyter server uses jupyter_server_terminal extension to provide terminal support in JupyterLab and notebook. This extension is based on terminado based on web sockets. I dont know all the internals of these components but I doubt you can log commands executed via notebook or JupyterLab interfaces.