Hello I have jupyterhub installed on k8s. If i look at the pods logs i see only
[I 2023-09-26 10:18:53.530 SingleUserLabApp log:186] 200 GET /user/mcostach/api/terminals?1695723533496 (mcostach@10.193.12.43) 1.36ms
[I 2023-09-26 10:18:53.848 SingleUserLabApp log:186] 200 GET /user/mcostach/api/metrics/v1?1695723533822 (mcostach@10.193.12.43) 1.76ms
[I 2023-09-26 10:18:58.864 SingleUserLabApp log:186] 200 GET /user/mcostach/api/metrics/v1?1695723538841 (mcostach@10.193.12.43) 1.75ms
My security team is concerned that they cannot see the user activity inside the pod. Is there a way to do this for example to see what cammands were executed from kernel or what notes were triggered?
The logs you posted are the logs from JupyterHub. So, if you look into the logs of the pods where single user servers are running, you will have more details on what users are doing with their servers.
Fair enough, you are right! So, if you need more verbose logs, you can enable debug logging. I dont think you can log what commands were executed from kernel or terminal.
As @manics says, you will need to do this at the operating system level, rather than at the Jupyter level. Even if Jupyter Server (or similar) logged it, it would not be suitable for auditing purposes, as the end user may as well modify the code inside to emit false info or redact info. Doing it at the os level (while not allowing end users the ability to get root) is the only way to get audit-quality information.
On kubernetes, Falco seems like something that should do what you need. You can set it up and configure it to emit auditing events to a secure output sink (probably whatever you are already using for logging). This can also assure you that auditing events actually came from Falco and not end users / attackers trying to fake it.
Sorry to not be clear – I added auditctl to my jupyter docker image. It logs commands just fine if I start the container with docker run. But when I use Jupyterhub (swarm spawner) to start the container, or if I use docker service create to start the container, then there is no logging of commands. I wonder what the difference between using docker and docker swarm is.
Yes, that works – but it doesn’t give any information on which container the command came from (as far as I can tell). I thought that if I could have each container logging the commands, then I could know who did what.
It’s just weird that I get different behavior from running the container vs starting it as a service in a swarm. I wonder if there’s some difference in the logging between the two methods.
Just to follow up: Sysdig gives tons of info (including terminal cli execution), and it does specify container names. I am still figuring out how to continuously log and monitor that data though.
I assumed that it would log the shell commands from within a notebook (! commands), but it doesn’t. What does the notebook use to execute them? (I’ve tried clone and execv).
Jupyter server uses jupyter_server_terminal extension to provide terminal support in JupyterLab and notebook. This extension is based on terminado based on web sockets. I dont know all the internals of these components but I doubt you can log commands executed via notebook or JupyterLab interfaces.