Severs Being Culled With Open Terminal - How Is Terminal Activity Determined?

Hi, I am trying to implement server shutdowns and culling using the NotebookApp.shutdown_no_activity_timeout, MappingKernelManager.cull_idle_timeout and TerminalManager.cull_inactive_timeout options.

Kernel management is behaving as I expected, if a kernel is open the server will not be shutdown until all are culled according to the given timeout.

The problem I am having is that open terminals are having no effect on the activity timeout and the server is being shutdown despite having a terminal open. Similarly with the terminal manager, if I start a long running command that gets for longer than the timeout value the terminal is culled before the command completes.

I can see in logs that whenever I execute a new command in the terminal it resets the last activity time/counter but am I missing something in regards to open terminals or long running commands?

We tried something similar for culling and we noticed that if there is no interaction with terminal, the terminal will cull itself. So, if there is a long running command, it will be killed.

Ok thanks, I’m glad to hear that what I am seeing is normal. Do you know if there is a way, possibly through other configuration options, to not cull open terminals?

Unfortunately, I dont see an easy way to get it working for terminals. Maybe core developers of jupyter server may have more suggestions. We ended up advising our users to run long running terminal commands using a multiplexer like screen, tmux, etc.

EDIT: this is kind of off topic i realized, because the jupyterhub-idle-culler isn’t involved, but it includes info on the jupyter server side as well.

The readme of jupyterhub-idle-culler provides some insights on how the system works as a whole currently.

GitHub - jupyterhub/jupyterhub-idle-culler: JupyterHub service to cull idle servers and users

@consideRatio Thanks, that does provide a great description and led me to this link which as far as I can tell confirms that the the activity is only ever updated when text is written to the terminal screen either by the user or from the command itself.

@mahendrapaipuri I am not very familiar with multiplexers, I tried using screen but the terminal still gets culled. Would you be able to give some more detail on how you are doing this please?

1 Like

Sorry, I wasnt very clear in my comment. Multiplexers will not stop terminal from culling. Terminals will cull if there is no user interaction with it.

However, if we execute a long running command within a multiplexer like screen session, even if the terminal has been culled, the command will keep executing within the screen session. Users can open a new terminal at any moment and access the existing screen sessions. This way the workflows will not be disrupted due to terminals being culled. Does it make sense?

2 Likes

All good, that makes perfect sense. Thanks for your help.

1 Like