How to Start Jupyter Automatically over SSH without Sudoer Privilege

General
,
#1

Using jupyter notebook version 6.3.0 with python 3.6.9 on an Azure VM running Ubuntu 18.04.

My desktop is running Linux Mint Ulyana (basically ubuntu 20.04)

I am trying to run the jupyter server automatically on every start up of the azure vm, and tunnel the port 8888 of the VM to my local port 8081. I am trying the following script.

#!/usr/bin/env bash
# encoding:utf-8

# Start the virtual machine
az vm start --resource-group mcg_corp --name ml_dev
SERVER=59.179.93.26

# Start the jupyter service

# Option 1
# ssh "$USER@$SERVER" bash << EOF
# nohup jupyter-notebook --no-browser --port 8888 > /dev/null 2>&1 &
# exit
# EOF

# Option 2
ssh -t "$USER@$SERVER" 'nohup jupyter-notebook --no-browser --port 8888 > /dev/null 2>&1 &'

# Tunnel the jupyter service
nohup ssh -N -L localhost:8081:localhost:8888 $SERVER & #8081:Local port 8888:remote port


# Mount the home directory of the remote locally
sshfs -o allow_other $SERVER:$HOME dev

The username $USER is same in local and the remote. The problem is the remote machine is starting jupyter notebook with the root privilege. When I look up the running processes by
ps -ef|grep jupyter, the command I find running is

/usr/bin/python3 /usr/bin/jupyter-notebook --no-browser --port 8888

But if I log in manually with ssh and start jupyter notebook in the server, this is the command running.

/usr/bin/python3 /home/$USER/.local/bin/jupyter-notebook --no-browser

So how to make sure that jupyter notebook starts with my user privilege in my home directory in the remote? If that happens, then I believe I can get the rest of the flow working correctly.

I am in the sudoer list in the remote as well, but I do not want to run jupyter from root.

#2

That’s very bizarre that it would run as a different user. What do you get from

ssh -t "$USER@$SERVER" id

in your script? You’re not running your script locally with sudo?

#3

Thanks for getting back. When I run your command, this is what I am getting back in return.

uid=1000($USER) gid=1000($USER) groups=1000(barmanroys),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(lxd),114(netdev),999(docker)
Connection to 59.179.93.26 closed.

And no, I am not running my local script with sudo privilege either.

Fortunately, after some more experimentations, I am able to run jupyter in the remote by specifying the path

# Start the jupyter service
ssh "$USER@$SERVER" bash << EOF
nohup $HOME/.local/bin/jupyter-notebook --no-browser --port 8888 > /dev/null 2>&1 &
EOF

and able to connect. So problem solved. But still wondering why the command jupyter-notebook will start by invoking the root executable instead of the executable in my home directory.

Some more trial and error also revealed that when I run the jupyter-notebook over automated SSH without specifying path, it invokes the /usr/bin/jupyter-notebook but actually the process is still under my username and running with non-sudo privilege.

I used to believe user space applications (such as jupyter, firefox or thunderbird) have only one executable which runs with the privilege of whichever user invokes it. Was not aware that jupyter can have an executable for each user. Likely, I have to fill some gap in my understanding of PATH and executables. Thanks a lot for your support here though.

#4

I wonder that, too! It’s truly wild. The notebook process (which is just python3 in the end) has no ability to do that itself, so it must be something in the surrounding environment before python really starts. Maybe in nohup? I just can’t think of what. It does seem important that invoking bash runs as the requested user, but invoking other commands directly does not.

You might try ssh -it "$USER@$SERVER" -- python3 -i and poke around, but I’m mostly guessing at this point. Or even ssh "$USER@$SERVER" nohup id.

I used to believe user space applications (such as jupyter, firefox or thunderbird) have only one executable which runs with the privilege of whichever user invokes it.

Unless setuid is set on the executable (which is rare), that’s true.

Was not aware that jupyter can have an executable for each user.

Jupyter doesn’t itself ever install more than one executable, but every user certainly has permission to do pip install --user notebook or install in as many envs as they want, just like any other Python package (numpy, etc.). So which -a jupyter can indeed reveal multiple installations, but each one does correspond to a separate, explicit installation by a user.

#5

Thanks again. I opened the interactive Python REPL by

ssh -it "$USER@$SERVER" -- python3 -i

and tried these

>>> import os 
>>> os.environ['USER']
'myusername'
>>> os.environ['PATH']  
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'

Note the PATH.

But when I get into the server by SSH and invoke Python REPL from bash, the result is as expected.

>>> import os
>>> os.environ['USER']
'myusername'
>>> os.environ['PATH']
'/home/myusername/.local/bin:/home/myusername/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'

That explains the cause at least one level deeper than jupyter, or may be even deeper than python itself, i.e. something is setting the PATH as that of the root in the first instance.

Anyway, my problem is solved, so this is just out of academic interest, hoping to eventually tease out whatever underlying mechanism is causing this mismatch between reality and my (incomplete) mental model of what SSH does. But it may even be something azure specific, or too deep into the weeds of how the openSSH daemon accepts connections, more than my knowledge allows in those areas. Appreciate your inputs here, but don’t want to take up your time on this unless you are curious.

#6

The PATH difference is a common frustration with launching commands over ssh - if you launch an interactive bash session, it’ll do source ~/.bashrc (or bash_profile if it’s a login shell - bash does this, not ssh itself) which loads your usual environment, but if you do ssh HOST -- other-command it starts with a very bare-bones environment - none of your shell initialization is invoked, because no shell is started. ssh $host -- bash -i -l -c "actual command" tells bash to launch a full interactive login shell before invoking actual command