Does anyone has succeeded in spawning notebooks via DockerSwarm and using NFS using docker ‘bind’ ?
I have a NFS directory mounted on the docker node
(nosuid,nodev,rw,async,bg,hard,intr,tcp,rsize=65536,wsize=65536,nfsvers=4,acl,sec=sys)
I have setup all the stuff for setuid, setgid, username to map the NFS username (chambon in my test) and bind NFS local directory to container HOME directory
When I launch the notebook with a ‘clean NFS area’ (clean means none of those directories exist : .local/ .ipython/ .ipynb_checkpoints/ .jupyter/ .cache/ .config/ )
It’ OK. I can address the NFS area from within the notebooks, cool.
After that, some of the mentioned above directories are created with 0755 permission as I work with notebook. ok.
But if I stop and start a new notebook, spawn will fail with a message from jupyterhub logs : ‘ERROR:asyncio:Task exception was never retrieved’
As workaround I increase the file permissions to 0777 for those directories (.local/ .ipython/ .ipynb_checkpoints/ .jupyter/ .cache/ .config/ ) and after that it ok. All of the following attempts spawning notebook work. But this wararound can be deployed in production.
Has anyone encountered the same failure ? Any clue ?
I didn’t face this problem yet, but from docker logs, I got errror message 'usermod: Failed to change ownership of the home directory'
so the failure probably comes from start.sh script when running usermod command "usermod -u $NB_UID $NB_USER"
So, in my understanding, when launching a notebook as user ‘demo’ and using a docker bind to NFS area :
a directory /home/demo is created, then usermod is run and at the same time, or in concurrency, the docker bind to NFS area occurs leading to failure with usermod
Not sure the understand correctly, and no idea how to fix this for now
Excerpt from start.sh
# Change UID of NB_USER to NB_UID if it does not match
if [ "$NB_UID" != $(id -u $NB_USER) ] ; then
echo "Set $NB_USER UID to: $NB_UID"
usermod -u $NB_UID $NB_USER
fi
I confirm that the problems I had to face come from the usermod command with HOME bind to NFS area and probably when trying to change file user ID , and here is my workaround to get stuff working.
Since I am interrested only in setting the $NB_UID for $NB_USER, I decided to make this command succeed in all cases by appending || true, I mean : usermod -u $NB_UID $NB_USER || true
I made the same change a few lines below, that is to say : usermod -g $NB_GID -aG 100 $NB_USER || true
Since now, everything is ok, I can use NFS with SwarmSpawner ans it’s a big step forward for me
Feel free to tell me if my workaround is acceptable or not.