Hi,
I am setting KERNEL_WORKING_DIR in the kernel via Enterprise Gateway running on kubernetes. While I am able to do so successfully, I am not able to make the directory writeable to the user running the kernel pod. It is writeable only for root. I do not have any external volume mapped to the kernel working directory path. Also I am passing KERNEL_USERNAME env variable to the kernel with the value same as the user running the kernel.
Any idea how I could change the ownership of KERNEL_WORKING_DIR path to the non root user?
Thanks in advance!
You can start the pod using UID and GID attributes but KERNEL_WORKING_DIR really has no benefit unless its using a mount. Also, it would be good to know if this is a factory-built image or something you’ve created/extended. No images we provide should be running as root. However, if you are using the UID/GID attributes (conveyed via the envs KERNEL_UID and KERNEL_GID), your pod will run as root unless the RunAsGroup feature gate is enabled. (Hmm. Unfortunately, it looks as though this particular feature gate was removed following > 1.22.)
Please open an issue or discussion item in enterprise gateway and we can try to understand and work through this.
It is an image that we built. I will do a quick comparison of the behavior by running the official build. In this case the kernel container is not running as root. It is running as a custom user but the directory that the pod/EG creates in the container using the path provided in KERNEL_WORKING_DIR has root as the owner and the custom user does not have access to write to it.
EG does not create the KERNEL_WORKING_DIR. It merely sets the container’s working directory to that value and only when EG_MIRROR_WORKING_DIRS is "true". As a result, the directory referenced by KERNEL_WORKING_DIR needs to exist - either within the container or via a volume mount. That said, the directory referenced by KERNEL_WORKING_DIR may already exist (internally or via a mount) and be owned by root, but we’re not creating a directory using that value.
I understand how the kernel working directory is being used through the kernel pod config. In my case directory referenced by KERNEL_WORKING_DIR is a new non existent directory. Even when I use the factory built elyra/kernel-py:3.0.0 image the newly created directory is owned by root and therefore the current working directory is not writeable by the user (jovyan)
The way I got around this issue is by modifying the kernel pod yaml to:
- Adding an init container that uses the same kernel working directory as a mount path to an emptyDir type volume
- Have the kernel container also use the same emptyDir type volume as a mount path mapped to the working directory
- Have the initcontainer run as a root user to execute a command to change ownership of the kernel working directory
Please open an issue. At a minimum we should adjust the permissions in the image.
Hmm - you might try EG 3.1.0, although I don’t think there were any changes in this particular are.
What created your workspace and/or my-dynamic-directory?
Here’s the output of various commands using the factory 3.1.0 kernel-py image:
Jupyter lab requested this directory to be created via env var KERNEL_WORKING_DIR
export KERNEL_WORKING_DIR=/home/jovyan/workspace/my-dynamic-directory
jupyter lab --debug --gateway-url=http://127.0.0.1:53392
Jupyter lab requested this directory to be created via env var
KERNEL_WORKING_DIR
This isn’t a correct statement. Lab knows nothing about this environment variable. That said, I see what is going on.
When a docker container (and this applies to a k8s pod apparently) specifies a working directory that does not exist, it is created on the fly and prior to the switch of the user in which the container should be running as - as evident in this example:
$ docker run -it --rm -w /home/jovyan/workspace/my-dynamic-directory elyra/kernel-py:3.0.0 ls -ld
drwxr-sr-x 2 root root 4096 Dec 8 20:17 .
See WORKDIR creates directories as root instead of USER · Issue #36677 · moby/moby · GitHub
There might be a way to resolve this using DOCKER_BUILDKIT=1 which I’d be happy to look into. Please open an issue.
You can workaround this by ensuring the directory is properly provisioned within the image prior to its use.
1 Like
Thanks for your response @kevin-bates Really appreciate the detailed information and additional insights into the issue. I do understand that it is more of a kubernetes or docker behavior than a Enterprise Gateway one
I am sorry that I was not clear enough in my earlier response.
A little bit about our use case:
There is a set of users who use jupyter lab. They request remote kernels from jupyter lab. As users they have the freedom to use any KERNEL_WORKING_DIR path they wish to and they expect that directory to be writeable.
I am happy with my init container approach that I was able to come up with. So from my perspective I would mark this issue as resolved. Thanks!
1 Like