I have a repo that opens in mybinder.org without problems, but when stepping through some notebooks I come across a PyTorch problem that I believe would be solved by running ‘docker run’ with the --ipc=“host” flag.
Is there any way to encourage mybinder to use that flag when it runs its equivalent of ‘docker run’?
I may have misunderstood docker or repo2docker or many other components of this as I am new to some of these things. So thank you for your patience!
Dan
The short answer is that as a repo author you can’t.
What does the IPC flag do and why does PyTorch care? From a quick read of the man page it seems to change how you can do IPC to make it more performant but it should work even at the default setting?
Do you have some pointers to material discussing the (side) effects of enabling this for a kubernetes cluster in terms of performance, security and if it is even possible on GKE?
Thank you for your quick reply - I really appreciate it.
This is the kind of error I’m seeing, which can supposedly be solved through ipc=host or shm-size flags in Docker. (My repo is here.)
Regarding the side effects of enabling these flags on GKE, or if that’s even possible, I’m afraid I’m out of my depth there if I wasn’t already…
No problems if this is pushing the boundaries, but great if anyone knows how to proceed!
Dan
Unfortunately running containers with --ipc=host is not secure, and we can not increase shared memory on a per container basis. This wouldn’t work on mybinder.org - we could do it for all users on mybinder.org, but I think that also has some serious implications we will have to account for.
If you have a JupyterHub or run your own BinderHub and want to work around this issue for pytorch users, see https://github.com/kubernetes/kubernetes/issues/28272 for a workaround.