Kernel keeps using outdated environment variables

Hi,
I am currently encountering an issue with environment variables.
So I had to reinstall CUDA and had to update from 11.0 to 11.1.
I changed my PATH and LD_LIBRARY_PATH variables accordingly (replacing /usr/local/cuda-11.0/bin with /usr/local/cuda-11.1/bin in PATH and /usr/local/cuda-11.0/lib64 with /usr/local/cuda-11.1/lib64 in LD_LIBRARY_PATH). My python and ipython import tensorflow correctly, and I get a success message regarding GPU usage.
However, in jupyter, when importing tensorflow in a notebook, I get the following messages

W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-11.0/lib64

And when I check the environment variables within my notebook, I see that the old values were kept (i.e. the python kernel is seeking the /usr/local/cuda-11.0 folder which does not exist anymore).
As a matter of proof, when I copied the /usr/local/cuda-11.1 folder and named it /usr/local/cuda-11.0, my notebook imported tensorflow correctly.
I do not know how to force my python kernel to update the environment variables values (changing them within the notebook didn’t help).
Thank you in advance!

Hi @wesbz, welcome to the community!

Here are a couple of things to check…

  1. Did you restart your notebook server from a console that has the updated environment? The kernel processes will typically inherit their env from their parent process (notebook server).
  2. Although unlikely, these variables can be specified in the kernel’s kernel.json file and specify the older values - so that might be worth checking if #1 isn’t a solution. You can use jupyter kernelspec list to identify the kernel and its directory, then check that directory’s kernel.json file. These would be specified in an env stanza although, again, these files don’t typically contain this kind of specific information.

Let us know how that goes. I’m sure someone will have some further ideas if necessary.

Thanks.

1 Like

Hi! Thanks a lot for your response!
I indeed restarted the notebook server before considering asking here :wink:
Your second guess was the deal, actually! As unlikely as it may sound!
I corrected the paths in the json file, however, I feel like this is a dangerous thing to keep. May I delete the whole env section and hope for the kernel to probe the variables from the shell?
Thanks again for your time and expertise! :+1:

Right on - glad you’re moving forward.

I find it interesting that your kernel.json contained those values which implies that, had they not been there, the kernel would not have been properly provisioned. You could certainly try removing them altogether to see if that works.

The kernel’s environment should be seeded from the server’s if nothing is there. However, if there are values you do need or find their removal didn’t work, you can replace the values you want to inherit with their templated form. For example, you could have "PATH": "${PATH}" and "LD_LIBRARY_PATH": "${LD_LIBRARY_PATH}" as described in the kernelspecs section of the docs. That should result in the inheritance of those templated values from that of the server’s.

Thank you! This sounds like a much safer way of specifying env variables in the JSON file. I removed them and it works fine!
Thanks again for your help!

1 Like