This is a short communication about issues we have found trying to deploy Keras + Tensorflow in a custom Python + R environment/image. Hopefully, it helps others in the community!
RStudio uses the “system” version of OpenSSL. Conda also installs OpenSSL. If you use RStudio to run a
conda-installed package that calls OpenSSL, there is a good chance that it won’t work due to an OpenSSL “mismatch”. This is because RStudio forces the use of a system version of OpenSSL, while
conda expects its own version of OpenSSL. To fix it, either call the function that requires OpenSSL from a Jupyter interface, or separate your
RStudio environments entirely.
Recently, 2i2c received a request to install Tensorflow and Keras in an image containing conda environments along wit several R packages, including RStudio: GitHub - 2i2c-org/utoronto-image: User image for the UToronto Hub.
We also needed to set up the
RETICULATE_PYTHON environment variable so the R packages could properly find the python ones: utoronto-image/Rprofile.site at d422076c3695e44f09a959b14cb10f89b6e5f538 · 2i2c-org/utoronto-image · GitHub
Our users began reporting issues when trying to download example datasets from within RStudio. For example:
Error in py_call_impl(callable, dots$args, dots$keywords) : Exception: URL fetch failure on https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz: None -- unknown url type: https
which suggested some underlying OpenSSL-related issues.
Upon several rounds of debugging sessions, we have found that RStudio seems to load the “system” OpenSSL libraries when it is opened. Our hypothesis for what is happening:
- when RStudio starts the session, it loads the OpenSSL “system” components
- when you try to load a dataset, the TensorFlow + Keras libraries (and underlying packages) try to use this OpenSSL “system” version
- but the conda (python) libraries are (somehow) expecting the
conda-installed OpenSSL version
- This then fails!
To test if this specific mismatch was causing the problem, we tried a symbolic link hack:
mv /opt/conda/lib/libssl.so.1.1 /opt/conda/lib/libssl.so.1.1.backup ln -s /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /opt/conda/lib/libssl.so.1.1
and then, it worked!
This confirmed our suspicion but is likely not a long-term solution because it is probably a very brittle fix that will break in unexpected ways.
After spending a lot of hours on this issue we finally decided to stop trying, look for reasonable workarounds, and post here to disseminate the information we collected.
We have verified the OpenSSL mismatch does NOT happen when you use the Jupyter Notebook application with the R-kernel. So the problem seems to be an RStudio-specific issue when you have multiple co-existing environments (most likely caused by RStudio somehow loading the “system” OpenSSL libraries instead of the conda one). Hence, an immediate workaround is to use a Jupyter interface to download the dataset and then return to RStudio for the rest of your task.
Another alternative would be to create a different image without a conda environment to run your RStudio workflows, so that any python package (including TF or Keras) actually uses the “system” OpenSSL library instead of a conflicting one.
There might be other options involving fixes/enhancements at the RStudio level, but this is outside of our expertise to fix. If others have experience with RStudio and an idea for how to resolve this, please share your ideas!
Hopefully, all this information is useful for future readers!
First, we thought about syncing the OpenSSL versions in both environments (“system” and conda): Try downgrading the openssl version on the conda environment by damianavila · Pull Request #28 · 2i2c-org/utoronto-image · GitHub. But that approach did NOT work!
Then we tried pointing the
LD_LIBRARY_PATH environment variable to conda-specific OpenSSL-related paths (to “force” RStudio to load the expected OpenSSL libraries) but that approach also failed: A different approach trying to make RStudio lo load the proper libssl stuff by damianavila · Pull Request #29 · 2i2c-org/utoronto-image · GitHub and triggered other potential issues!
To check the OpenSSL “system” version being used, you used the
$ ldd /usr/lib/rstudio-server/bin/rserver | grep ssl libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f0413833000)
and the “system” version was
1.1.1f. We also confirmed the installed version with the
dpkg -l command:
$ dpkg -l | grep openssl ii libcurl4-openssl-dev:amd64 7.68.0-1ubuntu2.7 amd64 development files and documentation for libcurl (OpenSSL flavour) ii openssl 1.1.1f-1ubuntu2.12 amd64 Secure Sockets Layer toolkit - cryptographic utility
To check the OpenSSL conda-associated version, we listed the
openssl conda package and also directly checked the version:
$ conda list openssl # packages in environment at /opt/conda: # # Name Version Build Channel openssl 1.1.1l h7f98852_0 conda-forge pyopenssl 19.1.0 py_1 conda-forge $ openssl version OpenSSL 1.1.1l 24 Aug 2021
and the conda main environment had version