Need Help with JupyterHub on Kubernetes - Issue with TensorFlow Image

[Douglas]

Hello Jupyter Forum Community,

I hope this message finds you well. I’m reaching out for assistance as I am relatively new to JupyterHub on Kubernetes and have encountered an issue while trying to set up the TensorFlow Image.

I’ve been following the “Zero to JupyterHub with Kubernetes” tutorial available at https://z2jh.jupyter.org/en/stable/, and I have successfully completed the following steps:

1. Installed Helm v3
2. Configured the Kubernetes cluster (self-hosted)
3. Installed JupyterHub
4. Configured different profiles for Data Science, Spark, TensorFlow, etc.

However, I’ve run into a problem specifically with the TensorFlow Image. Whenever I try to import the TensorFlow library, I consistently encounter an error stating “Kernel Restarting.”

After analyzing the log I found the following message:

Debugger warning: It seems that frozen modules are being used, which may make the debugger miss breakpoints. Please pass -Xfrozen_modules=off to python to disable frozen modules. Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

I have attached screenshots to this ticket to provide a visual representation of the issue, hoping that it will help in understanding and diagnosing the problem.

I would greatly appreciate it if someone could guide me in the right direction to resolve this TensorFlow Image issue. Any insights or suggestions would be immensely helpful.

Thank you in advance for your time and assistance.

Best regards,
Douglas

[mahendrapaipuri]

Could you share the logs of your single user server pod? Probably there is something wrong with your environment. You can open a terminal, start a Python session and try to import tensorflow to check if tensorflow is properly installed.

[Douglas]

Hello Mahendra,

I appreciate your guidance. Following your instructions led me to encounter the following error message:

jovyan@jupyter-douglas:~$ python
Python 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Illegal instruction (core dumped)
jovyan@jupyter-douglas:~$

This error indicates a compatibility or architecture issue. Upon further investigation into the architecture problem, I discovered that TensorFlow requires, among other things, the AVX (Advanced Vector Extension) instruction set, which is turned off by default in Proxmox 8.

The problem was resolved by changing the CPU type to host.

Thank you for your assistance!