How to suppress errors of spark at Jupyterhub notebook level

Notebook
,
1

Hi,

As I’m new to Jupyterhub, I tried to install Jupyterhub on miniconda with a successful outcome. But when tried to launch Jupyter note book and establish the spark session, I’m getting the warnings displayed on the Jupyterhub UI which are form spark. Is there any way to hide them at Jupyterhub UI.

One way i found is - creating the ipython_kernel_config.py under /home/username/.ipython/profile_defautls/

Also I’m unable to copy the same config file to other users home directories at the time of Jupyterhub server bootup.

Could you please help in solving this blocker.

I’m using the below config in jupyterhub_config.py for copying the ipyhton_kernel_config.py

python
import os
import shutil

def copy_ipython_kernel_config(spawner):
# Get the home directory of the user
user_home = spawner.user.home_dir

# Path to the destination .ipython directory
ipython_dir = os.path.join(user_home, '.ipython', 'profile_default')

# Path to the source config file (your custom template)
source_config = '/path/to/templates/ipython_kernel_config.py'

# Ensure the destination directory exists
os.makedirs(ipython_dir, exist_ok=True)

# Copy the config file to the user's .ipython directory
shutil.copy(source_config, os.path.join(ipython_dir, 'ipython_kernel_config.py'))

Register the hook in the config

c.Spawner.pre_spawn_hook = copy_ipython_kernel_config

2

Have you tried adding some logging to your pre_spawn_hook function to verify it’s running as expected, e.g. by printing out the paths? Is the file visible in the user’s home directory?

2 Likes
3

this worked. just by tweaking the code. thank you very for the idea.