Schedule Jhub notebook using papermill

JupyterHub
#1

Hi there,

We are trying to see the stability of the Jhub by frequently running some hive queries during throughout the day. May be every two hours. This is something like a health check.
I am able to run a Jhub not book using papermill. I have one doubt, at the moment I copy my healthcheck.ipynb file from /home/myid/ to /home/admin because I need to run few queries that have few tables and I don’t have access to those tables so I must run as admin. My question is if I create a ipynb from Jhub and then move that file to admin location and then run that using papermill will it really capture the current jupyterhub status. For instance if Jhub web app is down then when I run it from admin location using papermill will, it throw error or not. I want it should capture error so that I know the health check is failed due to web app is down.

Please clarify.

Thanks,
Ilango

#2

Hi! I’m not familiar with papermill, but I’m assuming you’ve configured it to launch your notebook as a particular JupyterHub user? This is probably stating the obvious, but the results of the notebook will depend on whether you’re making JupyterHub API calls that differ between admin and non-admin users, or accessing locations that are only accessible to particular users.

If you share your notebook we might be able to provide more help.

#3

@manics My question is not really clear. Let me rephrase it. When I run the Jupyterhub notebook ipynb file using papermill in shell script, what happens when Jupyterhub web is down. Will it throw error?
I want this shell script also to throw error.

My requirement is to automate a jupyterhub notebook that has pyspark code to run a hive select query to check the stability of jupyterhub and also hive connectivity.
At the moment I created this notebook and running successfully. My question is what happens when the jupyterhub web app is just down and hive is up and running. Will the papermill run also throw error? Please clarify

#4

Can I just check, by “Jupyterhub notebook” do you mean a notebook running in JupyterHub, or did you actually mean a notebook running in JupyterLab/notebook?

#5

@manics , It is notebook running in Jupyterhub.

#6

In this situation where are you executing the shell script? In your singleuser server?

#7

Yes @manics , The Jupyterhub is installed in edge node and I am executing my shell script from the same machine.