Not able to get User logs below the notebook cells when a spark session is created using sparkmagics

JupyterLab
,
1

Hi,
I am unable to see the user defined log statements in the same cell where spark session (spark notebook is created) by using the sparkmagics pip package (initialising it as “%%spark”).
Even after adding the logger handlers explicitly, I am able to see that “streamhandler” is added to the logger, but I am unable to see the log statements.

More Context: Sparkmagic uses Livy, a REST server for Spark, to remotely execute all user code . The library then automatically collects the output of your code as plain text or a JSON document, displaying the results to you as formatted text or as a Pandas dataframe as appropriate.

Does Livy server provide this feature of logging user statements on jupyterlab?

jupyterspark issue
jupyterspark issue1052×368 29.3 KB

2

You’ve tagged this with JupyterHub but this may be a problem with your singleuser server, e.g. JupyterLab. If you add more information it’s more likely that someone will be able to help.

3

Thanks. I have retagged this query to Jupyterlab, and have added more context.

1 Like