Hello jupyter community, I need your help
I am trying to connect kerberized Hortonworks hadoop clusters livy server with jupyter and I have 401 error when connecting
Is it possible to connect sparkmagic to kerberized spark livy server? if it is then I think I have some misconfiguration in the sparkmagic config json. username and password is the technical user which runs server and have right of impersonation in the hadoop cluster(proxy user) not actual username when I login in the jupyterhub
it my config.json in the sparkmagic:
{
“kernel_python_credentials” : {
“username”: “user”,
“password”: “pass”,
“url”: “ht tp://livy.server:8999”,
“auth”: “Kerberos”
},“kernel_scala_credentials” : {
“username”: “user”,
“password”: “pass”,
“url”: “ht tp://livy.server:8999”,
“auth”: “Kerberos”
},
“kernel_r_credentials”: {
“username”: “user”,
“password”: “pass”,
“url”: “ht tp://livy.server:8999”
},“logging_config”: {
“version”: 1,
“formatters”: {
“magicsFormatter”: {
“format”: “%(asctime)s\t%(levelname)s\t%(message)s”,
“datefmt”: “”
}
},
“handlers”: {
“magicsHandler”: {
“class”: “hdijupyterutils.filehandler.MagicsFileHandler”,
“formatter”: “magicsFormatter”,
“home_path”: “~/.sparkmagic”
}
},
“loggers”: {
“magicsLogger”: {
“handlers”: [“magicsHandler”],
“level”: “DEBUG”,
“propagate”: 0
}
}
},“wait_for_idle_timeout_seconds”: 15,
“livy_session_startup_timeout_seconds”: 600,“fatal_error_suggestion”: “The code failed because of a fatal error:\n\t{}.\n\nSome things to try:\na) Make sure Spark has enough available resources for Jupyter to create a Spark context.\nb) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.\nc) Restart the kernel.”,
“ignore_ssl_errors”: true,
“session_configs”: {
“driverMemory”: “2048M”,
“executorCores”: 2,
“proxyUser”: “bdjupyter”,
“conf”: { “spark.master”: “yarn-cluster” }
},“use_auto_viz”: true,
“coerce_dataframe”: true,
“max_results_sql”: 2500,
“pyspark_dataframe_encoding”: “utf-8”,“heartbeat_refresh_seconds”: 30,
“livy_server_heartbeat_timeout_seconds”: 0,
“heartbeat_retry_seconds”: 10,“server_extension_default_kernel_name”: “pysparkkernel”,
“custom_headers”: {},“retry_policy”: “configurable”,
“retry_seconds_to_sleep_list”: [0.2, 0.5, 1, 3, 5],
“configurable_retry_policy_max_retries”: 8
}
this is error message when I try some “hello world” in the spark or pyspark notebook or shell in the jupyter:
print(“Hello World”)
The code failed because of a fatal error: Invalid status code ‘401’ from ht t p:/ /livy.server:8999/sessions with error payload: Error 401
HTTP ERROR: 401
Problem accessing /sessions. Reason:
Authentication required
Powered by Jetty:// 9.3.24.v20180605.
Some things to try: a) Make sure Spark has enough available resources for Jupyter to create a Spark context. b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly. c) Restart the kernel.
plz help if you have any idea
thnx