I am having a issue scheduling a note book which was built on scala , can some on help me what should I do ?
Hi! Could you provide more details on what you’re trying to achieve? Have a read though Getting good answers to your questions to find out what kind of information is useful when asking for help.
@manics Thank you for the reply, I have created a note book in splon kernel with below info
import org.apache.spark.SparkContext
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.hive.HiveSessionStateBuilder
/*val conf = new SparkConf().set(“spark.sql.catalogImplementation”,“hive”)
val spark = SparkSession.builder()
.master("local[]")
.config(conf)
.enableHiveSupport()
.getOrCreate()/
import java.io.File
import org.apache.spark.sql.{Row, SaveMode, SparkSession}
val warehouseLocation = new File(“spark-warehouse”).getAbsolutePath
val conf = new SparkConf()
.set(“spark.sql.warehouse.dir”, “hdfs://namenode/sql/metadata/hive”)
.set(“spark.sql.catalogImplementation”,“hive”)
.setMaster(“local[*]”)
.setAppName(“Hive Example”)
val spark = SparkSession.builder()
.config(conf)
.enableHiveSupport()
.getOrCreate()
val sqlContext = SQLContext.getOrCreate(SparkContext.getOrCreate())
import sqlContext.implicits._
val driverClass = “com.mysql.jdbc.Driver”
val connectionProperties = new java.util.Properties()
connectionProperties.setProperty(“Driver”, “com.mysql.jdbc.Driver”)
val uri = “jdbc:mysql://com:3306/dbname??useSSL=false”
val query = s"""(select * from tab1 as em, tab2 u
where em.col1 = u.col2
) as tempTable"""
print(query)
It is running fine when I am running the note book manually
here is the out put
But when I am running this in jupter scheduler its giving me this error
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.$anonfun$apply$4(rules.scala:462)
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.$anonfun$apply$4$adapted(rules.scala:460)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:173)
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:460)
at org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:458)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$46(CheckAnalysis.scala:699)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$46$adapted(CheckAnalysis.scala:699)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:699)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:90)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:155)
at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:176)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:63)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
… 42 elided
org.apache.spark.sql.AnalysisException: Hive support is required to CREATE Hive TABLE (AS SELECT);
I am using this syntax to setup the cronschedule
/home/ubuntu/anaconda3/bin/jupyter nbconvert --ExecutePreprocessor.timeout=600 --ExecutePreprocessor.kernel_name=spylon-kernel --to notebook --execute /backup/script.ipynb
I’ve changed the category from JupyterHub to Notebook/nbconvert so it’s more likely to be seen by someone who can help.
Sure , Thank you , resolving this issue is very important for me , I have tried all options seems jupyter lab scheduler is causing this issue