Can anyone help to answer a simple technical issue I have ? I am writing this query in a readable nice format in Jupyter Note book, But it will not work unless I write the whole statement in one big ugly single line !! I can not find anything in the setup to fix this , any help ?
Here is a very simple example, just imagine the only way this work in the cell is to write it in an unreadable single line:
spark.sql(“select concat(firstName,’ ',lastName) as fullName
,case when a.income > b.avg_income then ‘above_avg’ else ‘below_avg’ end as hi_income
from jsql as a
(select dep, mean(income) as avg_income from jsql group by 1 order by dep) as b on a.dep=b.dep”).show()