Hue Spark Sql with livy connector jars support

#1

Hi,

With regards to this thread: https://gethue.com/blog/quick-task-sql-editor-for-apache-spark-sql-with-livy/ , do you know if we could get Hue Spark SQL Livy connector to work with jar packages?

Thanks a lot,
Elisabeta

#2

Could you explain some more what it is?

#3

Hi Romain,

Sorry for the delay. Sure.
If a client would like to add extra libraries/jars to the livy interpreter through spark.jars.package is it something that could be achievable through Hue?

Thanks

#4

Could you be even more explicit?

Hue is leveraging the Livy API, https://livy.incubator.apache.org/docs/latest/rest-api.html, so if you can do it there you can do it via Hue.

#5

Thanks for the quick reply.

Given the example livy rest api json call below:
{
“className”: “org.apache.spark.examples.SparkPi”,
“jars”: [“a.jar”, “b.jar”],
“pyFiles”: [“a.py”, “b.py”],
“files”: [“foo.txt”, “bar.txt”],
“archives”: [“foo.zip”, “bar.tar”],

“driverMemory”: “10G”,
“driverCores”: 1,
“executorCores”: 3,
“executorMemory”: “20G”,
“numExecutors”: 50,
“queue”: “default”,
“name”: “test”,
“proxyUser”: “foo”,
“conf”: {“spark.jars.packages”: “xxx”},
“file”: “hdfs:///path/to/examples.jar”,
“args”: [1000],
}

I couldn’t find yet in the Hue docs( didn’t check yet the code) how individual Hue users can pass the conf parameters for their session.

Thanks