Scala object script in Spark

June 15, 2021

What's inside this article ⌄

How to run Scala object Spark
Scala script with object Spark

Suppose you have code like this in Scala test.scala:

import org.apache.spark.sql.{RuntimeConfig, SparkSession}

val spark: SparkSession = SparkSession.builder().appName("YourApp").getOrCreate()
val conf: RuntimeConfig = spark.conf

val yourParameter: String = conf.get("spark.driver.your_parameter")
print(yourParameter)

And you run it like this run.sh:

spark2-shell -i /path/to/script/test.scala --conf spark.driver.your_parameter="value"

And you want to put it in a Scala Object. Firstly, you are going to do smth like that:

import org.apache.spark.sql.{RuntimeConfig, SparkSession}

val spark: SparkSession = SparkSession.builder().appName("YourApp").getOrCreate()
val conf: RuntimeConfig = spark.conf

object MyObject {
	def main(): Unit = {
		val yourParameter: String = conf.get("spark.driver.your_parameter")
		print(yourParameter)
	}
}

But when you run the script, nothing happens and spark2-shell just opens. In order for Spark to run this code, and also to close when the code is executed, you need to add the sys.exit line inside MyObject, and also make a call outside it:

import org.apache.spark.sql.{RuntimeConfig, SparkSession}

val spark: SparkSession = SparkSession.builder().appName("YourApp").getOrCreate()
val conf: RuntimeConfig = spark.conf

object MyObject {
	def main(): Unit = {
		val yourParameter: String = conf.get("spark.driver.your_parameter")
		print(yourParameter)
		sys.exit
	}
}

MyObject.main()