Get the Current Spark Context Settings/Configurations
In Spark, there are a number of settings/configurations you can specify including application properties and runtime parameters.
Get current configurations
To retrieve all the current configurations, you can use the following code (Python):
from pyspark.sql import SparkSession appName = "PySpark Partition Example" master = "local[8]" # Create Spark session with Hive supported. spark = SparkSession.builder \ .appName(appName) \ .master(master) \ .getOrCreate() configurations = spark.sparkContext.getConf().getAll() for conf in configurations: print(conf)
* The above code is for Spark 2.0+ versions.
The output for the above code looks similar like the following:
('spark.rdd.compress', 'True')
('spark.app.name', 'PySpark Partition Example')
('spark.app.id', 'local-1554464117837')
('spark.master', 'local[8]')
('spark.serializer.objectStreamReset', '100')
('spark.executor.id', 'driver')
('spark.submit.deployMode', 'client')
('spark.driver.host', 'Raymond-Alienware')
('spark.driver.port', '11504')
('spark.ui.showConsoleProgress', 'true')