By using this site, you acknowledge that you have read and understand our Cookie policy, Privacy policy and Terms .
access_time 9 months ago comment 0 languageEnglish
visibility 393 arrow_upward arrow_downward

In Spark, there are a number of settings/configurations you can specify including application properties and runtime parameters.

https://spark.apache.org/docs/latest/configuration.html

Get current configurations

To retrieve all the current configurations, you can use the following code (Python):

from pyspark.sql import SparkSession

appName = "PySpark Partition Example"
master = "local[8]"

# Create Spark session with Hive supported.
spark = SparkSession.builder \
    .appName(appName) \
    .master(master) \
    .getOrCreate()

configurations = spark.sparkContext.getConf().getAll()
for conf in configurations:
    print(conf)

* The above code is for Spark 2.0+ versions.

The output for the above code looks similar like the following:

('spark.rdd.compress', 'True')
('spark.app.name', 'PySpark Partition Example')
('spark.app.id', 'local-1554464117837')
('spark.master', 'local[8]')
('spark.serializer.objectStreamReset', '100')
('spark.executor.id', 'driver')
('spark.submit.deployMode', 'client')
('spark.driver.host', 'Raymond-Alienware')
('spark.driver.port', '11504')
('spark.ui.showConsoleProgress', 'true')

info Last modified by Raymond at 9 months ago

Please log in or register to comment. account_circle Log in person_add Register
comment Comments (0)
No comments yet.
account_circle Raymond

Articles about Apache Spark

open_in_new View