Pass Environment Variables to Executors in PySpark
Sometime it is necessary to pass environment variables to Spark executors. To pass environment variable to executors, use setExecutorEnv function of SparkConf class.
Code snippet
In the following code snippet, an environment variable name ENV_NAME is set up with value as 'ENV_Value'.
from pyspark import SparkConf from pyspark.sql import SparkSession appName = "Python Example - Pass Environment Variable to Executors" master = 'yarn' # Create Spark session conf = SparkConf().setMaster(master).setAppName( appName).setExecutorEnv('ENV_NAME', 'ENV_Value') spark = SparkSession.builder.config(conf=conf) \ .getOrCreate()
copyright
This page is subject to Site terms.
comment Comments
No comments yet.
Log in with external accounts
warning Please login first to view stats information.
article
Debug PySpark Code in Visual Studio Code
article
Diagnostics: Container is running beyond physical memory limits
article
Spark DEBUG: It is possible the underlying files have been updated.
article
Differences between spark.sql.shuffle.partitions and spark.default.parallelism
article
Convert Python Dictionary List to PySpark DataFrame
Read more (127)