By using this site, you acknowledge that you have read and understand our Cookie policy, Privacy policy and Terms .
access_time 11 days ago comment 0 languageEnglish

Sometime it is necessary to pass environment variables to Spark executors. To pass environment variable to executors, use setExecutorEnv function of SparkConf class.

Code snippet

In the following code snippet, an environment variable name ENV_NAME is set up with value as 'ENV_Value'.

from pyspark import SparkConf
from pyspark.sql import SparkSession

appName = "Python Example - Pass Environment Variable to Executors"
master = 'yarn'

# Create Spark session
conf = SparkConf().setMaster(master).setAppName(
    appName).setExecutorEnv('ENV_NAME', 'ENV_Value')

spark = SparkSession.builder.config(conf=conf) \
    .getOrCreate()

Please log in or register to comment. account_circle Log in person_add Register
comment Comments (0)
No comments yet.
account_circle Raymond

Code snippets for various programming languages/frameworks.

open_in_new View