Pass Environment Variables to Executors in PySpark

access_time 2 years ago visibility1075 comment 0

Sometime it is necessary to pass environment variables to Spark executors. To pass environment variable to executors, use setExecutorEnv function of SparkConf class.

Code snippet

In the following code snippet, an environment variable name ENV_NAME is set up with value as 'ENV_Value'.

from pyspark import SparkConf
from pyspark.sql import SparkSession

appName = "Python Example - Pass Environment Variable to Executors"
master = 'yarn'

# Create Spark session
conf = SparkConf().setMaster(master).setAppName(
    appName).setExecutorEnv('ENV_NAME', 'ENV_Value')

spark = SparkSession.builder.config(conf=conf) \
    .getOrCreate()
copyright This page is subject to Site terms.
Like this article?
Share on

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

Follow Kontext

Get our latest updates on LinkedIn or Twitter.

Want to publish your article on Kontext?

Learn more

More from Kontext

visibility 24
thumb_up 0
access_time 2 months ago

This article shows how to add a constant or literal column to Spark data frame using Scala.  Follow article  Scala: Convert List to Spark Data Frame to construct a Spark data frame. +----------+-----+------------------+ | Category|Count| ...

visibility 153
thumb_up 0
access_time 4 months ago

Recently, one of my colleague asked me one question about Spark: for the same SQL statement on finding max value of partition column, different values are returned in Spark SQL and Hive/Impala SQL. The SQL statement looks like the following: SELECT MAX(PART_COL) FROM HiveDb.TestSQL; ...

Spark Read from SQL Server Source using Windows/Kerberos Authentication
visibility 1441
thumb_up 0
access_time 12 months ago

In this article, I am going to show you how to use JDBC Kerberos authentication to connect to SQL Server sources in Spark (PySpark). I will use  Kerberos connection with principal names and password directly that requires  Microsoft JDBC Driver 6.2  or above. The sample code can run ...