Raymond Raymond

PySpark - Read Parquet Files in S3

event 2022-08-21 visibility 3,625 comment 0 insights
insights Stats
PySpark - Read Parquet Files in S3

Code description

This code snippet provides an example of reading parquet files located in S3 buckets on AWS (Amazon Web Services). 

The bucket used is from New York City taxi trip record data.

S3 bucket location is: s3a://ursa-labs-taxi-data/2009/01/data.parquet.

To run the script, we need to setup the package dependency on Hadoop AWS package, for example, org.apache.hadoop:hadoop-aws:3.3.0. This can be easily done by passing configuration argument using spark-submit:

 spark-submit --conf spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.0

This can also be done via SparkConf:

conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0')

Use temporary AWS credentials

In this code snippet, AWS AnonymousAWSCredentialsProvider is used. If the bucket is not public, we can also use TemporaryAWSCredentialsProvider.

conf.set('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider')
conf.set('spark.hadoop.fs.s3a.access.key', <access_key>)
conf.set('spark.hadoop.fs.s3a.secret.key', <secret_key>)
conf.set('spark.hadoop.fs.s3a.session.token', <token>)

If you have used AWS CLI or SAML tools to cache local credentials ( ~/.aws/credentials), you then don't need to specify the access keys assuming the credential has access to the S3 bucket you are reading data from.

Code snippet

from pyspark.sql import SparkSession
from pyspark import SparkConf

app_name = "PySpark - Read from S3 Example"
master = "local[1]"

conf = SparkConf().setAppName(app_name)\

# setup anonymoue AWS credential

spark = SparkSession.builder.config(conf=conf) \


file_path = 's3a://ursa-labs-taxi-data/2009/01/data.parquet'

df = spark.read.format('parquet').load(file_path)

More from Kontext
comment Comments
No comments yet.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts