Diagram Raymond Raymond

Spark Partition Discovery

event 2021-12-22 visibility 429 comment 0

Spark supports partition discovery. All built in file sources (Text/CSV/JSON/ORC/Parquet) supports partition discovery and partition information inference. 

This data shows a example data set that is stored by two partition levels: month and country.

The following code snippet will read all the underlying parquet files:

df = spark.read.option("basePath","/data").parquet("/data")
comment Comments
No comments yet.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

tag Tags
info Info
Image URL