Spark Partition Discoveryimage

visibility 66 access_time 6 months ago languageEnglish

Spark supports partition discovery. All built in file sources (Text/CSV/JSON/ORC/Parquet) supports partition discovery and partition information inference. 

This data shows a example data set that is stored by two partition levels: month and country.

The following code snippet will read all the underlying parquet files:

df = spark.read.option("basePath","/data").parquet("/data")
info Last modified by Raymond 6 months ago copyright This page is subject to Site terms.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

comment Comments
No comments yet.
timeline Stats
Page index 0.35
local_offer Tags

info Info
Image URL
SVG URL
URL
More from Kontext
PySpark - Fix PermissionError: [WinError 5] Access is denied
visibility 10,036
thumb_up 0
access_time 4 years ago
Add Constant Column to PySpark DataFrame
visibility 7,897
thumb_up 0
access_time 2 years ago