spark-2-x

Resolve: Python in worker has different version 2.7 than that in driver 3.8...

PySpark Read Multiline (Multiple Lines) from CSV File

Save DataFrame to SQL Databases via JDBC in PySpark

Spark Read from SQL Server Source using Windows/Kerberos Authentication

Schema Merging (Evolution) with Parquet in Spark and Hive

PySpark: Convert Python Dictionary List to Spark DataFrame

Improve PySpark Performance using Pandas UDF with Apache Arrow

Read and Write XML files in PySpark

Convert Python Dictionary List to PySpark DataFrame

Pass Environment Variables to Executors in PySpark

Save DataFrame as CSV File in Spark

Read JSON file as Spark DataFrame in Python / Spark

Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object.

Read JSON file as Spark DataFrame in Scala / Spark

Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object.

Convert List to Spark Data Frame in Scala / Spark

In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.

Convert List to Spark Data Frame in Python / Spark

In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.