spark-dataframe

Filter Spark DataFrame Columns with None or Null Values

This article shows you how to filter NULL/None values from a Spark data frame using Python. Function DataFrame.filter or DataFrame.where can be used to filter out null values.

Change Column Type in PySpark DataFrame

Add Constant Column to PySpark DataFrame

Delete or Remove Columns from PySpark DataFrame

Rename DataFrame Column Names in PySpark

PySpark: Convert Python Dictionary List to Spark DataFrame

Convert Python Dictionary List to PySpark DataFrame

Convert List to Spark Data Frame in Python / Spark

In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.

Convert PySpark Row List to Pandas Data Frame

PySpark: Convert Python Array/List to Spark Data Frame

PySpark: Convert JSON String Column to Array of Object (StructType) in Data Frame