Delete or Remove Columns from PySpark DataFrame

access_time 2 months ago visibility113 comment 0

This article shows how to 'delete' column from Spark data frame using Python. 

Construct a dataframe 

Follow article Convert Python Dictionary List to PySpark DataFrame to construct a dataframe.

+----------+---+------+
|  Category| ID| Value|
+----------+---+------+
|Category A|  1| 12.40|
|Category B|  2| 30.10|
|Category C|  3|100.01|
+----------+---+------+

'Delete' or 'Remove' one column

The word 'delete' or 'remove' can be misleading as Spark is lazy evaluated. 

We can use drop function to remove or delete columns from a DataFrame.

df1 = df.drop('Category')
df1.show()

Output:

+---+------+
| ID| Value|
+---+------+
|  1| 12.40|
|  2| 30.10|
|  3|100.01|
+---+------+

Drop multiple columns

Multiple columns can be dropped at the same time:

df2 = df.drop('Category', 'ID')
df2.show()

columns_to_drop = ['Category', 'ID']
df3 = df.drop(*columns_to_drop)
df3.show()
Output:
+------+
| Value|
+------+
| 12.40|
| 30.10|
|100.01|
+------+

+------+
| Value|
+------+
| 12.40|
| 30.10|
|100.01|
+------+

Run Spark code

You can easily run Spark code on your Windows or UNIX-alike (Linux, MacOS) systems. Follow these articles to setup your Spark environment if you don't have one yet:

info Last modified by Administrator at 2 months ago copyright This page is subject to Site terms.
Like this article?
Share on

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

Kontext Column

Created for everyone to publish data, programming and cloud related articles.
Follow three steps to create your columns.


Learn more arrow_forward

More from Kontext

local_offer tutorial local_offer pyspark local_offer spark local_offer how-to local_offer spark-dataframe

visibility 103
thumb_up 0
access_time 2 months ago

This article shows how to change column types of Spark DataFrame using Python. For example, convert StringType to DoubleType, StringType to Integer, StringType to DateType. Construct a dataframe  Follow article  ...

local_offer tutorial local_offer pyspark local_offer spark local_offer how-to local_offer spark-dataframe

visibility 125
thumb_up 0
access_time 2 months ago

This article shows how to add a constant or literal column to Spark data frame using Python.  Construct a dataframe  Follow article  Convert Python Dicti...

local_offer tutorial local_offer pyspark local_offer spark local_offer how-to local_offer spark-dataframe

visibility 125
thumb_up 0
access_time 2 months ago

This article shows how to add a constant or literal column to Spark data frame using Python.  Construct a dataframe  Follow article  Convert Python Dicti...

About column

Spark

Apache Spark installation guides, performance tuning tips, general tutorials, etc.

*Spark logo is a registered trademark of Apache Spark.

rss_feed Subscribe RSS