Replace Values via regexp_replace Function in PySpark DataFrame

Code description

PySpark SQL APIs provides regexp_replace built-in function to replace string values that match with the specified regular expression.

It takes three parameters: the input column of the DataFrame, regular expression and the replacement for matches.

pyspark.sql.functions.regexp_replace(str, pattern, replacement)

Output

The following is the output from this code snippet:

+--------------+-------+----------------+
|       str_col|int_col|str_col_replaced|
+--------------+-------+----------------+
|Hello Kontext!|    100|  Hello kontext!|
|Hello Context!|    100|  Hello kontext!|
+--------------+-------+----------------+

All uppercase 'K' or 'C' are replaced with lowercase 'k'.

Code snippet

from pyspark.sql import SparkSession
from pyspark.sql.functions import regexp_replace

app_name = "PySpark regex sql functions"
master = "local"

spark = SparkSession.builder \
    .appName(app_name) \
    .master(master) \
    .getOrCreate()

spark.sparkContext.setLogLevel("WARN")

# Create a DataFrame
df = spark.createDataFrame(
    [['Hello Kontext!', 100], ['Hello Context!', 100]], ['str_col', 'int_col'])

# Replace str_col with regular expressions
df = df.withColumn('str_col_replaced',
                   regexp_replace('str_col', r'[C|K]', 'k'))

df.show()