This code snippet provides one example of concatenating columns using a separator in Spark DataFrame. Function concat_ws
is used directly. For Spark SQL version, refer to Spark SQL - Concatenate w/o Separator (concat_ws and concat).
Syntax of concat_ws
pyspark.sql.functions.concat_ws(sep: str, *cols: ColumnOrName)
Output:
+-----+--------+--------------+ | col1| col2| col1_col2| +-----+--------+--------------+ |Hello| Kontext| Hello,Kontext| |Hello|Big Data|Hello,Big Data| +-----+--------+--------------+