Spark DataFrame: Show Full Column Content without Truncation

When calling Spark showfunction to display the content of a DataFrame, it will not print out the full content of a column by default. For example, the following output prints out truncated column content:

+---+--------------------+
| ID|               Value|
+---+--------------------+
|  1|Adddddddddddddddd...|
|  1|                   A|
|  3|                   C|
+---+--------------------+

To show the full content of the column, we just need to specify the truncateparameter to False:

:param truncate: If set to ``True``, truncate strings longer than 20 chars by default.
            If set to a number greater than one, truncates long strings to length ``truncate``
            and align cells right.

Code snippet

from pyspark.sql import SparkSession
from pyspark.sql.types import IntegerType, StringType, StructField, StructType

appName = "PySpark Example - Show Column Full Content"
master = "local"

# Create Spark session
spark = SparkSession.builder \
    .appName(appName) \
    .master(master) \
    .getOrCreate()

# Sample data
data = [(1, 'Addddddddddddddddddddddddddddddddddddddddddd'),
        (1, 'A'), (3, 'C')]

# schema
schema = StructType([StructField("ID", IntegerType(), True),
                     StructField("Value", StringType(), True)])

# Create Spark DaraFrame from pandas DataFrame
df = spark.createDataFrame(data, schema)
df.show(truncate=False)

spark.stop()

Output:

+---+--------------------------------------------+
|ID |Value                                       |
+---+--------------------------------------------+
|1  |Addddddddddddddddddddddddddddddddddddddddddd|
|1  |A                                           |
|3  |C                                           |
+---+--------------------------------------------+

Code snippet

In this article