Spark DataFrame: Show Full Column Content without Truncation
When calling Spark show function to display the content of a DataFrame, it will not print out the full content of a column by default. For example, the following output prints out truncated column content:
+---+--------------------+ | ID| Value| +---+--------------------+ | 1|Adddddddddddddddd...| | 1| A| | 3| C| +---+--------------------+
To show the full content of the column, we just need to specify the truncate parameter to False:
:param truncate: If set to ``True``, truncate strings longer than 20 chars by default. If set to a number greater than one, truncates long strings to length ``truncate`` and align cells right.
Code snippet
from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType, StringType, StructField, StructType appName = "PySpark Example - Show Column Full Content" master = "local" # Create Spark session spark = SparkSession.builder \ .appName(appName) \ .master(master) \ .getOrCreate() # Sample data data = [(1, 'Addddddddddddddddddddddddddddddddddddddddddd'), (1, 'A'), (3, 'C')] # schema schema = StructType([StructField("ID", IntegerType(), True), StructField("Value", StringType(), True)]) # Create Spark DaraFrame from pandas DataFrame df = spark.createDataFrame(data, schema) df.show(truncate=False) spark.stop()
Output:
+---+--------------------------------------------+ |ID |Value | +---+--------------------------------------------+ |1 |Addddddddddddddddddddddddddddddddddddddddddd| |1 |A | |3 |C | +---+--------------------------------------------+
info Last modified by Raymond 3 years ago
copyright
This page is subject to Site terms.
comment Comments
No comments yet.