PySpark DataFrame - Calculate Distinct Count of Column(s)

Code description

This code snippet provides an example of calculating distinct count of values in PySpark DataFrame using countDistinct PySpark SQL function.

Output:

+---+-----+
| ID|Value|
+---+-----+
|101|   56|
|101|   67|
|102|   70|
|103|   93|
|104|   70|
+---+-----+

+-----------------+------------------+
|DistinctCountOfID|DistinctCountOfRow|
+-----------------+------------------+
|                4|                 5|
+-----------------+------------------+

Code snippet

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

app_name = "PySpark countDistinct Example"
master = "local"

spark = SparkSession.builder \
    .appName(app_name) \
    .master(master) \
    .getOrCreate()

spark.sparkContext.setLogLevel("WARN")

data = [
    [101, 56],
    [101, 67],
    [102, 70],
    [103, 93],
    [104, 70]
]

df = spark.createDataFrame(data, ['ID', 'Value'])

df.show()

df_agg = df.groupBy() \
    .agg(F.countDistinct('ID').alias('DistinctCountOfID'),
         F.countDistinct('ID', 'Value').alias('DistinctCountOfRow'))
df_agg.show()