Spark SQL
In Spark-SQL CLI tool, the result print will omit headings (column names) by default. To display columns, we need to update Spark setting spark.hadoop.hive.cli.print.header. To make the changes for all spark-sql sessions, edit file $SPARK_HOME/conf/spark-defaults.conf . Add the following ...
In Spark SQL, virtual table can be constructed directly from constant values (literals) using SELECT statement. spark-sql> SELECT StrColumn FROM VALUES ('abc'),('def'),('ghi') table1(StrColumn); StrColumn abc def ghi spark-sql> SELECT ID,StrColumn FROM VALUES ...
Spark SQL - PIVOT Clause
Like other SQL engines, Spark also supports PIVOT clause. PIVOT is usually used to calculated aggregated values for each value in a column and the calculated values will be included as columns in the result set. PIVOT ( { aggregate_expression [ AS aggregate_expression_alias ] } [ , ... ] FOR ...
Spark SQL - ROW_NUMBER Window Functions
ROW_NUMBER in Spark assigns a unique sequential number (starting from 1) to each record based on the ordering of rows in each window partition. It is commonly used to deduplicate data. The following sample SQL uses ROW_NUMBER function without PARTITION BY clause: SELECT TXN.*, ROW_NUMBER() OVER ...
Spark SQL - Convert String to Timestamp
Similar as Convert String to Date using Spark SQL , you can convert string of timestamp to Spark SQL timestamp data type. Function to_timestamp(timestamp_str[, fmt]) p arses the `timestamp_str` expression with the `fmt` expression to a timestamp data type in Spark. Example ...
Spark SQL - UNIX timestamp functions
Function unix_timestamp() returns the UNIX timestamp of current time. You can also specify a input timestamp value. Example: spark-sql> select unix_timestamp(); unix_timestamp(current_timestamp(), yyyy-MM-dd HH:mm:ss) 1610174099 spark-sql> select unix_timestamp(current_timestamp ...
JSON string values can be extracted using built-in Spark functions like get_json_object or json_tuple. Values can be extracted using get_json_object function. The function has two parameters: json_txt and path. The first is the JSON text itself, for example a string column in your Spark ...
Spark SQL - Map Functions
In Spark SQL, MapType is designed for key values, which is like dictionary object type in many other programming languages. This article summarize the commonly used map functions in Spark SQL. Function map is used to create a map. Example: spark-sql> select ...
In Spark SQL, function std or stddev or stddev_sample can be used to calculate sample standard deviation from values of a group. std(expr) stddev(expr) stddev_samp(expr) The first two functions are the alias of stddev_sample function. SELECT ACCT ...
Spark SQL - RANK Window Function
RANK in Spark calculates the rank of a value in a group of values. It returns one plus the number of rows proceeding or equals to the current row in the ordering of a partition. The returned values are not sequential. The following sample SQL uses RANK function without PARTITION BY ...
