Spark SQL - Convert Object to JSON String

access_time 6 days ago visibility4 comment 0

In article Scala: Parse JSON String as Spark DataFrame, it shows how to convert JSON string to Spark DataFrame; this article show the other way around - convert complex columns to a JSON string using to_json function.

About function to_json

Function 'to_json(expr[, options])' returns a JSON string with a given struct value.

For parameter options, it controls how the struct column is converted into a JSON string and accepts the same options as the JSON data source. Refer to Spark SQL - Convert JSON String to Map for more details about all the available options.

Code snippet

select to_json(map(1, 'a', 2, 'b', 3, DATE '2021-01-01'));

Output:

to_json(map(1, a, 2, b, 3, CAST(DATE '2021-01-01' AS STRING)))
{"1":"a","2":"b","3":"2021-01-01"}

The following example changes date format:

select to_json(map(1, DATE '2021-01-01'), map('dateFormat','dd/MM/yyyy'));

Output:

spark-sql> select to_json(map(1, DATE '2021-01-01'), map('dateFormat','dd/MM/yyyy'));
to_json(map(1, DATE '2021-01-01'))
{"1":"01/01/2021"}

Note: the format of DATE is changed to dd/MM/yyyy as specified by options parameter. 

copyright This page is subject to Site terms.
Like this article?
Share on

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

Follow Kontext

Get our latest updates on LinkedIn or Twitter.

Want to publish your article on Kontext?

Learn more

More from Kontext

visibility 4
thumb_up 0
access_time 6 days ago

In article  Scala: Parse JSON String as Spark DataFrame , it shows how to convert JSON string to Spark DataFrame; this article show the other way around - convert complex columns to a JSON string using to_json function. Function ' to_json(expr[, options]) ' returns a JSON string with a ...

visibility 6
thumb_up 0
access_time 7 days ago

Spark SQL function from_json(jsonStr, schema[, options]) returns a struct value with the given JSON string and format. Parameter options is used to control how the json is parsed. It accepts the same options as the  json data source in Spark DataFrame reader APIs. The following code ...

visibility 8
thumb_up 0
access_time 7 days ago

Function unix_timestamp() returns the UNIX timestamp of current time. You can also specify a input timestamp value.  Example: spark-sql> select unix_timestamp(); unix_timestamp(current_timestamp(), yyyy-MM-dd HH:mm:ss) 1610174099 spark-sql> select unix_timestamp(current_timestamp ...