Code python

PySpark DataFrame - Add or Subtract Milliseconds from Timestamp Column

Kontext Kontext visibility 2,163 comment 0 access_time 2 years ago language English

descriptionCode description

This code snippets shows you how to add or subtract milliseconds (or microseconds) and seconds from a timestamp column in Spark DataFrame.

It first creates a DataFrame in memory and then add and subtract milliseconds/seconds from the timestamp column ts using Spark SQL internals. 

Output:

+---+--------------------------+--------------------------+--------------------------+--------------------------+
|id |ts                        |ts1                       |ts2                       |ts3                       |
+---+--------------------------+--------------------------+--------------------------+--------------------------+
|1  |2022-09-01 12:05:37.227916|2022-09-01 12:05:37.226916|2022-09-01 12:05:37.228916|2022-09-01 12:05:38.227916|
|2  |2022-09-01 12:05:37.227916|2022-09-01 12:05:37.226916|2022-09-01 12:05:37.228916|2022-09-01 12:05:38.227916|
|3  |2022-09-01 12:05:37.227916|2022-09-01 12:05:37.226916|2022-09-01 12:05:37.228916|2022-09-01 12:05:38.227916|
|4  |2022-09-01 12:05:37.227916|2022-09-01 12:05:37.226916|2022-09-01 12:05:37.228916|2022-09-01 12:05:38.227916|
+---+--------------------------+--------------------------+--------------------------+--------------------------+

*Note - the code assuming SparkSession object already exists via variable name spark

fork_rightFork
more_vert
copyright This page is subject to Site terms.
comment Comments
No comments yet.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts