Code python

Time Travel with Delta Table in PySpark

Kontext Kontext visibility 1,020 comment 0 access_time 2 years ago language English

descriptionCode description

Delta Lake provides time travel functionalities to retrieve data at certain point of time or at certain version. This can be done easily using the following two options when reading from delta table as DataFrame:

  • versionAsOf - an integer value to specify a version.
  • timestampAsOf - A timestamp or date string.

This code snippet shows you how to use them in Spark DataFrameReader APIs. It includes three examples:

  • Query data as version 1
  • Query data as 30 days ago (using computed value via Spark SQL)
  • Query data as certain timestamp ('2022-09-01 12:00:00.999999UTC 10:00')

You may encounter issues if the timestamp is earlier than the earlier commit:

pyspark.sql.utils.AnalysisException: The provided timestamp (2022-08-03 00:00:00.0) is before the earliest version available to this
table (2022-08-27 10:53:18.213). Please use a timestamp after 2022-08-27 10:53:18.

Similarly, if the provided timestamp is later than the last commit, you may encounter another issue like the following:

pyspark.sql.utils.AnalysisException: The provided timestamp: 2022-09-07 00:00:00.0 is after the latest commit timestamp of
2022-08-27 11:30:47.185. If you wish to query this version of the table, please either provide
the version with "VERSION AS OF 1" or use the exact timestamp
of the last commit: "TIMESTAMP AS OF '2022-08-27 11:30:47'".

References

Delta Lake with PySpark Walkthrough

fork_rightFork
more_vert
copyright This page is subject to Site terms.
comment Comments
No comments yet.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts