Convert string to date in Python / Spark

access_time 2 years ago visibility1471 comment 0

This code snippet shows how to convert string to date.

In PySpark/Python, we can use unix_timestamp and from_unixtime functions.

Code snippet

from pyspark.sql import SparkSession
from pyspark.sql.functions import unix_timestamp, from_unixtime

appName = "PySpark Date Parse Example"
master = "local"

# Create Spark session with Hive supported.
spark = SparkSession.builder \
    .appName(appName) \
    .master(master) \
    .getOrCreate()

df = spark.createDataFrame([('2019-06-01',)], ['DATE_STR_COL'])
df.select(from_unixtime(unix_timestamp(df.DATE_STR_COL, 'yyyy-MM-dd')).alias('DATE_COL'))
df.show()
info Last modified by Raymond 2 years ago copyright This page is subject to Site terms.
Like this article?
Share on

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

Follow Kontext

Get our latest updates on LinkedIn or Twitter.

Want to publish your article on Kontext?

Learn more

More from Kontext

visibility 1838
thumb_up 0
access_time 2 years ago

In one of my previous articles about Password Security Solution for Sqoop , I mentioned creating credential using hadoop credential command. The credentials are stored in JavaKeyStoreProvider. Credential providers are used to separate the use of sensitive tokens, secrets and passwords from the ...

visibility 136
thumb_up 0
access_time 10 months ago

SQLite is one of the most commonly used embedded file databases. All the mainstream programming language/framework provides APIs to interact with SQLite database. In my previous article  SQLite in .NET Core with Entity Framework Core , code snippet is provided to interact with SQLite via ...

Pandas DataFrame Plot - Area Chart
visibility 311
thumb_up 0
access_time 10 months ago

This article provides examples about plotting area chart using  pandas.DataFrame.plot  or  pandas.core.groupby.DataFrameGroupBy.plot   function. The data I'm going to use is the same as the other article  Pandas DataFrame Plot - Bar Chart . I'm also using Jupyter ...