By using this site, you acknowledge that you have read and understand our Cookie and Privacy policy. Your use of Kontext website is subject to this policy. Accept

Convert String to Date in Spark (Scala)

3790 views 0 comments last modified about 2 years ago Raymond Tang

lite-log scala

Context

This pages demonstrates how to convert string to java.util.Date in Spark via Scala.

Prerequisites

If you have not installed Spark, follow the page below to install it:

Install Big Data Tools (Spark, Zeppelin, Hadoop) in Windows for Learning and Practice

Sample code

The following code snippet uses pattern yyyy-MM-dd to parse string to Date.

import java.text.SimpleDateFormat

import java.util.Date

val format = new SimpleDateFormat("yyyy-MM-dd")

val date = format.parse("2018-03-03")

Output

image

Retrieve week day name

scala> val format2 = new SimpleDateFormat("EEEEE")
format2: java.text.SimpleDateFormat = java.text.SimpleDateFormat@3ecbf05

scala> format2.format(date)
res0: String = Saturday

Summary

For the complete list of Java date and time pattern, please refer to the following link.

https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html

Related pages

Debug PySpark Code in Visual Studio Code

24 views   0 comments last modified about 17 days ago

The page summarizes the steps required to run and debug PySpark (Spark for Python) in Visual Studio Code. Install Python and pip Install Python from the official website: https://...

View detail

Implement SCD Type 2 Full Merge via Spark Data Frames

336 views   0 comments last modified about 2 months ago

Overview For SQL developers that are familiar with SCD and merge statements, you may wonder how to implement the same in big data platforms, considering database or storages in Hadoop are not designed/optimised for record level updates and inserts. In this post, I’m going to demons...

View detail

PySpark: Convert JSON String Column to Array of Object (StructType) in Data Frame

442 views   0 comments last modified about 3 months ago

This post shows how to derive new column in a Spark data frame from a JSON array string column. I am running the code in Spark 2.2.1 though it is compatible with Spark 1.6.0 (with less JSON SQL functions). Prerequisites Refer to the following post to install Spark in Windows. ...

View detail

Write and Read Parquet Files in Spark/Scala

7267 views   2 comments last modified about 2 years ago

In this page, I’m going to demonstrate how to write and read parquet files in Spark/Scala by using Spark SQLContext class. Reference What is parquet format? Go the following project site to understand more about parquet. ...

View detail

Install Big Data Tools (Spark, Zeppelin, Hadoop) in Windows for Learning and Practice

1189 views   2 comments last modified about 11 months ago

Are you a Windows/.NET developer and willing to learn big data concepts and tools in your Windows? If yes, you can follow the links below to install them in your PC. The installations are usually easier to do in Linux/UNIX but they are not difficult to implement in Windows either since the...

View detail

Load Data into HDFS from SQL Server via Sqoop

1077 views   0 comments last modified about 12 months ago

This page shows how to import data from SQL Server into Hadoop via Apache Sqoop. Prerequisites Please follow the link below to install Sqoop in your machine if you don’t have one environment ready. ...

View detail

Add comment

Comments (0)

No comments yet.

Contacts

  • enquiry[at]kontext.tech

Subscribe