By using this site, you acknowledge that you have read and understand our Cookie policy, Privacy policy and Terms .
account_circle Raymond

Code snippets for various programming languages/frameworks.

local_offer pyspark local_offer spark-2-x local_offer spark

visibility 3
comment 0
thumb_up 0
access_time 3 days ago

Sometime it is necessary to pass environment variables to Spark executors. To pass environment variable to executors, use setExecutorEnv function of SparkConf class. Code snippet In the following code snippet, an environment variable name ENV_NAME is set up with value ...

open_in_new View

local_offer python

visibility 13
comment 0
thumb_up 1
access_time 18 days ago

Different programming languages have different package management tools.

open_in_new View

local_offer teradata local_offer SQL

visibility 10
comment 0
thumb_up 0
access_time 18 days ago

This code snippet shows how to calculate time differences.

open_in_new View

local_offer hadoop local_offer shell

visibility 6
comment 0
thumb_up 0
access_time 18 days ago

Hadoop provides a number of CLIs. hadoop job command can be used to retrieve running job list.

You can also use YARN resource manager UI to view the jobs too.

open_in_new View

local_offer hadoop local_offer shell

visibility 6
comment 0
thumb_up 0
access_time 18 days ago

Hadoop provides a number of CLIs that can be used to perform many tasks/activities. This code snippet shows you how to check file/folder size in HDFS.

open_in_new View

local_offer scala local_offer spark-2-x

visibility 38
comment 0
thumb_up 0
access_time 18 days ago

In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.

open_in_new View

local_offer python local_offer spark-2-x

visibility 31
comment 0
thumb_up 0
access_time 18 days ago

In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.

open_in_new View

local_offer scala local_offer spark-2-x

visibility 68
comment 0
thumb_up 0
access_time 18 days ago

Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object. 

open_in_new View

local_offer python local_offer spark-2-x

visibility 168
comment 0
thumb_up 0
access_time 18 days ago

Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object. 

open_in_new View

local_offer Javascript

visibility 3
comment 0
thumb_up 0
access_time 18 days ago

JSON is commonly used in modern applications for data storage and transfers. Pretty much all programming languages provide APIs to parse JSON. 

open_in_new View