Code snippets for various programming languages/frameworks.
local_offer pyspark local_offer spark-2-x local_offer spark
Sometime it is necessary to pass environment variables to Spark executors. To pass environment variable to executors, use setExecutorEnv function of SparkConf class. Code snippet In the following code snippet, an environment variable name ENV_NAME is set up with value ...
Different programming languages have different package management tools.
local_offer teradata local_offer SQL
This code snippet shows how to calculate time differences.
local_offer hadoop local_offer shell
Hadoop provides a number of CLIs. hadoop job command can be used to retrieve running job list.
You can also use YARN resource manager UI to view the jobs too.
local_offer hadoop local_offer shell
Hadoop provides a number of CLIs that can be used to perform many tasks/activities. This code snippet shows you how to check file/folder size in HDFS.
local_offer scala local_offer spark-2-x
In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.
local_offer python local_offer spark-2-x
In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.
local_offer scala local_offer spark-2-x
Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object.
local_offer python local_offer spark-2-x
Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object.
JSON is commonly used in modern applications for data storage and transfers. Pretty much all programming languages provide APIs to parse JSON.