local_offer pyspark local_offer spark-2-x local_offer spark
This article shows you how to read and write XML files in Spark. Sample XML file Create a sample XML file named test.xml with the following content: <?xml version="1.0"?> <data> <record id="1"> <rid>1</rid> <nam...
local_offer python local_offer pandas
Pickle files are commonly used Python data related projects. This article shows how to create and load pickle files using Pandas. Create pickle file import pandas as pd import numpy as np file_name="data/test.pkl" data = np.random.randn(1000, 2) # pd.set_option('displ...
local_offer pyspark local_offer spark-2-x local_offer spark
Sometime it is necessary to pass environment variables to Spark executors. To pass environment variable to executors, use setExecutorEnv function of SparkConf class. Code snippet In the following code snippet, an environment variable name ENV_NAME is set up with value ...
Different programming languages have different package management tools.
local_offer teradata local_offer SQL
This code snippet shows how to calculate time differences.
local_offer hadoop local_offer shell
Hadoop provides a number of CLIs. hadoop job command can be used to retrieve running job list.
You can also use YARN resource manager UI to view the jobs too.
local_offer hadoop local_offer shell
Hadoop provides a number of CLIs that can be used to perform many tasks/activities. This code snippet shows you how to check file/folder size in HDFS.
local_offer scala local_offer spark-2-x
In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.
local_offer python local_offer spark-2-x
In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.
local_offer scala local_offer spark-2-x
Spark has easy fluent APIs that can be used to read data from JSON file as DataFrame object.
