Analytics & BI

Data Analytics,Big Data,Data Storage and Business Intelligence.


hadoop yarn hdfs

Install Hadoop 3.0.0 in Windows (Single Node)

21,153   28   about 2 years ago

This page summarizes the steps to install Hadoop 3.0.0 in your Windows environment. Reference page: ...

View detail
hadoop linux wsl

Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

3,948   16   about 3 months ago

In my previous post , I showed how to configure a single node Hadoop instance on Windows 10. The steps are not too difficult to follow if you have Java programming backgr...

View detail
.net dotnet core spark parquet hive

.NET for Apache Spark Preview with Examples

618   2   about 5 months ago

I’ve been following Mobius project for a while and have been waiting for this day. .NET for Apache Spark v0.1.0 was just published on 2019-04-25 on GitHub. It provides high performance APIs for programming Apache Spark applications with C# and F#. It is .NET Standard complaint and can run in Wind...

View detail
spark hadoop pyspark oozie hue

Run Multiple Python Scripts PySpark Application with yarn-cluster Mode

84   0   about 22 days ago

When submitting Spark applications to YARN cluster, two deploy modes can be used: client and cluster. For client mode (default), Spark driver runs on the machine that the Spark application was submitted while for cluster mode, the driver runs on a random node in a cluster. On this page, I am goin...

View detail
python pyspark pandas

Convert PySpark Row List to Pandas Data Frame

30   0   about 25 days ago

In Spark, it’s easy to convert Spark Dataframe to Pandas dataframe through one line of code: df_pd = df.toPandas() In this page, I am going to show you how to convert a list of PySpark row objects to a Pandas data frame. Prepare the data frame The fo...

View detail
power-bi bigquery

Use Google Cloud BigQuery as Data Source in Power BI

2,617   2   about 2 years ago

BigQuery is Google’s serverless data warehouse in Google Cloud. Power BI can consume data from various sources including RDBMS, NoSQL, Could, Services, etc. It is also easy to get data from BigQuery in Power BI. In this article, I am going to demonstrate how to connect to BigQuery to create...

View detail
hadoop hive wsl

Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

908   2   about 5 months ago

Previously, I demonstrated how to configured Apache Hive 3.0.0 on Windows 10. Apache Hive 3.0.0 Installation on Windows 10 Step by Step Guide...

View detail
spark hadoop yarn oozie

Diagnostics: Container is running beyond physical memory limits

164   0   about 3 months ago

Scenario Recently I created an Oozie workflow which contains one Spark action. The Spark action master is yarn and deploy mode is cluster. Each time when the job runs about 30 minutes, the application fails with errors like the following: Application applicatio...

View detail
lite-log spark pyspark

Fix PySpark TypeError: field **: **Type can not accept object ** in type <class '*'>

334   0   about 3 months ago

When creating Spark date frame using schemas, you may encounter errors about “field **: **Type can not accept object ** in type &lt;class '*'&gt;”. The actual error can vary, for instances, the following are some examples: field xxx: BooleanType can not accept object 100 in type ...

View detail
python spark pyspark

PySpark: Convert Python Array/List to Spark Data Frame

736   0   about 3 months ago

In Spark, SparkContext.parallelize function can be used to convert Python list to RDD and then RDD can be converted to DataFrame object. The following sample code is based on Spark 2.x. In this page, I am going to show you how to convert the following list to a data frame: data = [(...

View detail