Featured columns/forums
Apache Spark installation guides, performance tuning tips, general tutorials, etc.
Articles about Apache Hadoop installation, performance tuning and general tutorials.
Code snippets for various programming languages/frameworks.
Featured posts
local_offer python local_offer spark local_offer pyspark
In Spark, SparkContext.parallelize function can be used to convert Python list to RDD and then RDD can be converted to DataFrame object. The following sample code is based on Spark 2.x. In this page, I am going to show you how to convert the following list to a data frame: data = [(...
local_offer python local_offer spark local_offer pyspark
Data partitioning is critical to data processing performance especially for large volume of data processing in Spark. Partitions in Spark won’t span across nodes though one node can contains more than one partitions. When processing, Spark assigns one task for each partition and each worker threa...
local_offer python local_offer spark
This post shows how to derive new column in a Spark data frame from a JSON array string column. I am running the code in Spark 2.2.1 though it is compatible with Spark 1.6.0 (with less JSON SQL functions). Prerequisites Refer to the following post to install Spark in Windows. ...
Connect to SQL Server in Spark (PySpark)
local_offer SQL Server local_offer python local_offer spark local_offer pyspark
Spark is an analytics engine for big data processing. There are various ways to connect to a database in Spark. This page summarizes some of common approaches to connect to SQL Server using Python as programming language. ...
local_offer hadoop local_offer hive
If you have been following my website, you would know I’ve published a number of articles about installing big data tools/framewo...
Spark - Save DataFrame to Hive Table
local_offer python local_offer spark local_offer pyspark local_offer hive
From Spark 2.0, you can easily read data from Hive data warehouse and also write/append new data to Hive tables. This page shows how to operate with Hive in Spark including: Create DataFrame from existing Hive table Save DataFrame to a new Hive table Append data ...
local_offer .net core local_offer entity-framework
SQLite is a self-contained and embedded SQL database engine. In .NET Core, Entity Framework Core provides APIs to work with SQLite. This page provides sample code to create a SQLite database using package Microsoft.EntityFrameworkCore.Sqlite . Create sample project ...
local_offer spark local_offer scala local_offer parquet
In this page, I’m going to demonstrate how to write and read parquet files in Spark/Scala by using Spark SQLContext class. Reference What is parquet format? Go the following project site to understand more about parquet. ...
local_offer angular local_offer lite-log
Problem When you follow Angular CLI installation guide in Windows, you may encounter the following error: ng is not recognized as an internal or external command The resolutions are available in the following link: ...
Latest posts
local_offer hive local_offer hdfs
In Hive, there are two types of tables can be created - internal and external table. Internal tables are also called managed tables. Different features are available to different types. This article lists some of the common differences. Internal table By default, Hive creates ...
local_offer pyspark local_offer SQL Server local_offer spark-2-x
In this article, I am going to show you how to use JDBC Kerberos authentication to connect to SQL Server sources in Spark (PySpark). I will use Kerberos connection with principal names and password directly that requires ...
local_offer parquet local_offer pyspark local_offer spark-2-x local_offer hive local_offer hdfs
Schema evolution is supported by many frameworks or data serialization systems such as Avro, Orc, Protocol Buffer and Parquet. With schema evolution, one set of data can be stored in multiple files with different but compatible schema. In Spark, Parquet data source can detect and merge schema ...
Kontext release v0.x
Kontext release information before v.1.0.
local_offer windows10 local_offer hadoop local_offer hdfs
Issue When installing Hadoop 3.2.1 on Windows 10, you may encounter the following error when trying to format HDFS namnode: ERROR namenode.NameNode: Failed to start namenode. The error happens when running the following comm...
local_offer windows10 local_offer hadoop
This article provides detailed steps about how to compile and build Hadoop (incl. native libs) on Windows 10. The following guide is based on Hadoop release 3.2.1. ...
local_offer windows10 local_offer hadoop local_offer yarn
This detailed step-by-step guide shows you how to install the latest Hadoop (v3.2.1) on Windows 10. It also provides a temporary fix for bug HDFS-14084 (java.lang.UnsupportedOperationException INFO).
Kontext release v0.6.7
Kontext v0.6.7 is now released with a few changes/enhancements. Changes The following sections list the new features/changes in release v0.6.7. Multi-la...
Kontext release v0.6.6
Kontext v0.6.6 is now released with a few changes/enhancements. Changes SEO enhancements Added a number of Facebook and twitter meta tags into head section of each page. Robots.txt is updated to make it simple. ...
