Hot sites
.NET Framework
Java Programming
Angular
ASP.NET Core
Scripting
PowerShell, Bash, ksh, sh, Perl and etc.
Teradata
Hadoop
Spark
Power BI
Code snippets
local_offer hadoop local_offer yarn local_offer hdfs
This page summarizes the steps to install Hadoop 3.0.0 in your Windows environment. Reference page: https://wiki.apache.org/hadoop/Hadoop2OnWindows ...
local_offer .net core local_offer entity-framework
SQLite is a self-contained and embedded SQL database engine. In .NET Core, Entity Framework Core provides APIs to work with SQLite. This page provides sample code to create a SQLite database using package Microsoft.EntityFrameworkCore.Sqlite . Create sample project ...
local_offer angular local_offer lite-log
Problem When you follow Angular CLI installation guide in Windows, you may encounter the following error: ng is not recognized as an internal or external command The resolutions are available in the following link: ...
local_offer python local_offer spark
This post shows how to derive new column in a Spark data frame from a JSON array string column. I am running the code in Spark 2.2.1 though it is compatible with Spark 1.6.0 (with less JSON SQL functions). Prerequisites Refer to the following post to install Spark in Windows. ...
Connect to SQL Server in Spark (PySpark)
local_offer SQL Server local_offer python local_offer spark local_offer pyspark
Spark is an analytics engine for big data processing. There are various ways to connect to a database in Spark. This page summarizes some of common approaches to connect to SQL Server using Python as programming language. ...
local_offer hadoop local_offer hive
If you have been following my website, you would know I’ve published a number of articles about installing big data tools/framewo...
Spark - Save DataFrame to Hive Table
local_offer python local_offer spark local_offer pyspark local_offer hive
From Spark 2.0, you can easily read data from Hive data warehouse and also write/append new data to Hive tables. This page shows how to operate with Hive in Spark including: Create DataFrame from existing Hive table Save DataFrame to a new Hive table Append data ...
local_offer python local_offer spark local_offer pyspark
Data partitioning is critical to data processing performance especially for large volume of data processing in Spark. Partitions in Spark won’t span across nodes though one node can contains more than one partitions. When processing, Spark assigns one task for each partition and each worker threa...
local_offer hadoop local_offer linux local_offer WSL
In my previous post , I showed how to configure a single node Hadoop instance on Windows 10. The steps are not too difficult to follow if you have Java programming backgr...
local_offer python local_offer spark local_offer pyspark
In Spark, SparkContext.parallelize function can be used to convert Python list to RDD and then RDD can be converted to DataFrame object. The following sample code is based on Spark 2.x. In this page, I am going to show you how to convert the following list to a data frame: data = [(...
Kontext Pre-release v0.6.0
local_offer kontext local_offer Azure
Kontext website is now upgraded to ASP.NET Core 3.0 with many new features.
Calculate time difference in PowerShell
This code snippet shows how to calculate time differences.
Convert string to date in SQL / Teradata
local_offer teradata local_offer SQL
This code snippet shows how to convert string to date.
List and kill jobs in Shell / Hadoop
local_offer hadoop local_offer shell
The following code snippet shows how to list and kill Hadoop jobs including (MapReduce and YARN jobs).
local_offer mssql local_offer t-sql
In different databases, the syntax of selecting top N records are slightly different. They may also differ from ISO standards.
Select top N records in SQL / Teradata
local_offer teradata local_offer SQL
In different databases, the syntax of selecting top N records are slightly different. They may also differ from ISO standards.
Select top N records in SQL / Hive
local_offer SQL local_offer hive
In different databases, the syntax of selecting top N records are slightly different. They may also differ from ISO standards.
local_offer python local_offer spark-2-x
In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.
local_offer scala local_offer spark-2-x
In Spark, SparkContext.parallelize function can be used to convert list of objects to RDD and then RDD can be converted to DataFrame object through SparkSession.
Check HDFS folder size in Shell / Hadoop
local_offer hadoop local_offer shell
Hadoop provides a number of CLIs that can be used to perform many tasks/activities. This code snippet shows you how to check file/folder size in HDFS.