Featured columns/forums
Articles about Apache Hadoop installation, performance tuning and general tutorials.
Apache Spark installation guides, performance tuning tips, general tutorials, etc.
Code snippets for various programming languages/frameworks.
About the releases of Kontext Project.
forum Kontext Feedbacks
Any Kontext website related questions, please publish here incl. feature suggestions, bug reports, other feedbacks, etc.
ML.NET is an open source and cross-platform machine learning framework. With ML.NET, you can create custom ML models using C# or F# without having to leave the .NET ecosystem. This column publish articles about ML.NET.
Featured posts
local_offer windows local_offer hadoop
This article provides detailed steps about how to compile and build Hadoop (incl. native libs) on Windows 10. The following guide is based on Hadoop release 3.2.1. ...
local_offer hadoop local_offer yarn local_offer hdfs
This page summarizes the steps to install Hadoop 3.0.0 on your Windows environment. Reference page: https://wiki.apache.org/hadoop/Hadoop2OnWindows ...
local_offer .net core local_offer entity-framework
SQLite is a self-contained and embedded SQL database engine. In .NET Core, Entity Framework Core provides APIs to work with SQLite. This page provides sample code to create a SQLite database using package Microsoft.EntityFrameworkCore.Sqlite . Create sample project ...
local_offer python local_offer spark local_offer pyspark
In Spark, SparkContext.parallelize function can be used to convert Python list to RDD and then RDD can be converted to DataFrame object. The following sample code is based on Spark 2.x. In this page, I am going to show you how to convert the following list to a data frame: data = [(...
local_offer windows local_offer hadoop local_offer yarn
This detailed step-by-step guide shows you how to install the latest Hadoop (v3.2.1) on Windows 10. It also provides a temporary fix for bug HDFS-14084 (java.lang.UnsupportedOperationException INFO).
local_offer hadoop local_offer yarn local_offer hdfs
This page summarizes the default ports used by Hadoop services. It is useful when configuring network interfaces in a cluster. Hadoop 3.1.0 HDFS The secondary namenode http/https server address and port. ...
local_offer hadoop local_offer hive
If you have been following my website, you would know I’ve published a number of articles about installing big data tools/framewo...
local_offer .NET local_offer dotnet core local_offer spark local_offer parquet local_offer hive
I’ve been following Mobius project for a while and have been waiting for this day. .NET for Apache Spark v0.1.0 was just published on 2019-04-25 on GitHub. It provides high performance APIs for programming Apache Spark applications with C# and F#. It is .NET Standard complaint and can run in Wind...
Kontext release v0.6.6
Kontext v0.6.6 is now released with a few changes/enhancements. Changes SEO enhancements Added a number of Facebook and twitter meta tags into head section of each page. Robots.txt is updated to make it simple. ...
Latest posts
local_offer windows local_offer hadoop
This article provides detailed steps about how to compile and build Hadoop (incl. native libs) on Windows 10. The following guide is based on Hadoop release 3.2.1. ...
local_offer windows local_offer hadoop local_offer yarn
This detailed step-by-step guide shows you how to install the latest Hadoop (v3.2.1) on Windows 10. It also provides a temporary fix for bug HDFS-14084 (java.lang.UnsupportedOperationException INFO).
Kontext release v0.6.7
Kontext v0.6.7 is now released with a few changes/enhancements. Changes The following sections list the new features/changes i...
Kontext release v0.6.6
Kontext v0.6.6 is now released with a few changes/enhancements. Changes SEO enhancements Added a number of Facebook and twitter meta tags into head section of each page. Robots.txt is updated to make it simple. ...
local_offer machine-learning local_offer jupyter-notebook local_offer C# local_offer dotnet core
In this article, I'm going to show you how to install Jupyter in Windows and then install .NET kernel for Jupyter notebooks. It also shows a machine learning example using ML.NET. The target audience are .NET developers who want to expand their skills in data engineering and science domain...
Kontext release v0.6.5
Kontext v0.6.5 is now released with a few changes/enhancements. Changes RSS changes RSS subscriptions are now increased to 200 items and the d...
local_offer pyspark local_offer spark-2-x local_offer python
This articles show you how to convert a Python dictionary list to a Spark DataFrame. The code snippets runs on Spark 2.x environments. Input The input data (dictionary list looks like the following): data = [{"Category": 'Category A', 'ItemID': 1, 'Amount': 12.40}, ...
local_offer pyspark local_offer spark local_offer spark-2-x local_offer pandas
Apache Arrow is an in-memory columnar data format that can be used in Spark to efficiently transfer data between JVM and Python processes. This currently is most beneficial to Python users that work with Pandas/NumPy data. In this article, ...
Read and Write XML files in PySpark
local_offer pyspark local_offer spark-2-x local_offer spark
This article shows you how to read and write XML files in Spark. Sample XML file Create a sample XML file named test.xml with the following content: <?xml version="1.0"?> <data> <record id="1"> <rid>1</rid> <nam...
