Kontext Column
Follow three steps to create your columns.
Series
Big Data Tools on WSL
This series provides detailed tutorials about installing big data tools such as Hadoop, Spark, Hive, Sqoop, etc. on Windows 10 via WSL (Windows Subsystem for Linux).
* Logos used in the picture are registered trademarks of Apache or Linux foundations.
Teradata Utilities - BTEQ, FastLoad, FastExport, TPT
Tutorials about Teradata utilities incl. BTEQ, FastLoad, FastExport, TPT, etc.
Examples are provided about loading XML, JSON and CSV files into Teradata and also about exporting data from Teradata database.
Teradata Functions
Explore frequently used Teradata SQL functions.
Spark Connect to Databases
This series includes tutorials about how to read data from or write data into databases via JDBC or other drivers. With Java, Spark can interact with many SQL databases such as SQL Server, Oracle, Teradata, MySQL, PostgreSQL, SQLite, etc.
Querying Teradata using SQL
This series collects commonly used SQL queries to retrieve data from Teradata database. It also shows you how to do some basic filters and transformation using SQL statements.
Big Data Tools on Windows 10
This series provides detailed tutorials about installing big data tools such as Hadoop, Spark, Sqoop, Zeppelin, Hive, Sqoop, etc. on Windows 10 using Java and Windows native libraries.
* Logos used in the picture are registered trademarks of Apache or Microsoft.
Parse, Load and Write Files in Spark
This series includes articles about how to read and write files in Spark incl. plain text file, CSV, TSV, XML, Parquet, Avro, Orc, etc.
Hive SQL - DDL
Hive SQL DDL (Data Definition Language) tutorials. This series provides examples about how to create Hive databases, tables (incl. managed and external), views, etc.
Featured articles
local_offer python local_offer spark local_offer pyspark local_offer spark-advanced
Data partitioning is critical to data processing performance especially for large volume of data processing in Spark. Partitions in Spark won’t span across nodes though one node can contains more than one partitions. When processing, Spark assigns one task for each partition and each worker threa...
local_offer windows10 local_offer hadoop local_offer yarn local_offer big-data-on-windows-10
This detailed step-by-step guide shows you how to install the latest Hadoop (v3.2.1) on Windows 10. It also provides a temporary fix for bug HDFS-14084 (java.lang.UnsupportedOperationException INFO).
local_offer python local_offer spark local_offer pyspark local_offer spark-dataframe
In Spark, SparkContext.parallelize function can be used to convert Python list to RDD and then RDD can be converted to DataFrame object. The following sample code is based on Spark 2.x. In this page, I am going to show you how to convert the following list to a data frame: data = [(...
Save DataFrame as CSV File in Spark
local_offer pyspark local_offer spark local_offer spark-2-x local_offer spark-file-operations
Spark provides rich APIs to save data frames to many different formats of files such as CSV, Parquet, Orc, Avro, etc. CSV is commonly used in data application though nowadays binary formats are getting momentum. In this article, I am going to show you how to save Spark data frame as CSV file in b...
Spark - Save DataFrame to Hive Table
local_offer python local_offer spark local_offer pyspark local_offer hive local_offer spark-database-connect
From Spark 2.0, you can easily read data from Hive data warehouse and also write/append new data to Hive tables. This page shows how to operate with Hive in Spark including: Create DataFrame from existing Hive table Save DataFrame to a new Hive table Append data ...
local_offer python local_offer spark local_offer spark-dataframe
This post shows how to derive new column in a Spark data frame from a JSON array string column. I am running the code in Spark 2.2.1 though it is compatible with Spark 1.6.0 (with less JSON SQL functions). Prerequisites Refer to the following post to install Spark in Windows. ...
local_offer pyspark local_offer spark-2-x local_offer teradata local_offer SQL Server local_offer spark-database-connect
In my previous article about Connect to SQL Server in Spark (PySpark) , I mentioned the ways t...
Connect to SQL Server in Spark (PySpark)
local_offer SQL Server local_offer python local_offer spark local_offer pyspark local_offer spark-database-connect
Spark is an analytics engine for big data processing. There are various ways to connect to a database in Spark. This page summarizes some of common approaches to connect to SQL Server using Python as programming language. ...
local_offer asp.net core local_offer identity core 2
The identity system in ASP.NET has evolved over time. If you are using ASP.NET Core, you probably found User property is an instance of ClaimsPrincipal in Controller or Razor views. Thus to retrieve the information, you need to utilize the claims.
local_offer sqlite local_offer .net core local_offer entity-framework
SQLite is a self-contained and embedded SQL database engine. In .NET Core, Entity Framework Core provides APIs to work with SQLite. This page provides sample code to create a SQLite database using package Microsoft.EntityFrameworkCore.Sqlite . Create sample project ...
local_offer linux local_offer WSL local_offer ubuntu local_offer big-data-on-wsl
This page shows how to install Windows Subsystem for Linux (WSL) system on a non-system drive manually. Enable Windows Subsystem for Linux system feature Open PowerShell as Administrator and run the following command to enable WSL feature: Enable-WindowsOptionalFea...
local_offer hadoop local_offer hive local_offer big-data-on-windows-10
In this article, I’m going to demo how to install Hive 3.0.0 on Windows 10. Prerequisites Before installation of Apache Hive, please ensure you have Hadoop available on your Windows environment. We cannot run Hive without Hadoop. ...
local_offer pyspark local_offer spark-2-x local_offer spark local_offer python local_offer spark-dataframe
This article shows how to convert a Python dictionary list to a DataFrame in Spark using Python. Example dictionary list data = [{"Category": 'Category A', "ID": 1, "Value": 12.40}, {"Category": 'Category B', "ID": 2, "Value": 30.10}, {"Category": 'Category C', "...
local_offer hadoop local_offer yarn local_offer hdfs local_offer big-data-on-windows-10
This page summarizes the steps to install Hadoop 3.0.0 on your Windows environment. Reference page: https://wiki.apache.org/hadoop/Hadoop2OnWindows ...
local_offer hadoop local_offer linux local_offer WSL local_offer big-data-on-wsl
In my previous post , I showed how to configure a single node Hadoop instance on Windows 10. The steps are not too difficult to follow if you have Java programming backgr...
Featured sites
Latest articles
LINQ to SQL - Select N Random Records
local_offer .NET local_offer .net core local_offer C#
Language-Integrated Query (LINQ) is a set of technologies based on the integration of query capabilities directly into the C# or VB language in .NET. It allows intuitive query against SQL databases, XML, object list, etc. This article shows how to return a top N records randomly. ...
R Introduction
R is an implementation of the S programming language (Bell Labs). It was created by Ross Ihaka and Robert Gentleman at the University of Auckland, New Zealand. R is named partly after the first names of the first two R authors and partly as a play on the name of S. It is currently developed by th...
local_offer SQL Server local_offer t-sql
SQL Server has built-in function HASHBYTES that can be used to calculate hash values. The supported hash algorithms include MD2, MD4, MD5, SHA, SHA1, SHA2_256 and SHA2_512. Function HASHBYTES The signature of this function is: HASHBYTES ( '<a...
Calculate MD5 in Teradata
local_offer teradata local_offer SQL
Teradata has no built-in MD5 function thus custom function needs to be implemented for calculating MD5. This article shows you how to do that using the MD5 message digest UDF provided on Teradata Downloads. Prerequisites Permission CREATE FUNCTION is required for creating UDF in Ter...
local_offer teradata local_offer SQL
A procedure contains a number of SQL statements that will be executed in sequence in Teradata. Procedures can be used to return result sets to the invoking clients. In many other databases, it is very easy to return set records to the client in a procedure as SELECT statement can be directly used...
Convert TimeStamp to Date in Teradata
local_offer teradata local_offer SQL
Current timestamp Function CURRENT_TIMESTAMP can be used to retrieve the current timestamp: SELECT CURRENT_TIMESTAMP; Sample output: 20/09/2020 20:55:35.390000-04:00 Convert TimeStamp to Date Function C...
Teradata RANDOM Number
local_offer teradata local_offer SQL
RANDOM function in Teradata returns a random integer number for each row of the results table. It is a Teradata extension to the ANSI SQL:2011 standard. Function syntax Random(lower_bound, upper_bound) The limits for&...
local_offer teradata local_offer SQL
This article demonstrates how to create volatile table in a Teradata procedure, perform DML actions (INSERT, DELETE, UPDATE) against it and then return the result set dynamically from the temporary table in the procedure.
local_offer Azure local_offer python
A Microsoft SharePoint list is a collection of data can be shared with team members or people who you give access to. It is commonly used to capture commonly maintained master data from manual inputs. This article summarizes steps to create a SharePoint list...
Load JSON File into Teradata via BTEQ
local_offer teradata local_offer teradata-utilities
BTEQ is a Teradata utility tool that can be used to run Teradata SQL statements incl. DDL, DML, etc. It can also be used to import data from text file into Teradata databases. It also works with XML and JSON files too. Like TPT and FASTLOAD, it can run in both batch and interactive modes. T...
Load XML File into Teradata via BTEQ
local_offer teradata local_offer teradata-utilities
BTEQ is a Teradata utility tool that can be used to run Teradata SQL statements incl. DDL, DML, etc. It can also be used to import data from text file into Teradata databases. Like TPT and FASTLOAD, it can run in both batch and interactive modes. This article demonstrates how to load XML fi...
Export CSV File via TPT
local_offer teradata local_offer teradata-utilities
Teradata Parallel Transporter (TPT) provides rich functions to load data into Teradata and to export data. In article Load CSV into Teradata via TPT , it shows how to load CSV files into Teradata....
Export Text File via Teradata FastExport
local_offer teradata local_offer teradata-utilities
Teradata FastExport is a command utility tool that can transfer large amount of data from Teradata database to a file. One of the commonly used scenarios is to export data from a table or view to a text file and then load the export file into a different server. ...
Teradata FastLoad with Optional Quotes
local_offer teradata local_offer fastload local_offer teradata-utilities
In article Teradata FastLoad - Load CSV File , it shows how to load CSV into Teradata. The input file is very basic CSV file. This article expands on t...
Teradata FastLoad Skip Header Line
local_offer teradata local_offer fastload local_offer teradata-utilities
Teradata FastLoad can be used to load CSV/TSV or other delimited files into database. Refer to article Teradata FastLoad - Load CSV File for more details about how to load CSV into Ter...
