Kontext Platform
Console

parquet

10 items tagged with "parquet"

9 Articles
1 Diagram

Articles

Spark 2.x to 3.x - Date, Timestamp and Int96 Rebase Modes

2022-06-19
Spark & PySpark

Create Partitioned Hive Table

2021-12-23
Code Snippets & Tips

Get Started on .NET 5 with Apache Spark

.NET for Apache Spark 1.0 was officially released on 14th Oct 2020. This version was released together with .NET Core 3.0. Since .NET for Apache Spark is written with .NET Standards, it should work with .NET 5 too. This articles how to use .NET 5 with Apache Spark.

2020-12-23
.NET Programming

Schema Merging (Evolution) with Parquet in Spark and Hive

2020-02-02
Spark & PySpark

.NET for Apache Spark Preview with Examples

2019-04-26
Spark & PySpark

Configure Sqoop in a Edge Node of Hadoop Cluster

2018-05-12
Sqoop

Load Data into HDFS from SQL Server via Sqoop

2018-04-23
Sqoop

Write and Read Parquet Files in HDFS through Spark/Scala

2018-03-17
Spark & PySpark

Write and Read Parquet Files in Spark/Scala

2018-03-17
Spark & PySpark

Diagrams

ACID Support for Data Lake with Delta Lake, Hudi, Iceberg, Hive and Impala

This diagram summarizes the commonly used frameworks to build a data lake that supports ACID (Atomic, Consistency, Isolation, Durability). Apache Hive/Impala with ORC based transactional tables: storage format is ORC.Hive ACID Inserts, Updates and Deletes with ORC. Delta Lake: storage format is parquet with transactional JSON log files. Delta Lake with PySpark Walkthrough. Apache Hudi: storage format is parquet. Apache Iceberg: stored as parquet, ORC or Avro They have different implementation mechanisms but can all support schema evolutions and integrate with Hive meta catalog (metastore) and computing frameworks like Apache Spark, Trino, etc.

2022-09-02
Solution Diagrams