arrow_back Spark - Save DataFrame to Hive Table

access_time 10 months ago link more_vert
#1707 Re: Spark - Save DataFrame to Hive Table
HI  Reymond, Although hive/hudi supports ACID but we cannot spark.sql() dont support update ,merge or delete is there any workaround for submitting DML .

person Raymond access_time 10 months ago
Re: Spark - Save DataFrame to Hive Table

Hive 3 does support ACID if table data is stored as ORC format and we can use INSERT, UPDATE and DELETE statements to manipulate the data as you can do with other databases. You can use SparkSession.sql to invoke these statements if necessary. 

Hudi and DeltaLake both support ACID too and can be used when working with data lake directly, for example, data stored as hudi and delta format in HDFS or S3, GCS, Azure Blob Storage, etc. 

recommendMore from Kontext