Data Engineering - Transactional Extractimage

visibility 16 access_time 7 months ago languageEnglish

When extracting data from source databases, there are usually three approaches:

  • Full extract - extracts all the records from the source table. This usually applies to small to medium size tables, for example, product tables. 
  • Transactional extract - extracts all the new records from the last extract. This is good for transactional sources like banking transactions, logs, etc. 
  • Delta extract - extracts all the changed records including updated, new and deleted ones. This can be used to reduce the amount of data that needs to be transferred from source system to target system. This usually involves CDC (change data capture). 

This diagram shows the high-level flow of transactional extract. Usually you will use your ETL (Extract, Transform, Load) framework to record last extract point, for example, date, timestamp or auto-increment column value. 

copyright This page is subject to Site terms.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

comment Comments
No comments yet.
timeline Stats
Page index 0.09
local_offer Tags

info Info
Image URL
SVG URL
URL
More from Kontext
[Diagram] Data Engineering - Full Extract image
visibility 29
thumb_up 0
access_time 7 months ago
Data Engineering - Full Extract
[Diagram] AWS ETL Solution with Glue Diagram image
visibility 189
thumb_up 0
access_time 5 months ago
AWS ETL Solution with Glue Diagram