AWS ETL Solution with Glue Diagram

2022-01-29 awsdata-engineeringsolution-diagram

This diagram shows one example of using AWS Glue to crawl, catalog and perform data stored in S3.

  1. Data landed in raw bucket is scanned by Glue Crawler and the metadata is stored in Glue Catalog.
  2. Glue ETL job loads the raw data and does transformations and eventually store the processed data in curated bucket.
  3. The processed files are scanned by Glue Crawler.
  4. Processed data is then queried by Amazon Athena. The data can be further utilized in reporting and dashboard.
AWS Glue Crawler
[Not supported by viewer]
Amazon S3
Raw Data
[Not supported by viewer]
AWS Glue
Data Catalog
[Not supported by viewer]
AWS Glue ETL Job
[Not supported by viewer]
Amazon S3
Curated Data
[Not supported by viewer]
Amazon Athena
[Not supported by viewer]
Amazon Quicksight
[Not supported by viewer]
1
[Not supported by viewer]
2
[Not supported by viewer]
3
[Not supported by viewer]
4
[Not supported by viewer]