aws-batch-processing
2 items tagged with "aws-batch-processing"
Diagrams
AWS Batch Processing Solution Diagram (using AWS Glue)
This diagram shows a typical batch processing solution on AWS with Amazon S3, AWS Lambda, Amazon Glue and Amazon Redshift: Amazon S3 is used to store staging data extracted from source systems on-premises or on-cloud. AWS Lambda is used to register data arrival in S3 buckets into ETL frameworks and trigger batch process process. Amazon Glueis then used to integrate data like merging, sorting, filtering, aggregations, transformations and load the data. Amazon Redshift is then used to store the transformed data. This diagram is forked from AWS Batch Processing Solution Diagram
AWS Batch Processing Solution Diagram
This diagram shows a typical batch processing solution on AWS with Amazon S3, AWS Lambda, Amazon EMR and Amazon Redshift: Amazon S3 is used to store staging data extracted from source systems on-premises or on-cloud. AWS Lambda is used to register data arrival in S3 buckets into ETL frameworks and trigger batch process process. Amazon EMR is then used to transform data like aggregations and load the data. Amazon Redshift is then used to store the transformed data. This pattern follow the traditional ETL pattern and you can change it to ELT pattern too to do transformations in Redshift directly. Amazon EMR can be replaced with many other products.