Streaming Big Data to Azure Synapse via Azure Functions

2021-12-24 azuresolution-diagram

This diagram shows a typical solution to stream big data to Azure Synapse:

  1. Data produced are streamed into Azure Event Hubs.
  2. Data in Event Hubs are captured in Azure Blob Storage via Event Hubs Capture feature.
  3. Once data capture is complete, an event is sent to Event Grid.
  4. Event Grid forwards the event information (with blob file path) to Azure Functions App (i.e. Event Grid triggered Functions App).
  5. Azure Functions App uses event data (blob path) to read the data.
  6. Azure Functions App loads data to Azure Synapse SQL data warehouse for analytics.
Event Hubs
[Not supported by viewer]
Storage Account
[Not supported by viewer]
Event Grid
[Not supported by viewer]
Functions App
[Not supported by viewer]
Synapse
[Not supported by viewer]
1
[Not supported by viewer]
2
[Not supported by viewer]
3
[Not supported by viewer]
4
[Not supported by viewer]
5
[Not supported by viewer]
6
[Not supported by viewer]
Data ingestion
[Not supported by viewer]
Capture data into Blob Storage
[Not supported by viewer]
Capture event
[Not supported by viewer]
Capture event data forwarding
[Not supported by viewer]
Retrieve data
[Not supported by viewer]
Data loading into data warehouse
[Not supported by viewer]
Icon-analytics-144 public:true sdk:MsPortalFx.Base.Images.Polychromatic.EventHub() category: Analytics, IoT image/svg+xml Icon-analytics-144 Icon-storage-86 public:true sdk:MsPortalFx.Base.Images.Polychromatic.Storage() category: Storage image/svg+xml Icon-storage-86 Icon-integration-221 public:true sdk:false category: Integration image/svg+xml Icon-integration-221 Icon-compute-29 public:true sdk:MsPortalFx.Base.Images.Polychromatic.Functions() category: Compute, IoT image/svg+xml Icon-compute-29 public:true sdk:false category: Databases, Analytics image/svg+xml