Solution Diagrams
Diagrams
Agent2Agent vs. MCP (Model Context Protocol)
References: A2A ❤️ MCP \* Agents can be part of your agentic application or from third parties.
Building RAG Application with Embedding Models
This diagram illustrates how to build a RAG application with embedding models. Reference Embedding models · Ollama Blog
Semantic Search with FAISS
This diagram illustrates how to build a semantic search engine with FAISS (Facebook AI Similarity Search). Reference Semantic search with FAISS - Hugging Face NLP Course
Text to SQL using duckdb-nsql model via ollama
DuckDB-NSQL is a text-to-SQL model with 7 billion parameters, specifically created for generating SQL queries. It builds upon Meta’s original Llama-2 7B model, undergoing additional pre-training using a dataset of general SQL queries. Subsequently, it is fine-tuned on a dataset containing DuckDB text-to-SQL pairs. This diagram shows you how to use ollama Python client library to call REST APIs to convert text to SQL.
Foundations of Vector Databases
This diagram shows the working principle of a vector database. Vector databases can be used in building a complex RAG (Retrieval Augmented Generation) application.
AI based Chatbot with LLaMA Models and Microsoft Semantic Kernel
This diagram illustrates an idea of implementing an AI based chatbot for Kontext on Azure. It leverages open source frameworks and LLMs (large language models) to implement a RAG (retrieval augmented generation). The chatbot will get the domain specific information from Kontext search engine as context information and provided it into backend API service which leverages Microsoft Semantic Kernel to construct the question and send to LLM service to generate the answers and then sends back to frontend. Alternatives In the diagram, LLMs can be replaced by other models like Open AI services of Azure Open AI services or any other relevant language models. Microsoft Semantic Kernel can also be replaced with LangChain too. References microsoft/SemanticKernelCookBook: This is a Semantic Kernel's book for beginners (github.com) Llama2Chat | 🦜️🔗 Langchain SciSharp/LLamaSharp: Run local LLaMA/GPT model easily and fast in C#!🤗 It's also easy to integrate LLamaSharp with semantic-kernel, unity, WPF and WebApp. (github.com) blazorly/LangChain.NET (github.com)
Azure Serverless Architecture for Web/Mobile Applications
This diagram shows you how to implement a serverless architecture on Azure. Reference Serverless web application - Azure Architecture Center | Microsoft Learn
Forward Proxy and Reverse Proxy
This diagram shows how forward proxy and reverse proxy works. Forward proxy is commonly used to block access to certain content by companies and institutions while reverse proxy servers commonly as load balancer and caching purposes. References YARP Documentation What is a reverse proxy?
OAuth 2.0 and OIDC (OpenID Connect)
This diagram shows the relationships between OAuth and OIDC. OAuth is an consent and token based authorization standard/framework which allows third-party applications to access data in one application without sharing password and other credentials. OIDC builds on top of that with additional features like identity token (JWT) and profile information. OIDC is commonly used in SSO (Single Sign-On) scenarios. References An Illustrated Guide to OAuth and OpenID Connect | Okta Developer
Snowflake Object Type Hierarchy
This diagram summarizes the basic concepts in Snowflake: Accounts Database Schema Tables Views Stages File Formats Sequences Stored Procedures UDF(s) Streams Tasks
ACID Support for Data Lake with Delta Lake, Hudi, Iceberg, Hive and Impala
This diagram summarizes the commonly used frameworks to build a data lake that supports ACID (Atomic, Consistency, Isolation, Durability). Apache Hive/Impala with ORC based transactional tables: storage format is ORC.Hive ACID Inserts, Updates and Deletes with ORC. Delta Lake: storage format is parquet with transactional JSON log files. Delta Lake with PySpark Walkthrough. Apache Hudi: storage format is parquet. Apache Iceberg: stored as parquet, ORC or Avro They have different implementation mechanisms but can all support schema evolutions and integrate with Hive meta catalog (metastore) and computing frameworks like Apache Spark, Trino, etc.
Delta Lake Architecture
This diagram shows the architecture of Delta Lake. Delta Lake is an open-source storage framework that can be use to build a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive. It publishes APIs for Scala, Java, Rust, Ruby, and Python.
Spark Application Anatomy
This diagram depicts the relationships among Spark application, job, stage and task. One Spark application can contain multiple actions and each action will be related to one Spark job; to run the computation within a job, multiple stages might be involved as some actions cannot be done within just one stage; each stage will include many tasks and the task count is decided by the total partitions in the RDD/DataFrame. Task is a lowest parallelism unit in Spark.
PySpark Reading from S3
This diagram is used as article feature images, which depicts reading data from S3 bucket via PySpark.
SSL Forward Proxy
This diagram shows how a MITM (man-in-the-middle) firewall works as a SSL forward proxy.
Azure Cache Redis as Session Store for ASP.NET Core Application
When use Cookie authentication schema in ASP.NET core applications, session data by default is saved in client as Cookies. If the session data (incl. user claims) are big (more than 4090 characters), it can be split into multiple chunks. You can notice that via application identifier cookie: .AspNetCore.Cookies: chunks-2 .AspNetCore.CookiesC1: XXX .AspNetCore.CookiesC2: XXX Instead of storing session data in client cookies, we can also store the session data in any store that implements ITicketStore. It can be databases, memory cache or distributed memory cache like Redis. This diagram shows a simple solution of using Azure Cache Redis as session store. The following links can be referenced to implement a solution like this. References Cookie size and cookie authentication in ASP.NET Core - Honza’s Blarg (hajekj.net) ChunkingCookieManager Class (Microsoft.AspNetCore.Authentication.Cookies) | Microsoft Docs Session in ASP.NET Core | Microsoft Docs Matteo's Blog - Implementing a custom ITicketStore for ASP.NET Core Identity (ml-software.ch) Using Redis Cache for session data storage in ASP.NET Core - Joonas W's blog
AWS EMR Read and Write with S3
This diagram shows a typical EMR application that reads and writes data with S3. References EMR File System (EMRFS) - Amazon EMR
Spark Partitioning Physical Operators
This diagram shows how Spark decides which repartition physical operators will be used for each scenario. `` repartition(numPartitions, *cols) ``
Spark Memory Management Overview
This diagram shows an overview of Spark memory management when running in YARN. It helps you to understand how your Spark memory is allocated and how they are used. In Spark executor, there are two types of memory used: Execution memory - refers to that used for computation in shuffles, joins, sorts and aggregations; Storage memory - refers to that used for caching and propagating internal data across the cluster. When no storage memory is used, execution can use all the available memory and vice versa. These two types of memory usage are decided by two configuration items: spark.memory.fraction expresses the size of M as a fraction of the (JVM heap space - 300MiB) (default 0.6). The rest of the space (40%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors. spark.memory.storageFraction expresses the size of R as a fraction of M (default 0.5).
Diagram - Connect to HBase in Python
This diagram illustrates connecting to HBase using Python HappyBase package.
.NET Full Text Search on Azure using Apache Lucene
This diagram shows how to use Apache Lucene to implement full text search engine for Azure App Service ASP.NET core websites. Apache Lucene is an open source framework to provide world search capabilities. It is the search core of both Apache Solr™ and Elasticsearch™. The framework provide core libraries for Java, Python (PyLucene) and .NET.
AWS ETL Solution with Glue Diagram
This diagram shows one example of using AWS Glue to crawl, catalog and perform data stored in S3. Data landed in raw bucket is scanned by Glue Crawler and the metadata is stored in Glue Catalog. Glue ETL job loads the raw data and does transformations and eventually store the processed data in curated bucket. The processed files are scanned by Glue Crawler. Processed data is then queried by Amazon Athena. The data can be further utilized in reporting and dashboard.
Azure Automation - Call Stored Procedure in Azure SQL Periodically
This diagram shows the idea of using Azure Automation PowerShell Workflow runbook to call a stored procedure or execute other T-SQL in a Azure SQL database. For Kontext, this mechanism is used to regularly update tag, user, site and other stats.
AWS Streaming Processing Diagrams
This diagram is used as feature image for AWS streaming processing diagram series.
AWS Batch Processing Diagrams
This diagram is used as feature image for AWS batch processing diagram series.
AWS Big Data Lambda Architecture for Streaming Analytics
This diagram shows a typical lambda streaming processing solution on AWS with Amazon Kinesis, Amazon Glue, Amazon S3, Amazon Athena and Amazon Quicksight: Amazon Kinesis - capture streaming data via Data Firehose and then transform and analyze streaming data using Data Analytics; the result of analytics is stored into another Data Firehose process; for batch processing, the captured streaming data can be directly loaded into S3 bucket too. Amazon S3 - store streaming raw data and batch processed data. Amazon Glue - transform batch data in S3 and store the processed data into another bucket for consumption. Amazon Athena - used to read data in S3 via SQL. Amazon Quicksight - data visualization tool. References AWS IoT Streaming Processing Solution Diagram AWS IoT Streaming Processing Solution Diagram w Glue
AWS IoT Streaming Processing Solution Diagram w Glue
This diagram shows a typical streaming processing solution on AWS with Amazon Kinesis, Amazon Glue, Amazon S3, Amazon Athena and Amazon Quicksight: Amazon Kinesis - capture streaming data via Data Firehose and then load the data to S3. Amazon S3 - store streaming raw data and batch processed data. Amazon Glue - transform batch data in S3 and store the processed data into another bucket for consumption. Amazon Athena - used to read data in S3 via SQL. Amazon Quicksight - data visualization tool. Similar solution diagram using streaming transformation: AWS IoT Streaming Processing Solution Diagram.
AWS IoT Streaming Processing Solution Diagram
This diagram shows a typical streaming processing solution on AWS with Amazon Kinesis, Amazon S3, Amazon Athena and Amazon Quicksight: Amazon Kinesis - capture streaming data via Data Firehose and then transform and analyze streaming data using Data Analytics; the result of analytics is stored into another Data Firehose process. Amazon S3 - streaming processed data is stored in Amazon S3. Amazon Athena - used to read data in S3 via SQL. Amazon Quicksight - data visualization tool.
AWS Batch Processing Solution Diagram (using AWS Glue)
This diagram shows a typical batch processing solution on AWS with Amazon S3, AWS Lambda, Amazon Glue and Amazon Redshift: Amazon S3 is used to store staging data extracted from source systems on-premises or on-cloud. AWS Lambda is used to register data arrival in S3 buckets into ETL frameworks and trigger batch process process. Amazon Glueis then used to integrate data like merging, sorting, filtering, aggregations, transformations and load the data. Amazon Redshift is then used to store the transformed data. This diagram is forked from AWS Batch Processing Solution Diagram
AWS Batch Processing Solution Diagram
This diagram shows a typical batch processing solution on AWS with Amazon S3, AWS Lambda, Amazon EMR and Amazon Redshift: Amazon S3 is used to store staging data extracted from source systems on-premises or on-cloud. AWS Lambda is used to register data arrival in S3 buckets into ETL frameworks and trigger batch process process. Amazon EMR is then used to transform data like aggregations and load the data. Amazon Redshift is then used to store the transformed data. This pattern follow the traditional ETL pattern and you can change it to ELT pattern too to do transformations in Redshift directly. Amazon EMR can be replaced with many other products.
Data Distribution Approaches in Parallel Computing System
This diagram shows the typical algorithms to distribute data into a cluster for processing or computing. They are commonly used in systems like Teradata, SQL Server PWD, Azure Synapse, Spark, etc. Replicated- table are replicated to each node. This is useful to distribute small tables like reference tables to join with big tables. Round-robin distributed - data is randomly distributed to the nodes in the cluster using round-robin algorithm. This is useful for big tables without obvious candidate join keys. This will ensure data is evenly distributed across the cluster. Hash distributed - data is distributed using deterministic hashing algorithm on the key values. Same value will guarantee to be distributed to the same node. This is the most commonly used distribution approach for big tables. What are the other distribution algorithms you have used?
Process Automation Approaches on Azure
There are several typical ways to automate processes on Azure: Azure Automation Account runbooks (PowerShell or Python) Azure Functions (with triggers or schedules) Azure Logic App (Power Automate is built on top of it) Azure App Service WebJobs For differences, refer to this article: Choose the right integration and automation services in Azure. For data processes, you can use many other tools like Azure Data Factory, Databricks, etc. What other tools or products you use for automating your Azure processes? What are the other features that differentiate these products from each other? You can post a comment to share you ideas.
Apache Ambari Cluster and Hadoop Cluster
This diagram shows an sample setup of using Apache Ambari to provision a Hadoop cluster. \*Apache Ambari and Hadoop logos are registered trademarks of Apache Software Foundation.
Azure App Service Logging Provider
This diagram shows how to use Azure Blob storage (append blob) as storage for Azure App service. Extension package Microsoft.Extensions.Logging.AzureAppServices is used.
Kontext 2021 Infographic
This infographic shows the stats of Kontext in 2021 about its users, page views, top contents, etc.
Azure Virtual Network Example
This diagram shows an Azure Virtual network example that integrates Azure App services and Azure SQL database.
Go Programming [Banner]
Go programming banner for Column articles.
Azure Infrastructure as Code (IaC)
This diagram shows Infrastructure as Code tools for Azure: ARM (Azure Resource Manager) template Bicep template (covert to ARM template via Bicep CLI) Terraform References Azure Bicep playground
Streaming Big Data to Azure Synapse via Azure Data Factory
This diagram shows a typical solution to stream big data to Azure Synapse: Data produced are streamed into Azure Event Hubs. Data in Event Hubs are captured in Azure Blob Storage via Event Hubs Capture feature. Once data capture is complete, an event is sent to Event Grid. Event Grid forwards the event information (with blob file path) to Azure Factory Pipeline (i.e. Event Grid triggered pipeline). Azure Data Factory uses event data (blob path) as source. Azure Data Factory sinks data to Azure Synapse SQL data warehouse for analytics.
Streaming Big Data to Azure Synapse via Azure Functions
This diagram shows a typical solution to stream big data to Azure Synapse: Data produced are streamed into Azure Event Hubs. Data in Event Hubs are captured in Azure Blob Storage via Event Hubs Capture feature. Once data capture is complete, an event is sent to Event Grid. Event Grid forwards the event information (with blob file path) to Azure Functions App (i.e. Event Grid triggered Functions App). Azure Functions App uses event data (blob path) to read the data. Azure Functions App loads data to Azure Synapse SQL data warehouse for analytics.
Azure Event Grid, Event Hubs, Service Bus
Are you confused about the differences between Azure messaging services such as Event Grid, Event Hubs and Service Bus? This diagram summarizes some of the key features of these products to help you understand them easily. It's also important to understand the differences between events and messages. One key difference is that message publisher and consumer usually has established some contracts and when the message is delivered, the publisher usually expects certain response from the consumer.
Azure Automation Account - Runbooks
This diagram shows some of the key components in Azure Automation Account: Runbooks Schedule Connections Credentials Certificates
Python Libraries to Connect to SQL Server
There are several different ways to connect to SQL Server in Python: JDBC (py4j), for example, JayDeBeApi ODBC (pyodbc) FreeTDS (pymssql)
Nested folder with Parquet files
This diagram shows parquet files in nested folders.
Spark Partition Discovery
Spark supports partition discovery. All built in file sources (Text/CSV/JSON/ORC/Parquet) supports partition discovery and partition information inference. This data shows a example data set that is stored by two partition levels: month and country. The following code snippet will read all the underlying parquet files: `` df = spark.read.option("basePath","/data").parquet("/data") ``
Data Engineering - Transactional Extract
When extracting data from source databases, there are usually three approaches: Full extract - extracts all the records from the source table. This usually applies to small to medium size tables, for example, product tables. Transactional extract - extracts all the new records from the last extract. This is good for transactional sources like banking transactions, logs, etc. Delta extract - extracts all the changed records including updated, new and deleted ones. This can be used to reduce the amount of data that needs to be transferred from source system to target system. This usually involves CDC (change data capture). This diagram shows the high-level flow of transactional extract. Usually you will use your ETL (Extract, Transform, Load) framework to record last extract point, for example, date, timestamp or auto-increment column value.
Data Engineering - Full Extract
When extracting data from source databases, there are usually three approaches: Full extract - extracts all the records from the source table. This usually applies to small to medium size tables, for example, product tables. Transactional extract - extracts all the new records from the last extract. This is good for transactional sources like banking transactions, logs, etc. Delta extract - extracts all the changed records including updated, new and deleted ones. This can be used to reduce the amount of data that needs to be transferred from source system to target system. This usually involves CDC (change data capture). This diagram shows the high-level flow of full extract.
Data Engineering - Delta Extract
When extracting data from source databases, there are usually three approaches: Full extract - extracts all the records from the source table. This usually applies to small to medium size tables, for example, product tables. Transactional extract - extracts all the new records from the last extract. This is good for transactional sources like banking transactions, logs, etc. Delta extract - extracts all the changed records including updated, new and deleted ones. This can be used to reduce the amount of data that needs to be transferred from source system to target system. This usually involves CDC (change data capture). This diagram shows the high-level flow of delta extract. It is very important to include the file for deleted records unless your data source never deletes records. If your source team cannot provide the deleted records, you will need to request a full extract so that you can work out the changes by comparing with your target tables.
Static and Dynamic Data Masking
Static data masking will replace sensitive data at rest and this operation is usually irreversible. This is usually used to build data warehouse or data mart for analytical purposes where sensitive data is not required; it is also used to generate DEV and TEST environments using production data. Dynamic data masking will replace sensitive data in transition. The original data stored in the database will not be altered. When users issue query to retrieve the data, dynamic data masking process or engine will mask the data based on settings and users' access or roles. References Dynamic data masking (Azure SQL Database)
Scale Up/Down and Scale Out/In
Scale up/down increases/decreases a component's capacity to handle larger or smaller workload. For example, increasing a virtual machine from 1GB memory to 32GB memory. Scale out/in means adding more or less nodes to a system/cluster to increase or decrease processing capacity.
Azure Virtual Network Peering
Virtual network peering allows you to connect two or more virtual networks seamlessly. Azure supports two types of peering: Virtual network peering: Connect virtual networks within the same Azure region. Global virtual network peering: Connecting virtual networks across Azure regions. References Virtual network peering
Mount Azure Storage Volumes to Container Group
Azure Files can be mounted to container groups in Azure Container Instances or App Services. To avoid latency, it is good practice to place the application containers in the same region as file storage.
Azure Kubernetes Service Components
This diagram shows you the key components and concepts in a Azure Kubernetes Service (AKS) cluster. In AKS, nodes (virtual machines) of same configuration are groups into pools. A node is a virtual machine in the cluster. A deployment represents identical pods. It defines the number of pods replica to deploy. A pod represents a single instance of your application. Usually one pod has one container though one pod can have multiple containers.
Kontext Diagrams Logo
Kontext Diagrams Logo to be used in Diagrams related arcticles.
Kontext Cloud Diagram Example
This diagram is created for testing purpose to validate whether the software can draw diagrams with Azure, GCP and AWS product SVG icons correctly.