The world of computing has rapidly evolved over the years. We are now equipped with plenty of tools and software which is great for making our life a lot easier and more modern than ever before. Containers and container-orchestration systems have opened a brand new way of doing things, from allowing us to achieve automation of development and deployment to performing massive scale at fingertips and so much more. This post is an introduction to modernizing integration system software stack using a number of different tools and why they have been selected.
When it comes to building integration systems, below are the questions that often come into my mind.
What integration criterion am I trying to meet?
Answers to this question include application decoupling, integration simplicity, integration technology, data format, data timeliness, data or functionality and asynchronicity. Basically, the goal is to build an integration system that is less tightly couple and simple using up-to-date technologies. Well-accepted data formats such as XML, JSON or draw binary data can reduce the effort of producing and consuming messages, so this is important to consider. Components can interact with each other but how long it takes for them to exchange messages, can one component invoke the functionality of the others, does one component need to wait for others to complete their jobs before it can continue its own tasks.
What are the integration options do I have?
This kind of a huge topic but here I can think of the following:
- File transfer: does one application generate files so others can consume?
- Shared database: Are components in the whole integration system using the same data storage?
- Remote invocation: can one application call others remotely? (eg. via TCP, HTTP or another other network protocols)
- Messaging: does each component communicate through a common messaging system? (eg. Kafka, Event Hubs)
Does the ingratiation system need high availability?
How many instances of individual component it requires to ensure that the integration flow is not postponed?
The following discussion is based on my own experience and the way I build integration systems that is optimized for my organization. Different organizations have different criterion and resources. There are times using the most modern tools is not the most effective solution to certain problems so be sure to thoroughly analyze the requirements.
Any integration system needs a set of compute resources that allows cost-effective, high performance and reliability. The required resources depend on diffident types of integration but they can fall into the following categories:
- Networking (eg. VPN, bandwidth, routing, load balancing, network interfaces, etc)
- The underlying hardware (eg. physical servers, nodes, clusters, physical storage, RAM)
- The underlying software (eg. virtual nodes, virtual disks, frameworks, runtime environments)
- Security (eg. message encryption, authentication/authorization)
Traditionally, we would set and manage all these on our own premises but cloud vendors have made this much easier for us.
With cloud vendors such as Azure or AWS, you can spin up a set compute resources with just minimal steps. Deployment of these compute elements can now be done through tools such as Ansible or Terraform. They allow provisioning, configuration management, and application-deployment from code, enabling something called “infrastructure as code”. This post focuses on Azure but you can use AWS, GCP or any others. Containers are rapidly replacing physical and virtual servers because they save costs, and are essay to set up and manage. Most of my integration systems are now running in Docker containers, orchestrated by Kubernetes running in Azure Kubernetes platform. If you want find out how this was planned and deployed, see my other post discussing this approach.
Again, cloud vendors play a critical role in reducing the complexity of managing our integration workloads. In Azure, if one of the integration requirements is to enable high availability when the workload grows, we can achieve this through an availability zone or set. This is needed when the workload exceeds the underlying compute VMs’ capacity, resulting in unexpected downtime. However, this approach is still manual, a virtual machine scale set can be used to allow automatic scale out and in accordingly.
If you are using Kubernetes to manage your workload, there are often times you configure one of your node pools to be a virtual machine scale set so that you can automatically increase/decrease the number of the identical nodes when the workload grows and reduces.
Messaging and delivery
There are many integration patterns (eg. message endpoint, message channel, message router, etc) but the common one is messaging.
With messaging, we can use more powerful tools like Kafka or Event Hubs to deal with messages, they are not just publishing and consuming messages, they can also process those messages in a distributed, highly scalable, elastic, fault-tolerant, and secure manner before they arrive at the consumers.
Messages processors: Both Azure Durable Functions and Akka Actors can be used to process messages retrieved from any Kafka Topics or Event Hubs. With Akka streams, you can process millions of messages in parallel, Akka Streams is an extension of the existing Akka Actors so we can take advantage of it to perform stateful messaging.
Depending on the velocity of your data ingestion and egress and how data flows into your data store and how frequently it is being retrieved , different data stores offer different capabilities to meet your requirements. For instance, modern data warehousing services such as Snowflake and Microsoft Synapse allow you to spin up a number of nodes to process your data in parallel through a massive parallel processing engine (MPP). This is great for dealing with a large volume of data being written to and retrieved from the data store. Both allow you store semi structure (Json/Xml) and structured data (relational).
Here are a few questions you need to answer before choosing a data platform:
- What type of data are your storing? Is it based on flat files, semi-structure or structured data?
- Do you need to minimize administrative efforts?
- Do need to scale up and down your data store according to your workload?
- Do you need recovery for you data in case of a disaster?
Here are my answers to those questions:
- If your data is semi structured, choose a NoSQL data store (eg. Mongo DB, Cassandra, Azure Cosmo DB)
- If your data is flat files based, Azure Storage is efficient
- If you need to scale up and down your data store according to your workload, choose a managed service such as Azure SQL database, Azure Cosmo DB
- If you need data recovery in case of a disaster, choose a managed service such as Azure SQL database, Azure Cosmo DB through geo-replication and site recovery.
Any system needs to be monitored to detect any failures, causing the system to miss function or terminate. Monitoring is a continuous process based on the popular Build-Measure-Learn philosophy.
Monitoring can be at the following levels:
- Operating System
- Other interconnected components such as Networking Devices and Databases
Our integration systems fall into the application category. There are various tools to help us monitor our integration components. Azure Monitor or Elasticsearch Kibana is a centralized area for us to monitor both our infrastructure and services.
If you want to gain more insights into the application logs level, services like Azure Application insights is ideal for this purpose. Structured event loggers like Serilog, .NET Core logger are great to write custom logs and output them to our chosen platform such as SEQ, Azure Application Insight and Elasticsearch indices.