What is integration?
Integration - or in this context, system integration - is the interconnection of one or more systems by transferring information from one to another or many. For example, in addition to the Customer Relationship Management (CRM) system, the new marketing automation system needs customer information as well. So does the enterprise resource planning (ERP) system. This flow of customer information to another system is called a data stream. This article covers the basics of system integration as well as the latest winds in the integration field.
Controlled integration or data flow spaghetti?
Maturity level 0: point-to-point integrations, no integration platform
Point-to-point integration is an antipattern where integrations are made directly from one system to another. They are developed with the "best" technology of each era and deployed into a corner of a random server. Sometimes they are hosted in the integrated system itself.
Henry, the Heroic coder from the IT department, used to do these with PHP, after which Cora, the Cowboy coder, continued with his Python scripts. Both scheduled their scripts with separate Cron instances somewhere in the jungle of different servers. Over the years, tens or hundreds of integration point-to-point scripts accumulate. Think of one such script that executes a single data stream as one spaghetti ribbon. The consequence of this kind of hero integration is a network resembling a heap of spaghetti, to which making changes has unforeseen consequences. Also, changing one system from this spaghetti is surprisingly tricky. The integration links to and from the replaced system break and not all of the connections are even known — as Henry and Cora were notoriously lazy documenters of their code. And most importantly - centralized control of data flows is impossible.
Information is often compared to oil or water, as it is one of the most valuable assets of the business. Information is a critical part of running the business, the commercial secret of a company, or a valuable item to sell. Centralized control and management of these information flows are one of the most critical tasks in a modern business-driven IT organization. In a modern IT environment, troubleshooting and finding the problem in a process consume most of the time. For example, a CRM vendor says customer information has left the CRM system, and the vendor that is supplying the marketing automation as-a-service claims it never arrived. Centralized monitoring and decentralized execution of integration processes solve these problems. According to Gartner, centralized integration saves as much as 50% on IT costs compared to spaghetti integration. The problem with spaghetti integrations increases as the number of systems increases. In the era of cloud computing services, cooking integration spaghetti is easier than ever.
The IT experts, Cora and Henry, are also part of the problem. As they retire, nobody knows what those hero-coded scripts are doing, where they are running, and why they are running.
However, point-to-point integrations are not just the IT gimmick - in some cases, they are justifiable solutions. These include, e.g., Temporary integrations or data streams in which different modules of the same vendor system speak to one another, and the vendor is responsible for the whole entity. In other cases - the centralized integration platform described in the next section is the right choice
Why a centralized integration platform?
Maturity level 1: Centralized integration control, harmonized implementation.
A modern integration platform is a hosting environment for data flows. At the same time, it is a controller of these data flows. As its name implies, it is the platform - not the final solution that just works in a plug-n-play manner. Integration platforms and their functional sets are diverse and therefore challenging to compare. However, they have one common feature - they act as a central point of control, monitoring, and management for integrations and data flows. Centralized integration also significantly reduces the amount of data flows, as integrations pass through the integration platform on a one-to-many basis. This central point is the best point for controlling and monitor data flows.
The role of the central integration platform is to transfer data between all of the enterprise systems while handle the necessary data conversion, often referred to as data mapping. It means converting various data structures, such as XML or JSON, into another basic format like edifact. The term protocol conversion refers to, for example, first fetching a file with SFTP and then sending the payload as a JSON in a REST call.
A centralized integration platform typically also can schedule transfers or trigger an integration process execution from, for example, an incoming file, a database change, or an incoming REST / JSON call. This launching a reaction from something that is happening is triggering. A more sophisticated form of triggering is an integration platform with capabilities to implement event-driven architecture.
In centralized integration, a module capable of interfacing a particular system is called an adapter or connector; for example, most of the integration platforms have SAP connectors. Quite often, these adapters also implement a standard. For example, an integration platform may have an SFTP adapter just to comply with old legacy systems that can handle only files.
The centralized integration platform also provides a single implementation method for integrations. Unified platform and methodology lead to easier knowledge management and prevent a vital understanding of dataflows to create a person risks.
How do I make process automation on an integration platform?
Maturity 2: Process automation on a centralized integration platform.
Some existing iPaaS solutions also include process automation features and capabilities. The orchestration consists of steps like data transformations, complex logic, and calls to APIs or connectors. This way, systems are integrated as part of process automation. The integration platform is capable of automating processes many orders of magnitude more efficiently than, for example, the RPA tool itself, which also has process orchestration capabilities. The integration platform is faster in automation because it does not simulate user actions but works directly - often as compiled code - at the interface level. On the other hand - it is not possible to use an integration platform for automation if there are no interfaces available in the automation target system. Integration platforms cannot do process automation that occurs inside one user interface. Modern integration platforms include the ability to orchestrate processes and workflows in standard languages. For example, the FRENDS integration platform supports BPMN (Business Process Model Notation), which is a notation supported by Microsoft Visio and other modelling tools.
The image above is an illustration of a process that includes a robot as a part of the orchestration. The screencap is directly from Frends -integration platform that follows BPMN -standard as the orchestration and integration development language. The right way of process automation is not to just look Robotic part of it, but to include the whole business process automation and orchestration to it. You can get the best of both worlds.
Hyperautomation
When RPA meets BPA and the BPA process is using Machine Learning to make choices and decisions as humans would, we can speak about hyperautomation. Hyperautomation is a technology trend that Gartner estimates to be the number one trend in IT in 2020 and also present in top10 in 2021. Integration Platforms like Frends are essentially Hyperautomation Platforms including BPA and API features built-in.
What is the role of the service bus?
Maturity Level 3: Service bus-based integrations and internal interfaces or APIs
Integration platforms containing an enterprise service bus as-a-feature are capable of different message delivery methods. The bus usually includes a queue system to which messages can persist. Message persistence means that the sender can trust that the message - whether an electronic order or application from a webpage - has been permanently stored in the queue system. The sender can trust persistence even though it has not yet reached the system it is eventually going to. Message queue (MQ) is not always an integral part of the integration platform.
For example, the FRENDS integration platform includes the queue system itself. On the other hand, FRENDS is capable of using other queue systems because it supports the AMQP standard for queue communication.
Message queues - either individual software components or as part of an integration platform - still have their use cases, even though the world is increasingly moving towards event-based and synchronous messaging. Entering the event-based world means that messages - such as sales messages from the cash register to the central system - are no longer queued. Instead of queuing, Messages can be sent directly to the integration platform's REST / JSON interfaces, which at best synchronize them directly to the receiving systems. If there is a load spike - for example, one hundred times the number of sales messages during the Christmas sales - scaling both the integration platform and the receiving line-of-business system or microservices is an event-driven and synchronous solution. This kind of synchronous solution might require support for container technology, which is a topic of another chapter in this article. Conversely, if the receiving system is a monolith, the integration platform can buffer messages or queue them in the traditional model. Support for container technology is not an established feature in integration platforms, and the need for it should be considered when choosing an integration platform.
Also, the service bus implements publish-subscribe messaging. Systems that want information and messages to themselves can register themselves as listeners to topics or content. The service bus can route messages to the listener based on the topic of the message, for example, "all purchase orders." As an example of content-based routing, a CRM may be notified to listen to all messages containing a customer change and may come from multiple sources. Or, purchase order messages can be listened to on multiple systems, where the bus routes them and converts them both to protocol and message structure recipients.
However, the enterprise service bus as a whole is not a necessary part of the integration platform but is often present in the system integrations of larger organizations
The integration platform is the engine of the API-economy
Maturity Level 4: Internal and Public APIs and OpenAPI
Integration platforms, such as frends iPaaS, can act as hosting environments and development platforms for APIs both internally and externally. External in this context means that API is, for example, publicly available on the internet. During development, Integration platform connectors and adapters connect to the legacy systems such as ERP systems or cloud systems such as SalesForce CRM. These systems and the aggregated data from these systems can be used to publish business functions or data sharing services in a variety of technologies and ways. Today, the most common and recommended of these are the methods and technologies defined by the OpenAPI standard. The published APIs may be, for example, an input channel for messages from the previous paragraph. Equally, a public interface implemented on an integration platform can be synchronously connected to backend systems without a bus.
OpenAPI is a standard and, for example, Swagger is an example of its implementation: a set of tools for implementing this standard, which is also used by the FRENDS integration platform mentioned in this article. OpenAPI ensures that the API is always described and documented in the same way and is easy to test. OpenAPI is often referred to as the "Public API." Be careful: when talking about a public API, whether it is an API implementing the OpenAPI standard or an API published on the public Internet. The latter need not strictly implement the OpenAPI standard to be public, that is, publicly available.
Internal APIs
In the image below, an example from a real taxi company's client search services or custom editing services - APIs - can be implemented on an integration platform as REST / JSON interfaces. Before the REST / JSON time, these interfaces were implemented in SOAP / XML and talked about web service. Even if the interfaces in the image implement the OpenAPI standard, they cannot be called public interfaces. They are just implementing a standard that makes them easier to use. Still, they are not publicly available on the other side of the firewall on the open Internet. Internal interfaces can be called by systems implemented at different times and are deployed in each system deployment project, for example, a new ERP. There is already a clear benefit: the system below, such as its ERP, can be changed so that calling systems do not notice the change. On the other hand, search services are reusable: you don't have to do the same integration job twice. This kind of service layer is called Service Oriented Architecture (SOA).
External APIs
Let's continue to use this taxi company as an example for external APIs. The company is deciding to develop a new mobile app to serve its customers better to compete with Uber as an ultra quality taxi service. Information on free cars, drivers, and locations already exist in the legacy systems that have been there for ages. The old architecture of implementing this kind of mobile app would be using developers doing the front and back in the cloud. If you have a potent enterprise integration service-as-a-platform, you don't need a separate back-end at all. Modern eiPaaS handle the low latency requirements as well; they handle the need to implement user sessions within the services.
In the first stage, the taxi company developed a group of services: newOrder, getOrder, changeOrder, getCustomer, and changeCustomer. These were all internal APIs, followed by OpenAPI -standard and used within internal projects. Then the mobile app required the same functionalities - and so they were published within another Frends agent on the internet. These services were accessible only for their internal mobile apps. A modern integration architecture has built-in message queues that ensure that, for example, a DDoS -attack wouldn't endanger the internal systems.
Now, as they made their OpenAPI -standard-based APIs publicly available, they can be found and accessed by any developer. For example, real estates that have a reception with an "order taxi" -button, are direct calls to the same API with static address information. Third-party event-manager that are implementing their customer engagement mobile app can find the same API as well: "order taxi to my event." This way, APIs formulate new channels for sales.
API Management
When interfaces are built on an integration platform or built directly on a platform service such as AWS, managing them becomes the next challenge. How can I get other developers to find my public API? How do I make sure it's securely available on the public network? For this, there is an area of expertise: API Management. You can purchase a platform for managing these APIs only, such as RedHat 3Scale. Then API's hosting environment, the implementation, and the management of the APIs are separated.
On the other hand, modern integration platforms provide very comprehensive API management capabilities. For example, the FRENDS integration platform has API management features for APIs developed and hosted inside FRENDS, as well as APIs developed elsewhere.
API management divides into two main parts: the API Portal and the proxy gateway. These sections do not contain the implementation of the API itself, but are the API's management and publishing layer. On this portal, third-party developers can find a published API, say, a Google search and program the app to use it (in the illustration "Applications").
Modern integration platforms can serve as API execution and creation platforms, as well as API intermediation services and portals.
What are SOA, Micro Services, and Mini Services?
SOA (Service Oriented Architecture) was a development in the late 1990s. The idea was to create reusable business functions or more commonly known as services. A service "orderTaxi," such as the one above, is also a general service based on SOA thinking. Early SOA-based services often bloated too much. The service itself grew impossibly large already on the design table. Their development could often be so slow that businesses could not wait, and their needs became obsolete before the massive service could be deployed into use.
Architects solved this problem by rethinking the granularity of service to be a small exact business function. At the same time, scaling resources on the cloud became available. Scaling a service without limitations means that the service can not bind to a legacy or a line-of-business system. The coupling is the bottleneck for scaling the service as the line-of-business system probably can not be scaled similarly to an SOA-service. Therefore the architects defined that a service should own its data - for example, its database scales as the service scales. A fully uncoupled microservice was born.
Microservice architecture (MSA) is also a developer and technology independent. It is often thought that microservices must be on platforms like Azure or AWS. Still, they can equally be developed on an integration platform that can fulfill the microservice definition. Many organizations, such as Zalando, have built their business functions into microservices - it is a form of application development, not a way of integration. Many organizations have relied on microservices for their e-commerce or other e-commerce services, but have forgotten why ready-made software exists in the first place. Depending on the MSA, companies have to build all the monitoring, audit-trail, operative support, and management features for the services they create based on MSA. The DevOps team that implemented the MSA often tends to turn out to be the Ops team for the services.
Decades ago, a ready-made software for specific purposes was invented to solve this problem - now we are going back to coding everything? Choosing the right pattern is crucial. If your business relies on a technology that doesn't exist or there is no ready-made software available, choose MSA. But be careful, there is always the cost-of-code when you develop something you have to upkeep the next ten years.
The latest integration platforms, like frends eiPaaS, support microservices and miniservices. The considerable benefit of the frends approach is the centralized operations, monitoring, deployment, versioning, test automation, and other management for both micros and mini services.
What is containerization?
Containerization refers to the virtualization of applications or services into small and independent packages. Such a container is loaded with all the files and program components needed by the application. The container includes all the configuration settings of the application. A single container may contain a single service like "orderTaxi" of an earlier taxi example, or a set of services in one execution engine. Containers exist in a separate container environment, such as a well-established Docker environment or a popular RedHat OpenShift environment.
Containerization has the following benefits:
- When an application crashes, the container environment can simply restart the container with "factory settings" in seconds.
- Containers are scaled automatically based on need. For example, during rush hours, the "orderTaxi" service can be scaled in containers based on the increasing number of requests. The container environment, such as the previously mentioned Docker, can distribute the load evenly on the containers. Unfortunately, containerization does not scale the number of taxis.
- Containerization speeds up publishing and distribution of services, reducing time to market (time-to-market)
Whereas a virtual machine is a virtualization of a single server environment, a container virtualizes one application or service.
Some of the new integration platforms support containerization, for example, frends eiPaaS from version 5.3. (released 02/2020).
Containers are essential in building of elastically scalable Digital Integration Hub.
What is a Digital Integration Hub
In 2021 Gartner depicted an approach or system that can enable digitalization with the old legacy systems using an integration platform. The need is obvious - the demand for digital services and channels rose even more via COVID-19, and even the most traditional and slow-moving companies were forced to think about how to approach their customers.
Digital Integration Hub (DIH) idea is that APIs are used as facades to hide legacy systems. APIs are done in microservice-manner to be small. independent business functions like "getCustomerData" or "orderProduct". They may be composite APIs, including data from ERP and CRM.
If we weld these APIs directly to the backend systems, we will end up with something called coupled system. If your APIs are facing public and are used in, e.g. customer mobile apps, you'll end up with problems as the load eventually overloads the backend system.
The idea in DIH is to remove this tight bond between API and existing legacy and move to uncoupled architecture. In frends, the agent may run in a container environment and scale there - the legacy data source interfaces don't scale - unless we do something. In frends DIH, we pump the crucial business data to an intermediate database (e.g. in-memory) and APIs connect to this database. Data pumps - frends agents with specific orchestrations inside - read data from multiple data sources and create data structures to the intermediate data storage for fast reading. If the API is updating data in the legacy systems, e.g. "orderProduct", the data pump orchestrations handle the complexity of multiple source updates. For example, orders are coming from API and sales representatives that still use the legacy systems' UI - this might lead to problems inside the old legacy that we are prepared to solve in the orchestration part of the data pump.
With Digital Integration Hub, it is possible to extend the lifetime of an ERP even decade.
Following illustration depicts how DIH approach is used directly as an backbone of a mobile app in a ski resort app.
Will the Azure and AWS platforms replace the integration platforms?
Various cloud platforms, such as AWS and Azure, include the integration functionalities mentioned in this article as separate entities. For example, Azure integration services are divided into six separate components. The same features are available on new iPaaS platforms from a single coherent interface. This single view and monitoring of dataflows is something that is usually missing from cloud platforms like Azure and AWS. Both - cloud-based integration functionality or cloud-based integration platform - can make integrations ranging from file transfers to API management and microservices. But what are the differences then?
- Cloud Platform development requires a widely skilled developers. The same functionalities are ready-made on the integration platform. This aspect is highlighted, especially if the integration platform is of the no-code or low-code type, as discussed in the next section.
- Cloud platform development is a broader concept and includes not only integrations but also tailor-made business applications.
- Integration platforms typically have much better monitoring capabilities.
- Integration platforms with visual design tools have an order of magnitude more powerful monitoring. This is also known as low-code approach.
- Cloud platforms are also a distribution channel for integration platform services if the integration platform supports containerization.
- Hybrid integration platforms include a distributed architecture where integration or API can be deployed in the cloud and in a client's own data center - yet everything is developed, operated, and monitored through a single user interface.
Integration platform products are built on top of these same cloud platforms, providing a more coherent user experience and ease of maintenance over a cloud-only platform. As an example, many frends customers maintain and implement their integration solutions without the need for a separate integration vendor or a large team of integration developers.
So the answer to the title question is that cloud platforms are not replacing integration platforms. It is more like integration platforms, and cloud platforms work in symbiosis, where the integration platform provides coherent development and control for integrations, and the cloud platforms provide components for integration and much more, such as machine learning and data storage.
No-code, low-code, or full-code?
As I mentioned earlier, the functionalities and capabilities of integration platforms vary considerably. The development methods also vary between platforms. Some integration platforms are based on pure programming; some rely on visual modelling of processes, and technology or system adapters. Integration platforms based on programming are called full-code platforms. Integration platforms that require a little understanding of protocols and standards called low-code integration platforms.
Integration platforms that have plug-and-play adapters for enterprise systems like SAP and HubSpot are called no-code integration platforms. No-code easily and unpredictably becomes a full-code when the system to be integrated is used in a way that the ready-made adapter was not ready for. Sadly - this is the case quite often. How many ERP systems have been implemented without customizing or using its data fields for their original purpose and in the same way in different companies? That was meant to work in 80% of the solutions only works in 20% of the use cases. Because of this, no-code platforms often work well in demos, but in practice, they are laborious to implement if and when the adapter does not work as needed.
In full-code platforms, the problem is skill management: you need a large number of platform-aware developers. Often, for example, upkeep is expensive when the expensive, especially skilled developer is required to solve each problem or use case. For example, Apache Camel is a full-code platform that requires Java skills and understanding of underlying libraries and JVM engine.
Out of these options, the best one today is to go with low-code because it is the right combination of expression power and ease of operation. Below is an example of the frends integration platform BPMN representation. The visual process language in the picture is used for all development and operations. In fact, the image shows a single process execution instance: the operator can access every piece of information that has passed through the instance - whether it's content or even an internal log of the communication protocol.
What is the modern integration platform like?
Gartner is a well-known and valued research and advisory company. Their chief integration analyst Massimo Pezzini often speaks of the term "Pervasive integration platform." By this, he means an integration platform that includes a wide range or even all of the features mentioned in this article. In the frends integration platform, we have thought of the feature set even more extensively and have combined process automation (RPA) into the same set of integration features.
To sum up, the functionalities of a modern integration and hyperautomations platform are:
- Centralized monitoring and control of everything - from process automation to API performance history
- API Management
- API development / API Hosting
- Process automation visually (eg, BPMN)
- Ability to orchestrate processes and logic inside APIs based on Machine Learning.
- Ability to execute recordings that simulate what real humans would do with user interface (RPA, in a case there are no interfaces available in a legacy system).
- Creating APIs with Visual Notation
- Support for the new and old message or structure standards via connectors (eg, REST / JSON, SOAP / XML, oAuth2, edifact, etc.)
- Support for major systems via ready adapters (eg, SAP). Common modern iPaaS platforms, like Frends, has over 200 ready-made connectors.
- Ability to implement and publish mini or microservices, which may be the aforementioned internal or external services
- Complex scheduling - not everything is event-based yet. For example, the payroll process should start on the 25th of each month or on the first banking day before that if the 25th is not a banking day.
- Scaling with container technology in the cloud
- Ability to be truly hybrid = one user interface, but the ability to distribute execution of processes and hosting of APIs to on-premises or cloud environments simultaneously
- Support Digital Integration Hub approach by default
- Support DevOps and DevSecOps development cycles.
More reading/sources
- Micro Services, Martin Fowler https://martinfowler.com/articles/microservices.htm
- The granularity of Services, Christian Verstaete Cooking Breakfast and Microservice granularity
- Digital Integration Hub, Gartner
If you are interested in this topic or would like to hear more about integration, please contact us using the form below.
