Maturity level 0: point-to-point integrations, no integration platform
Controlled integration or data flow spaghetti?
Point-to-point integration is an antipattern where integrations are made directly from one system to another. They are developed with the "best" technology of each era and deployed into a corner of a random server. Sometimes they are hosted in the integrated system itself. Henry, the Heroic coder from the IT department, used to do these with PHP, after which Cora, the Cowboy coder, continued with his Python scripts. Both scheduled their scripts with separate Cron instances somewhere in the jungle of different servers. Over the years, tens or hundreds of integration point-to-point scripts accumulate. Think of one such script that executes a single data stream as one spaghetti ribbon. The consequence of this kind of hero integration is a network resembling a heap of spaghetti, to which making changes has unforeseen consequences. Also, changing one system from this spaghetti is surprisingly tricky. The integration links to and from the replaced system break and not all of the connections are even known — as Henry and Cora were notoriously lazy documenters of their code. And most importantly - centralized control of data flows is impossible.
Information is often compared to oil or water, as it is one of the most valuable assets of the business. Information is a critical part of running the business, the commercial secret of a company, or a valuable item to sell. Centralized control and management of these information flows are one of the most critical tasks in a modern business-driven IT organization. In a modern IT environment, troubleshooting and finding the problem in a process consume most of the time. For example, a CRM vendor says customer information has left the CRM system, and the vendor that is supplying the marketing automation as-a-service claims it never arrived. Centralized monitoring and decentralized execution of integration processes solve these problems. According to Gartner, centralized integration saves as much as 50% on IT costs compared to the spaghetti integration. The problem with spaghetti integrations increases as the number of systems increases. In the era of cloud computing services, cooking integration spaghetti is easier than ever.
The IT experts, Cora and Henry, are also part of the problem. As they retire, nobody knows what those hero-coded scripts are doing, where they are running, and why they are running.
However, point-to-point integrations are not just the IT gimmick - in some cases, they are justifiable solutions. These include, e.g., Temporary integrations or data streams in which different modules of the same vendor system speak to one another, and the vendor is responsible for the whole entity. In other cases - the centralized integration platform described in the next section is the right choice.
Maturity level 1: Centralized integration control, harmonized implementation.
Why a centralized integration platform?
A modern integration platform is a hosting environment for data flows. At the same time, it is a controller of these data flows. As its name implies, it is the platform - not the final solution that just works in a plug-n-play manner. Integration platforms and their functional sets are diverse and therefore challenging to compare. However, they have one common feature - they act as a central point of control, monitoring, and management for integrations and data flows. Centralized integration also significantly reduces the amount of data flows, as integrations pass through the integration platform on a one-to-many basis. This central point is the best point for controlling and monitor data flows.
The role of the central integration platform is to transfer data between all of the enterprise systems while handle the necessary data conversion, often referred to as data mapping. It means converting various data structures, such as XML or JSON, into another basic format like edifact. The term protocol conversion refers to, for example, first fetching a file with SFTP and then sending the payload as a JSON in a REST call.
A centralized integration platform typically also can schedule transfers or trigger an integration process execution from, for example, an incoming file, a database change, or incoming REST / JSON call. This launching a reaction from something that is happening is triggering. A more sophisticated form of triggering is an integration platform with capabilities to implement event-driven architecture.
In centralized integration, a module capable of interfacing a particular system is called an adapter or connector; for example, most of the integration platforms have SAP connectors. Quite often, these adapters also implement a standard. For example, an integration platform may have an SFTP adapter just to comply with old legacy systems that can handle only files.
The centralized integration platform also provides a single implementation method for integrations. Unified platform and methodology lead to easier knowledge management and prevent a vital understanding of dataflows to create a person risks.
Maturity 2: Process automation on a centralized integration platform.
How do I make process automation on an integration platform?
Some existing iPaaS solutions also include process automation features and capabilities. The orchestration consists of steps like data transformations, complex logic, and calls to APIs or connectors. This way, systems are integrated as part of process automation. The integration platform is capable of automating processes many orders of magnitude more efficiently than, for example, the RPA tool itself, which also has process orchestration capabilities. The integration platform is faster in automation because it does not simulate user actions but works directly - often as compiled code - at the interface level.
On the other hand - it is not possible to use an integration platform for automation if there are no interfaces available in the automation target system. Integration platforms cannot do process automation that occurs inside one user-interface. Modern integration platforms include the ability to orchestrate processes and workflows in standard languages. For example, the frends integration platform supports BPMN (Business Process Model Notation), which is a notation supported by Microsoft Visio and other modeling tools.
The image above is an illustration of a process that includes a robot as a part of the orchestration. The screencap is directly from frends -integration platform that follows BPMN -standard as the orchestration and integration development language. The right way of process automation is not to just look Robotic part of it, but to include whole business process automation and orchestration to it. You can get the best of both worlds.
Maturity Level 3: Service bus-based integrations and internal interfaces or APIs
What is the role of the service bus?
Integration platforms containing an enterprise service bus as-a-feature are capable of different message delivery methods. The bus usually includes a queue system to which messages can persist. Message persistence means that the sender can trust that the message - whether an electronic order or application from a webpage - has been permanently stored in the queue system. The sender can trust to persistence even though it has not yet reached the system it is eventually going to. Message queue (MQ) is not always an integral part of the integration platform.
For example, the FRENDS integration platform includes the queue system itself. On the other hand, frends is capable of using other queue systems because it supports the AMQP standard for queue communication.
Message queues - either individual software component or as part of an integration platform - still have their use cases, even though the world is increasingly moving towards event-based and synchronous messaging. Entering the event-based world means that messages - such as sales messages from the cash register to the central system - are no longer queued. Instead of queuing, Message can be sent directly to the integration platform's REST / JSON interfaces, which at best synchronize them directly to the receiving systems. If there is a load spike - for example, one hundred times the number of sales messages during the Christmas sales - scaling both the integration platform and the receiving line-of-business system or microservices is an event-driven and synchronous solution. This kind of synchronous solution might require support for container technology, which is a topic of another chapter in this article. Conversely, if the receiving system is a monolith, the integration platform can buffer messages or queue them in the traditional model. Support for container technology is not an established feature in integration platforms, and the need for it should be considered when choosing an integration platform.
Also, the service bus implements a publish-subscribe messaging. Systems that want information and messages to themselves can register themselves as listeners to topics or content. The service bus can route messages to the listener based on the topic of the message, for example, "all purchase orders." As an example of content-based routing, a CRM may be notified to listen to all messages containing a customer change and may come from multiple sources. Or, purchase order messages can be listened to on multiple systems, where the bus routes them and converts them both to protocol and message structure recipients.
However, the enterprise service bus as a whole is not a necessary part of the integration platform but is often present in the system integrations of larger organizations.
Maturity Level 4: Internal and Public APIs and OpenAPI
The integration platform is the engine of the API-economy
ntegration platforms, such as frends eiPaaS, can act as hosting environment and development platforms for APIs both internally and externally. External in this context means that API is, for example, publicly available on the internet. During development, Integration platform connectors and adapters connect to the legacy systems such as ERP systems or cloud systems such as SalesForce CRM. These systems and the aggregated data from these systems can be used to publish business functions or data sharing services in a variety of technologies and ways. Today, the most common and recommended of these are the methods and technologies defined by the OpenAPI standard. The published APIs may be, for example, an input channel for messages from the previous paragraph. Equally, a public interface implemented on an integration platform can be synchronously connected to backend systems without a bus.
OpenAPI is a standard and, for example, Swagger is an example of its implementation: a set of tools for implementing this standard, which is also used by the FRENDS integration platform mentioned in this article. OpenAPI ensures that the API is always described and documented in the same way and is easy to test. OpenAPI is often referred to as the "Public API." Be careful: when talking about a public API, whether it is an API implementing the OpenAPI standard or an API published on the public Internet. The latter need not strictly implement the OpenAPI standard to be public, that is, publicly available.
Internal APIs
In the image below, an example from a real taxi company's client search services or custom editing services - APIs - can be implemented on an integration platform as REST / JSON interfaces. Before the REST / JSON time, these interfaces were implemented in SOAP / XML and talked about web service. Even if the interfaces in the image implement the OpenAPI standard, they cannot be called public interfaces. They are just implementing a standard that makes them easier to use. Still, they are not publicly available on the other side of the firewall on the open Internet. Internal interfaces can be called by systems implemented at different times and are deployed in each system deployment project, for example, a new ERP. There is already a clear benefit: the system below, such as its ERP, can be changed so that calling systems do not notice the change. On the other hand, search services are reusable: you don't have to do the same integration job twice. This kind of service layer is called Service Oriented Architecture (SOA).
External APIs
Let's continue to use this taxi company as an example for external APIs. The company is deciding to develop a new mobile app to serve its customers better to compete with Uber as an ultra quality taxi service. Information on free cars, drivers, and locations already exist in the legacy systems that have been there for ages. Old architecture of implementing this kind of mobile app would be using developers doing the front and back in the cloud. If you have a potent enterprise integration service-as-a-platform, you don't need a separate back-end at all. Modern eiPaaS handle the low latency requirements as well; they handle the need to implement user-sessions within the services.
In the first stage, the taxi company developed a group of services: newOrder, getOrder, changeOrder, getCustomer, and changeCustomer. These were all internal APIs, followed by OpenAPI -standard and used within internal projects. Then the mobile app required the same functionalities - and so they were published within another frends Agent on the internet. These services were accessible only for their internal mobile apps. A modern integration architecture has built-in message queues that ensure that, for example, a DDoS -attack wouldn't endanger the internal systems.
Now, as they made their OpenAPI -standard-based APIs publicly available, they can be found and accessed by any developer. For example, real estates that have a reception with an "order taxi" -button, are direct calls to the same API with static address information. Third-party event-manager that are implementing their customer engagement mobile app can find the same API as well: "order taxi to my event." This way, APIs formulate new channels for sales.
API Management
When interfaces are built on an integration platform or built directly on a platform service such as AWS, managing them becomes the next challenge. How can I get other developers to find my public API? How do I make sure it's securely available on the public network? For this, there is an area of expertise: API Management. You can purchase a platform for managing these APIs only, such as RedHat 3Scale. Then API's hosting environment, the implementation, and the management of the APIs are separated.
On the other hand, modern integration platforms provide very comprehensive API management capabilities. For example, the frends integration platform has API management features for APIs developed and hosted inside FRENDS, as well as APIs developed elsewhere.
API management divides into two main parts: the API Portal and the proxy gateway. These sections do not contain the implementation of the API itself, but are the API's management and publishing layer. On this portal, third-party developers can find a published API, say, a Google search and program the app to use it (in the illustration "Applications").
Modern integration platforms can serve as API execution and creation platforms, as well as API intermediation services and portals.
How many boxes has the current state of your IT checked in?
Reaching integration maturity is a gradual process, that is dependent on your current business needs and business model. Nevertheless, it is better to start exploring all possible oportinities well in advance.
Get to know how different businesses are starting up with their integration processes in our Customer cases stories.
Interested to learn more about business benefits that APIs bring can bring to you? Our latest blog "The ABC of Business Growth" covers it all.