Why do integration projects fail?

Why do integration projects tend to fail more often than other projects? Or do they? At least they seem to have a reputation of the workload estimates often failing and the projects becoming costly and time-consuming. Don’t we, for heaven’s sake, have any good integration solutions on the market? Can't we just do point-to-points between systems? That is at least quick and cheap.

The problems are usually related to management issues regarding the integration, not the technology. Integration projects have many features that must be considered before starting the project.

Change is unavoidable – embrace it!

The traditional waterfall model doesn’t go well with integrations because the requirements change - constant change is more of a rule than an exception in integrations. If the business systems that are source or target of the integration are being developed during the project, also the integration will change – and many times. So, forget the waterfall and freezing requirements in the early stage. Instead, you should prepare to make the integration definitions at a right time when the end points are already defined and concentrate on making integrations in as an agile way as possible.

The difficulty in integrations is to recognize the common and reusable components that should be designed and implemented well and those that have just a single purpose and will never be developed further. The reusable components should be built in such a way, that the changes are easy and quick to make. They should be as generic as possible.

Sometimes, while replacing systems, you will also need temporary integrations that will be used for just a short period of time while both the old and new systems work in parallel. The temporary integrations should have a short time from the design table to production, and they can be MacGyvered anyway as you wish – if they do their job. But make sure that the temporary integrations stay temporary.

In the changing environment, the good integration architecture is valuable: the time and money you spend to design a suitable architecture, you will save for later. For example, it might be a good idea to hide the changing systems and interfaces behind reusable APIs.

The key is to know that you don’t know

Integrations are complex, and that should be considered when planning the project. People who do not understand the business processes that are automated can easily ignore essential details regarding the integration. This ignorance will lead to incorrect planning and scoping of the project. Therefore, integration experts should be involved in the early stage of the project.

When starting an integration project, stakeholders might have a different understanding of the outcome. The users of the integrated systems naturally think from their point of view and how their own systems work - and not the whole process. A mutual understanding between all stakeholders on the outcome should be clear before starting the project, and the processes should be documented. If you can’t document your process, you don’t know it.

Sometimes integration platforms and integrations are treated like the integrated systems even though integrations have much more dependencies than other software and they are not an off-the shelf-products. For example, you can’t test your integration without getting the end systems involved. One can’t design and implement an integration solution without knowing what kind of data is coming in and going out.

Technical knowledge and expertise alone are typically not enough to make the integration project a success; you need an integration expert that also has knowledge and understanding about different business processes and can guide you through the complex requirements involved. The co-operation of IT and business is essential in integration projects. The expert that can understand both technical details and business is key to success in these cases.

Who gets to decide?

Integration projects typically have multiple stakeholders that may have various, conflicting interests and requirements. Someone should be responsible for making the final decisions. Who gets their integrations first: HR or finance? Where does the master data live? Before the project is kicked off, the owner of integration processes and accountable for the final design and structure should be chosen. That person should be able to make final decisions between conflicting interests.

Sometimes the owners of different systems want to do it all. Third parties are not welcome to the projects because they make the communication more complex. Sometimes also, the organization assumes that their EAS (Enterprise Application Software) vendor can also handle integrations, and a separate integration vendor is not needed. Even if your ERP od CRM could provide an elaborative set of APIs, that approach results in point-to-point integrations. And they are troublesome to operate and maintain.

team

Get the business involved!

One can unite test integrations, but you always need an end-to-end testing, too. The integration testing process should have knowledge of the source and target system and their behaviors under defined scenarios. You cannot let it be the burden of the integration provider alone: you need to have support from the business if you want your project to succeed. If the business is not supporting the integration project and willing to spend their time in the project, the project will fail. The integrations are not developed for the IT but the business: we are automating the business processes, and the process owner should be involved.

To summarize

Five common reasons why integration projects fail are:

The project model doesn’t support a constant and rapid change, and that causes delays which costs money;
Project management lacks the understanding of integrations and creates unrealistic expectations;
Conflicting interests are not solved, and the decisions are left hanging which causes delays and more changes;
Point-to-point integrations are developed because an integration platform and integration experts are not used;
The business, whose processes are automated, is not involved, and IT must cope on its own.

What can we do about it?

Integration strategy

Before you start, take a deep breath, and spend some time thinking about your integration strategy. What is happening in your organization and IT infrastructure in the future? What are your goals in digitalization? If these are hard questions to be answered alone, strategic consultancy can be the answer. Sometimes you need someone to ask the right questions and that someone can be your integration partner.

Integration architecture

After you have a clear strategy, you can take the first steps towards integrations. To avoid a point-to-point spaghetti and messy integration landscape, you’ll need an integration architecture. A good way to start is to evaluate the current state of the integration architecture and then to make the strategic technology selections.

Forget the gateways!

Think outside the box and rethink the whole project: sometimes, other approaches are more suitable for integration development than making it a traditional (waterfall) project with gateways. When the requirements come one by one, it can be hard to give even rough workload estimates that hold. To avoid the laborious change management and applying for extra budgets, you might consider purchasing integration development as bandwidth and not as a project. For example, if you allocate 3 FTE integration development resources for the time span of your ERP project, the consultants can design and implement the integrations when they get the requirements. You can then concentrate on getting the requirements in time and forget the troublesome monitoring of the workload, budget, and scope.

Change management can also be reduced by establishing quality definitions like: Definition of Ready and Definition of Done that are familiar from the agile project models. Especially having a DoR reduces the number of iterations, but it comes with a price – if the customer organization is not used to agile projects, these gates become bottlenecks for efficiency.

If your organization is bigger, and you have more than one business unit, it might have overlapping integration needs. Then centralized integration management might be the right choice for you. The centralized integration team provides services to the whole organization and manages the “integration factory,” i.e., the development team that is allocated by FTEs.

Listen to the business

Who is a process owner? Who are you developing the integrations for? Get them involved! Sometimes there are difficulties in getting the IT and business to speak the same language. Here an integration expert with knowledge of the business processes can help and build bridges between different stakeholders.

Sometimes the IT doesn’t have enough time to talk with the business and inquire about integration needs. The product backlog doesn’t fill up by itself. A business analyst that concentrates on getting the business needs, use cases, and specifications from business to developers helps to avoid the bottlenecks in the integration development.

Check list: Four Levels of Integration Maturity

Retail and E-commerce stock management with Integrations - How to make it easy and efficient

Why plain coding and not using enterprise integration tools is not the way to do (many) things right in IT

Manage costs and speed up your business with process automation