“Can it be so hard?!?”, asked a Customer with frustrated tone. This question was about a problem in a simple integration interface that received product catalogs from separate suppliers, transformed them and loaded them into ERP -system.  Suddenly the receiving interface rejected incoming catalogs from certain suppliers due to a mismatch in expected data structure compared to the structures supplier was sending. The interface was already in production use and passed all executed test including unit, system, integration and customer acceptance tests. We analyzed the problem and explained that if the Supplier that sending catalogs does not follow the flat-file structure that was agreed in the beginning of the project, we cannot transform it to the format that ERP accepts. “But it worked so well with our old ERP when integrations were implemented directly between ERP and Supplies”, argued the Customer. “Why can’t we just integrate them directly and forget the hard data-mapping in your integration platform? It is SO easy just to put them directly to ERP’s database.” 

Is it really? In this case the reality was that most of the Suppliers followed the format that was agreed and defined - but not all of them. This lead to a situation where integration platform were receiving Catalogs with unknown structure. In the history of that Customer support for different supplier Catalog formats were just written again and again in separate code blocks with continuous if-then-else –statements. I bet they had specs about format when they did those implementations. They were also implemented during a long and evolutionary process and no one can say how much time was spent on them, who is responsible for them and how they were monitored.

Even though we at Frends have been evangelizing ESB, SOA and EDA, it seems that we should not forget the basics. As shown before, every now and then some of our customers want to build point-to-point integration. This infamous methodology leads to a situation commonly known as integration spaghetti. Wiki article about Enterprise Application Integration also identify this problem. It is a setup where systems are integrated with each other with a star-like topology. In this topology, systems have typically several separately hosted, managed and monitored point-to-point data flows to other systems in organization and other set of data flows with external business partners like customers and operators. After a while the amount of connections between systems is impossible to manage: one is hosted inside ERP and monitored by it, other is executed by scheduled database job, third is run by some OS-based task scheduler etc.  Also upkeep of these integrations typically requires constant help from those persons that made these one-off solutions during the project.

What was hard again? For certain suppliers to follow agreed data formats or lack of knowledge in customer side what were already agreed with certain suppliers?  In this case, the history of creating spaghetti integration had lead to black boxes of integration that no one knew what the boxes really did. Black boxes hidden in a plate full of spaghetti, lead to incomplete requirements for new integration solutions replacing the old ones.