In general enterprise software consists of large numbers of systems with complex cross-dependencies and with a high level of heterogeneity and redundancies. The ability of these software systems to address the needs of the enterprise changes over time for the worse. In order to understand the forces at work I will first consider entropy at the system level and then at the enterprise level.
Initially after a software system is delivered, making changes to the system is usually relatively easy. The longer the system is in maintenance and the more changes are incorporated into the code base; the more the ability to make changes to the software system deteriorates  . There are various reasons for the deterioration of software systems. Some of the reasons are due to technology, but mostly the reasons are organizational of nature .
From a technology perspective various best practices exist to counter entropy of software systems. For legacy technologies roadmaps exist to incorporate the latest systems design thinking to address short comings of earlier technologies. As an example IBM provides various strategies to improve the maintainability of RPG programs  .
At a system level the single most significant contributor to the decay of the code base is the lack of routine refactoring to ensure the long term maintainability of the system . These refactoring exercises can for instance be geared towards ensuring the soundness of the design in the face of changing business needs or taking advantage of technology improvements. The lack of timely refactoring may be due to factors such as lack of skilled staff or time-to-market demands.
As such technology is not entirely to blame for the entropy of software systems. Rather, organizational level challenges are the most important contributors to the decay of software systems. Therefore, in order to address software system entropy, change firstly needs to be effected at the organizational level and subsequently at the technology level if appropriate. For example, replacing the existing RPG programs with the latest technology will inevitably in time propagate entropy in the new technology as well if the organizational challenges are not addressed.
Further factors that contribute to the decay of software systems originate at an enterprise level.
Enterprise Software Entropy
The level of entropy found at the enterprise level is significantly higher than that of a single isolated software system. Typically the enterprise architecture is an accidental architecture. That is the enterprise architecture came about organically due to the independent evolution of the various systems and their cross dependencies rather than being planned. Since the various systems have been designed and developed in isolation, redundancies between systems are rife. Many enterprise architectures has grown over many years which results in a plethora of programming languages – often based on disparate programming paradigms – used across systems. With such a lengthy history these systems are frequently hosted on different kinds of servers running a variety of operating systems.
A characteristic of an accidental architecture is the occurrence of stovepipes (or silos). Stovepipes are systems that have been developed to fulfill a particular task without regard for the context of the task within the larger enterprise business process . Since these systems have not been designed to interoperate , this approach sabotages efficiency at an enterprise level . The most prevalent approach to alleviating the effect of silos in the enterprise is through different manifestations of point-to-point integrations. This results in the intertwined enterprise architecture as seen in Figure 1.
The effects of stovepipes are usually acutely noticeable between business units while the intra business unit level effects are often largely hidden from the enterprise. Understandably this causes business and management to have a vague idea of the true complexity of integration projects. This contributes to integration projects being invariable hugely under estimated in terms of cost and timelines . Given this background it is no wonder that an estimated 70% of Enterprise Application Integration (EAI) projects fail . The problems around integration are further exacerbated when e-Business and its associated need for business to business communications is considered.
If we want to address the problems of enterprise software effectively, we have to understand the forces contributing to entropy. These factors can be categorized into day-to-day operations and specific events .
The daily maintenance processes are geared towards the continuous delivery of economic value in the face of ever changing business needs. Enterprise systems generate economic value through value networks. These value networks frequently span multiple applications  since most applications or off-the-shelf solutions are not solving cross-functional or inter-enterprise needs   . In order to support the value network these applications need to be integrated  . Day-to-day operations are typically project and short term focussed instead of being enterprise and long term focussed. This explains the proliferation of point-to-point integrations found in many of today’s enterprises.
Besides day-to-day operations specific events contribute to the decay of enterprise systems . These events include but are not limited to the following:
- The maturation of organizations causes new specialized groups to be included in the Software Development Life Cycle (SDLC) .
- End-of-life events of a supported product  may cause skills to be available only at a premium.
- An enterprise may expand into new territories as a result of off shoring or globalization .
- Mergers or acquisitions can introduce new systems and technologies into the organization. This is likely to result in redundancies at various levels   .
- New lines of business may be introduced to take advantage of new business opportunities . Often business units have their own IT departments with opposing agendas causing infighting between business units .
- Obsoleteness or limitations of legacy technologies may drive technology modernization exercises . Legacy applications further provide challenges in managing a retiring workforce and ever decreasing skills base .
- Government legislation may change the way business is conducted .
It is significant to again note that the role technology play in driving entropy at the enterprise level is negligible. Thus in order to find solutions to enterprise software entropy we cannot consider technical change alone – change must be effected at the organizational level as well.
The traditional solution to the decay of software is to start from a clean slate and rewrite the application using a new technology. Often technology is blamed for the ills of the software industry, but technology is merely a tool at the disposal of the enterprise. The enterprise chose to use the technology because it has a business value. When a newer technology comes along it brings new opportunities for increased efficiencies – it does not nullify the value of the older technology. Therefore a rewrite should not be considered merely on technical grounds (for example the availability of a new technology), but should have a clear business case.
Continuous refactoring can be used to modify an application to take advantage of the latest design principles and stay relevant in the face of changing business needs. However, the prevalence of short term planning in organizations causes the continuous refactoring step often to be skipped altogether. Omitting the refactoring step in the long term leads to code that becomes too costly to refactor, hence the need for a rewrite.
|||D. Krafzig, K. Banke and D. Slama, “Enterprise SOA: Service-Oriented Architecture Best Practices,” Amazon.com, 2004. [Online]. Available: http://www.amazon.com/Enterprise-SOA-Service-Oriented-Architecture-Practices/dp/0131465759/.|
|||M. Juric, P. Sarang and B. Mathew, “Business Process Execution Language for Web Services BPEL and BPEL4WS 2nd Edition,” Amazon.com, 2006. [Online].|
|||L. Patterson, H. Araki, S. Bramley, G. Cobb, J. Eikenhorst, S. Milligan, J. Simons and M. Tregear, “Modernizing and Improving the Maintainability of RPG Applications Using X-Analysis Version 5.6,” http://www.redbooks.ibm.com/, 2006. [Online]. Available: http://www.redbooks.ibm.com/redpapers/pdfs/redp4046.pdf.|
|||D. Cruikshank, “Modernizing Database Access: The Madness Behind the Methods,” IBM, 2006. [Online]. Available: http://www-03.ibm.com/systems/resources/systems_i_software_db2_pdf_Performance_DDS_SQL.pdf.|
|||M. Matsumura, B. Brauel and J. Shah, “SOA Adoption for Dummies,” Software AG, 2009. [Online]. Available: http://zy.xjgame.com/SOBPAO/SOA%20Adoption%20for%20Dummies.pdf.|
|||J. W. Ross, P. Weill and D. C. Robertson, “Enterprise Architecture As Strategy: Creating a Foundation for Business Execution,” Amazon.com, 2006. [Online]. Available: http://www.amazon.com/Enterprise-Architecture-Strategy-Foundation-Execution/dp/1591398398.|
|||R. Schmelzer, “Understanding the Real Costs of Integration,” zapThink, 2002. [Online]. Available: http://www.zapthink.com/2002/10/23/understanding-the-real-costs-of-integration/.|
|||G. Trotta, “Business Process Management (BPM) Best Practices: Dancing Around EAI ‘Bear Traps’,” ebizQ: The Insider’s Guide to Next-Generation BPM, 2003. [Online]. Available: http://www.ebizq.net/topics/int_sbp/features/3463.html.|
|||F. A. Cummins, “Building the Agile Enterprise: With SOA, BPM and MBM,” Amazon.com, 2008. [Online]. Available: http://www.amazon.com/Building-Agile-Enterprise-SOA-Press/dp/0123744458.|
|||F. Kuglin and R. Hood, “Using Technology to Transform the Value Chain,” Amazon.com, 2008. [Online]. Available: http://www.amazon.com/Using-Technology-Transform-Value-Chain/dp/1420047590.|
|||M. Weske, “Business Process Management: Concepts, Languages, Architectures,” Amazon.com, 2012. [Online]. Available: http://www.amazon.com/Business-Process-Management-Languages-Architectures/dp/3642286151.|
|||C. C. Poirier, L. Ferrara, F. Hayden and D. Neal , “The Networked Supply Chain: Applying Breakthrough BPM Technology to Meet Relentless Customer Demands,” Amazon.com, 2003. [Online]. Available: http://www.amazon.com/The-Networked-Supply-Chain-Breakthrough/dp/1932159088.|
Hi there, just came across your blog via your ADF piece. Great writing.
Why do you think that software entropy is mostly an organizational problem? It seems to me it’s a complexity problem (of course, I’m a project manager, not a software engineer). At some point, there are too many connections within the application for any human mind to understand, and your options are a massive refactoring, or chuck it and start again.
How do you see organizational processes playing a role in this?
It is a very good and relevant question you are asking! Fred Brooks has way back then already lamented the complexity of software. In his opinion the complexity of software is essential rather than accidental. Perhaps it is here that I disagree with him in that I believe that the complexity of software is partly essential and partly accidental.
If a new system is deployed and there is never a need to change it, chances are that there will be no need to replace it. It is making changes to a system that is causing its complexity to increase, as you have rightly pointed out. This complexity is due to the system more-and-more deviating from its original intend. What this means is that the original design of the system does not fit its latest purpose any longer. In order to get the system to serve a different purpose it is “hacked” to appear to serve the latest purpose.
It is at this point where organizational culture comes into play: An organization can decide to pile even more changes onto the system, thereby exacerbating the ever increasing deviation from the original intend. Or alternatively, they can make a decision to do a redesign and refactor the system to realign it to fit the latest business purpose.
If the organization piles on more changes, the resulting complexity I see as accidental rather than essential. That is the organization had the opportunity to reduce complexity but for some reason has decided to not do so.
Thank you for your question!