Sunday, December 16, 2012

Complexity, Entropy & Maintenance

This post is inline with three other posts I made here, here & here. They all touch on the exponential increase in complexity and the limits to the continuation of that complexity. This post is about an oft overlooked limit to complexity: Entropy and the maintenance required to counter it. The degree of complexity of a system increases the cost of maintenance. When you get to a point where the maintenance of a system consumes all available resources you can no longer increase the complexity of that system. There are many reasons why I believe we are at or close to that inflection point.

Complexity
As a seasoned software developer I am always interested in reducing complexity. My many bad experiences with having to make changes to kludgy code have made me very conscious of writing clean, simple, easy to understand code. Anything I can do to reduce the amount of code or simplify how its put together makes my job easier latter on when I'm required to maintain or update it.

Many times you start out with bad code because of inexperienced or uncaring developers. Other times you start out well but over the years you accumulate layer upon layer of cruft from different updates, often by different programmers with different ideas on how the app should work. I have seen some legacy systems like this that were mission critical and yet over the years through the process of neglect and entropy they had reached the point of collapse. The same principle applies to any system. As it grows and adds layer upon layer it requires more and more maintenance to keep running. Once the maintenance required reaches a level that resources can no longer sustain, it collapses.

A system's complexity can be composed of not just its size or the number of its parts but more importantly the interconnectivity of those parts. As an example the internet is quite large expanding over six continents. It also has many different parts ranging from a handfull of DNS root servers to billions of individual server nodes and telecom cables. What makes this system really complex though is the interconnectivity of each of those nodes.

Entropy 
For all those pedantic Thermodynamic sticklers I am using the word entropy in a more general sense of the word to mean the tendency for things to go from an ordered state to a disordered state or, more specifically in this article, from a usable state to an unusable one. From my experience, it appears the more complex a system the greater the force of decay. This would also seem to be in harmony with the thermodyamic definition of entropy (S = K * LogW).

Every system exihibits this characteristic. From the virtual world where your codebase becomes more bug prone with each new feature to the real world where servers and fiber optic cable break down and malfunction with time and use. This constant decay makes a system less reliable and in some cases unsafe. Of course any system can be repaired or rebuilt. A codebase that is regularly refactured can stay clean and functional. Likewise a city that is regularly maintained can be kept clean and safe. The real dilema occurs when the complexity of your system has outgrown your ability to maintain it.

Maintanence
When a software department has more decaying systems than it does maintanence programmers to keep it tamed, it has reached a point where collapse is inevitable. The company has two choices at that point. Either it acquires more maintenance programmers, perhaps pulling them off of new development, or it has to reduce complexity by elliminating services. The third option, which unfortunately seems to be the one most often chosen, is to just keep limping along with decreasing reliability and squeezing more out of your programmers.

Like a sprawling legacy codebase that has reached critical mass, the U.S has reached a point where its urban infrastructure is so complex that the cost of maintenance has become too high to sustain. In some areas that have been recently built this may not be as noticable, but in older areas its painfully obvious.

A really critical area where this effect can be seen is the interstate highway system. Here in the united states life for the last 60 years or so has revolved around the automobile. So natuarally the increase in popluation along with that popluation's desire to drive anywhere at anytime has created a need for more road space to contain them. The result today is millions of miles of aging asphalt, bridges and tunnels.

The highway system is not the only aging system in the U.S. The sewer systems, the electric grid, dams, schools and government buildings especially in older cities are all decaying. Of course some of these systems are being repaired and rebuilt all the time. Unfortunately there aren't enough resources to keep all of them in safe working order. For that reson we are seeing reports like these more and more frequently:

NY Parks Close Due To Severe Budget Cuts
ND Asphalt Is Replaced By Cheaper Gravel 
America's Aging Bridges
America's Aging Electric Grid


Since there are strong indications we have reached a peak in resources of almost every kind, its only a mater of time before the force of entropy takes our civilization to greater and greater levels of disorder.