Getting a handle on chaos and complexity

There's a lot of interest these days in the notion of "controlling complexity," particularly when it comes to networks.

It's obvious why: Complexity can translate directly to increased operational costs (complicated systems are more time-consuming to manage) and decreased reliability (systems with many moving parts are more likely to fail unpredictably).

This perspective is summed up perfectly by some of the IT professionals I work with, who say things like, "I have no idea what's actually in my network anymore."

So everyone agrees that reducing complexity is, in principle at least, a good thing. The problem lies in determining exactly what complexity is. There's an entire scientific discipline of "complexity theory," but there's no firm definition of complexity, or of complex systems. Is complexity simply a matter of having lots of moving parts? Is being "complicated" the same as being "complex?"

Although there's no universally-accepted answer to these questions, a good definition of a complex system is the following: Complex systems are built out of myriad simple components that interact, and exhibit behavior that is not a simple consequence of pairwise interactions, but rather, emerges from the combination of interactions at some scale.

For networked systems, this means you can think about complexity in terms of the number of devices or agents in the system, their possible states, and the potential interconnections between them. If there are N agents in a system, it takes N*(N-1)/2 interconnections to interlink these agents directly to each other—meaning that the number of interconnections scales geometrically with N.

Another thing to keep in mind is that complexity can generate chaotic behavior. (A personal note: Chaos theory was seriously trendy in the 1980s and 90s, when I was a physics grad student, and I was hell-bent on studying it until an older, wiser physicist pointed out there were very few grants in the subject. Being fond of eating and paying rent, I ended up focusing on particle physics instead.)

Mathematically speaking, chaotic behavior is neither predictable nor random — infinitesimally small changes in a starting state can produce arbitrarily large changes in a later state. Obviously this is undesirable in a system (or network) that's designed to consistently deliver a specific function predictably and manageably.

For example, in a networked environment, a minor difference in configuration could trigger a downstream failure that's unpredictable and thus, unpreventable. These types of problems arise in virtually any complex environment (including nuclear power plants and airplanes in flight).

Interestingly, chaotic behavior often arises from very simple relationships. In other words, a complex system that is constructed of simple, deterministic building blocks can nonetheless display chaotic behavior. (Surprisingly, this mathematical conception of chaos was accurately captured in 1945 by the poet Edna St. Vincent Millay, who described it as, "Something simple yet not understood.")

The challenge of reducing complexity therefore becomes, in essence, the challenge of containing chaos. Some approaches to doing so can be drawn from other fields in which chaotic behavior arises; others are specific to networks. Here are a few:

* Compartmentalize. One technique for reducing complexity in inherently complicated systems is to design using "black boxes"—minimizing the potential for an agent to have a system-wide effect.

* Optimize. Try to reduce the amount of unnecessary state in the system. Do you really need 15 levels of QoS, or can you reduce to 7, or 5?

* Test at every level. Don't blindly deploy equipment, or simply test for functionality. Test it in a controlled setting in various configurations—and check to see that it responds as anticipated.

There are plenty more. But thinking in terms of minimizing complexity is a great first step.

Johnson is president and senior founding partner at Nemertes Research, an independent technology research firm. She can be reached at johna@nemertes.com.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Take IDG’s 2020 IT Salary Survey: You’ll provide important data and have a chance to win $500.