Getting a handle on chaos and complexity

There's a lot of interest these days in the notion of "controlling complexity," particularly when it comes to networks.

It's obvious why: Complexity can translate directly to increased operational costs (complicated systems are more time-consuming to manage) and decreased reliability (systems with many moving parts are more likely to fail unpredictably).

This perspective is summed up perfectly by some of the IT professionals I work with, who say things like, "I have no idea what's actually in my network anymore."

To continue reading, register here to become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

There's a lot of interest these days in the notion of "controlling complexity," particularly when it comes to networks.

It's obvious why: Complexity can translate directly to increased operational costs (complicated systems are more time-consuming to manage) and decreased reliability (systems with many moving parts are more likely to fail unpredictably).

This perspective is summed up perfectly by some of the IT professionals I work with, who say things like, "I have no idea what's actually in my network anymore."

So everyone agrees that reducing complexity is, in principle at least, a good thing. The problem lies in determining exactly what complexity is. There's an entire scientific discipline of "complexity theory," but there's no firm definition of complexity, or of complex systems. Is complexity simply a matter of having lots of moving parts? Is being "complicated" the same as being "complex?"

Although there's no universally-accepted answer to these questions, a good definition of a complex system is the following: Complex systems are built out of myriad simple components that interact, and exhibit behavior that is not a simple consequence of pairwise interactions, but rather, emerges from the combination of interactions at some scale.

For networked systems, this means you can think about complexity in terms of the number of devices or agents in the system, their possible states, and the potential interconnections between them. If there are N agents in a system, it takes N*(N-1)/2 interconnections to interlink these agents directly to each other—meaning that the number of interconnections scales geometrically with N.

Another thing to keep in mind is that complexity can generate chaotic behavior. (A personal note: Chaos theory was seriously trendy in the 1980s and 90s, when I was a physics grad student, and I was hell-bent on studying it until an older, wiser physicist pointed out there were very few grants in the subject. Being fond of eating and paying rent, I ended up focusing on particle physics instead.)

Mathematically speaking, chaotic behavior is neither predictable nor random — infinitesimally small changes in a starting state can produce arbitrarily large changes in a later state. Obviously this is undesirable in a system (or network) that's designed to consistently deliver a specific function predictably and manageably.

For example, in a networked environment, a minor difference in configuration could trigger a downstream failure that's unpredictable and thus, unpreventable. These types of problems arise in virtually any complex environment (including nuclear power plants and airplanes in flight).

Interestingly, chaotic behavior often arises from very simple relationships. In other words, a complex system that is constructed of simple, deterministic building blocks can nonetheless display chaotic behavior. (Surprisingly, this mathematical conception of chaos was accurately captured in 1945 by the poet Edna St. Vincent Millay, who described it as, "Something simple yet not understood.")