Now testing never captures a time lapsed fault until it happens! The most appropriate strategy is to have redundant systems and updated them sequentially. This may sound logical, but once I encountered techies updating (patching) a redundant system at the same time. When I inquired as to why this obviously flawed strategy was being used, I was informed that this is what had been decided by the change forum. The reasoning was that since there was going to be downtime, they wanted to minimize the downtime. The strategy might have been correct for a time analysis of the task but the risk of time lapsed bugs had been discounted. The resultant downtime as a result of a major incident makes it crucial that updates are planned in a sequential manner.
If the update results in the redundant systems not being able to maintain its state, because they are on different levels, due to version incompatibility, then besides kicking your vendor’s backside to kingdom come for bad practice then it is better to have the system function with limited (or no) redundancy for a short time period. This strategy has less risk associated with it, than a simultaneous time lapsed issue causing a major incident on your redundant systems.
Imagine the surprise in the data centre operations area when a failure like this occurs. “Don’t worry mate. No biggie. We have two of those. What the heck! The second one is also rebooting! RED ALERT! SEV 1! WAKE THE NEIGHBOURHOOD UP!”
When we had realised the cause of the lock-ups, we immediately had customers do a controlled reboot as a workaround, after which we had 255 days in which to load the new code without the bug.
Lesson learned: Patch sequentially. Reality: Super Tuesday habits. Patch everything at once. Warning: One day there is going to be a major incident tsunami, due to a time lapsed bug.
