Why IT and Operations are on a collision course

What would you do if you found out some of the software running your company was behind schedule on dozens of patches and updates — including critical security patches that were several years out of date? You might think that could never happen. You’d be wrong.

CIOs in enterprises across many industries are discovering that surprisingly antiquated and purpose-built applications are supporting their companies’ operations, controlling everything from factory equipment to pumps, heating and cooling systems and electrical power.

This is the world of operational technology, a world that has existed ever since the first engineer put a speed limiter on a steam engine in the 1800s. Since that time, operational technology has run completely independently from IT in most organizations. But these days, several converging forces — increasingly complex digital devices, growing concerns over security, and a desire to benefit from big data and the Internet of Things — are forcing operations and IT to work together. It’s not always easy.

“Operational technology is the domain of engineering and operations staff that is separate from IT,” says Gartner analyst Kristian Steenstrup. “In most companies, they’re in separate buildings and never talk to each other. How do you manage things in your organization when you’ve got more technology and data traffic outside IT governance than within?”

This may sound like shadow IT, but it’s very different, he warns. Shadow IT occurs when a department with a tradition of using technology provided by IT chooses to go outside that framework for some reason, usually frustration over IT’s rules about which technology to use or impatience at how long IT takes to evaluate and adopt a new technology. It nearly always involves a relatively new, relatively small and often cloud-based deployment.

“With the wave of a hand, IT departments could look after shadow IT,” Steenstrup says. “IT departments as they’re currently structured could not run operational technology. They’re unfamiliar with the architecture, vendors, applications, availability requirements and real-time criticality involved. It’s completely different from what most IT departments do.”

Beyond the ‘air gap’

So why can’t IT and operations keep on operating separately, as they’ve done for years? Because changes at most industrial organizations have rendered that approach impractical. First and foremost are growing concerns from the C-suite about cybersecurity, sometimes coupled with worries about compliance with newly enacted government regulations.

“The thing that’s happened in the last three to five years that’s really pushing these guys together is security,” says Peter Zornio, chief strategic officer at Emerson Process Management, part of St. Louis-based Emerson Electric, a $24 billion global manufacturing company. Emerson Process Management provides automation equipment, solutions and services for businesses in the oil and gas, refining, and chemicals industries, among others. “Security concerns around running these facilities are very different from what IT is looking at,” Zornio says. “IT is usually concerned about someone coming in and stealing information. On process control, you’re worried about people coming in and changing something in the equipment running a process. Some of these are dangerous processes that, if disrupted, could result in accidents, injury, environmental damage or even death.”

As C-suite executives grow ever more aware of these dangers, they’re increasingly concerned that operations be secured, something that techies know how to do and engineers don’t. In the past, many industrial enterprises kept their operational technology relatively safe with an “air gap” — meaning operations equipment was not connected to the Internet or the rest of the corporate network.

But that approach means sacrificing the efficiencies and just-in-time inventory savings that become available when a manufacturing company gives its suppliers and other business partners a direct connection to its systems. It also impedes the data analysis that can benefit the bottom line by providing the information necessary to, for example, direct material to machines that are operating below their capacity, predict what customers will want or identify machines that may soon need maintenance or replacement. As top executives start seeking some of those benefits — often by asking IT for them — the air gap security strategy looks less and less feasible.

Most operations engineers aren’t trained to deal with security. That was the case at Hunter Douglas North America, part of $2.4 billion Hunter Douglas, a maker of window treatments and architectural products based in Rotterdam, Netherlands. This year, the IT department and product engineers collaborated on a new motorized window treatment that could be controlled by a smartphone app. The IT folks were surprised that the product engineers hadn’t given much thought to cybersecurity, says Hunter Douglas North America CIO Rob Meilen.

But, he notes, if your job has always been to design window treatments, and you’ve never before connected one to the Web, it makes sense that cyber­security might not be top of mind. “Our teams in IT weren’t quite sure how to deal with a group of product engineers who were wondering why you’d design that in from the outset,” he says. “They weren’t resistant, but they had never thought about it. Thinking about cybersecurity is second nature to us.”

Another factor pushing operations and IT to work together is the way operational technology is evolving. The software that runs operational devices is increasingly dependent on traditional IT operating systems, often Windows. That’s a trend that’s likely to continue because the economics of it make too much sense for it not to. “Companies like ours use off-the-shelf technology because it’s a lot more cost-effective than creating it ourselves,” Zornio says. “It makes integration with IT systems easier and takes advantage of the higher cost-to-performance ratio of a Dell PC. Dell’s making millions of them; if we were doing it, we’d be making a couple of thousand.”

This means that operations engineers, trained and accustomed to dealing with heavy equipment, now find themselves managing a growing portfolio of software, often Windows-based. They must also manage the frequent updates and patches that go with Windows — and that’s a problem.

“Our systems were becoming more and more sophisticated, and more and more digital. Our people were looking more and more like IT specialists and less like engineers,” says Chris Heimgartner, assistant general manager for distribution and engineering services at Snohomish County Public Utility (SnoPUD) District No. 1 in Everett, Wash. “We don’t have the skill set,” he says. “Engineers fix things when they’re broken.”

So, like a growing number of engineering chiefs, Heimgartner turned to SnoPUD CIO Benjamin Beberness and his IT team for assistance with operating system patching and maintenance. When Heimgartner sat down to discuss this change with one of the engineers who worked on supervisory control and data acquisition (SCADA) software, he asked whether SnoPUD’s engineering team was good at handling patches.

“That’s when he pulled out the patch record,” Heimgartner recalls. It turned out one of SnoPUD’s SCADA systems was behind on 28 rounds of patches, and some critical patches were three years out of date. “He said, ‘No, we’re not good at this.'”

‘You’re not the boss of me’

Though many IT and operations leaders recognize a growing need to work together, there are still some formidable obstacles. For openers, there can be deep distrust between IT staff engineering personnel (especially the latter), who are accustomed to operating independently.

“You have a little bit of ‘You’re not the boss of me’ going on,” says Rick Davidson, president of Cimphoni, a Delafield, Wis.-based consultancy that provides IT leadership services and IoT solutions across many industries.

Though he’s now in a tech leadership role, Davidson spent years on the manufacturing side, so he has a good view of both disciplines. “When we in the factory look at IT, what we typically see is an insular organization,” he says. “An engineer will say, ‘All you want to do is layer on a lot of bureaucracy and controls that aren’t needed.’ The reality is, some of it is needed. But IT is working against the desires of a lot of guys on the factory floor who just want to do their own thing.”

Even when engineering staffers welcome IT’s assistance, IT experts may be reluctant to deal with the technology that runs operations. And you can’t blame them. “Often, there are more problems than there are solutions,” Steenstrup says. “Some are cultural and organizational. Engineering and IT in some companies have no track record of working together. And if IT gets involved, they’re going to inherit seven different vendors they’ve never heard of, running outdated technology that can’t be patched and often isn’t architected in a way that makes it easy to manage.”

For example, “There are a lot of operational technology systems running on Windows 95 and Windows NT 4.0,” he says. “The vendors of these systems are also engineers, and they like stability so they haven’t been as diligent about sending out patches or even approving updates,” he explains.

Even when they have, sometimes their efforts to keep their software patched are rebuffed by their engineer customers. “We used to think we needed to issue major releases every year and a half, and our customers said, ‘Whoa, too fast! Three years is great,'” Zornio reports. “The IT industry has somehow managed to program CIOs with the idea that of course you’re always upgrading and we’re not going to support this thing that’s over six years old. And they think that’s just the way it is. But engineers don’t think that way, any more than you’d think you have to replace the engine in your car just because it’s 10 years old.”

Underlying these differing attitudes to patches is a fundamental difference in approaches, priorities and even work culture, born of the profound differences in the purpose and workings of operations and IT. You can begin to understand this difference by considering a simple action that serves as the first step in troubleshooting most IT problems: the reboot. Rebooting a PC, server or network is a routine matter to IT. But when a system runs a 24/7 manufacturing operation, a reboot that brings things to a halt isn’t usually an option. Neither are planned downtime, updates and upgrades that are handled on weekends, or many other practices that are a normal part of managing corporate IT.

“The problem-solving approach is actually quite different,” Steenstrup says. “If you have an engineering problem-solving requirement, you look at physical characteristics, temperature, weather and what might have been done in a similar situation elsewhere. You iterate it for your environment, get it running as well as it can, and then you lock it down or weld it in place.”

Hunter Douglas North America’s IT team encountered that mindset in product engineers as it was putting the finishing touches on the company’s app-controlled window treatment. “Someone on the engineering side made the comment, ‘We built the app. I think we’re done,'” Meilen recalls. “The IT folks scratched their heads and said, ‘You know . . . generally apps evolve over time.'”

That sort of evolution comes naturally to IT, especially in this era of lean and agile development. “IT says, ‘Here are the technology standards. Here’s the cheapest, soonest solution we can do, and then we’ll prepare for change. We’ll be upgrading later,'” Steenstrup says. “It’s two completely different psychologies, neither of which can be discarded.”

Get rid of the stereotypes

Understanding that both attitudes are legitimate and that the two must be combined is the first step toward effective IT-operations alignment. “Engineers are not really good at patching, and not as good as IT is at software deployment and making changes to technology,” Beberness says. “But IT’s not as good as they are at knowing what the impacts of a change would be.”

Engineers complain that IT doesn’t “get it” when it comes to operational technology, but Beberness says, “I think they do. Sometimes they need to be educated a little bit, but it doesn’t take long for them to understand the criticality and urgency around operational technology.” That being the case, he adds, “both organizations need to get rid of the stereotypes and get their teams working together.”

Sounds good — but how exactly do you make that work? Begin with a frame of reference that will help engineers understand the bigger picture, Davidson suggests. “As an engineer, having a mental model — a picture from the shop floor through execution, through a traditional MRP [material requirements planning] system — is a big help,” he says. “An engineer could look at that and say, ‘Now I understand what my role is in the larger scheme of things.'”

Beyond that, Steenstrup says, you need a specific plan for how you will integrate operational technology with IT systems. “You’ve got things like maintaining data standards, data architecture and the ability to transform sensor data and use it in an IT environment.” This isn’t a process that will be completed quickly, he adds.

Achieving ongoing IT-operations integration requires support from top management and from the leaders of both disciplines — and those leaders must work closely together. “One CIO told me a while back that he could see that every future project his company does will be a hybrid IT-operations project,” Steenstrup says. “And he could have a battle every time, or else do it once and sort out standards, vendors and methods — and then all those other projects would be quicker.”

Some forward-thinking companies are taking this a step further and creating cross-functional IT and operations teams. “At Codelco, a large mining company in Chile, they have innovation run by engineers and the data scientists who are there to help them mine better,” says Kim Smith, vice president of Digital Innovation at Capgemini. “But they realized they needed to infuse some IT capabilities into the relationship, such as cloud technology and remotely operated vehicles.” That company created an innovation center where engineers and IT folks could explore new technologies together. “It created a marriage of engineering and IT,” she says.

When that happens, Steenstrup says, “I’ve sometimes seen the walls start to come down completely. If you have joint investments and architectures, you won’t have these problems in the future. So people building things like oil platforms are looking to get started on creating alignment now.”

After all, engineers are engineers — whether they work with algorithms or conveyor belts. “Manufacturing engineers and software engineers,” Davidson says, “are both concerned about designing systems that work and don’t break.”

Building a smart grid, together

At SnoPUD, CIO Beberness and Heimgartner, assistant general manager for distribution and engineering services, have always seen eye to eye. And Heimgartner knew he needed IT’s expertise to complete the project he was hired to undertake.

“We started [around 2006] with a system that Thomas Edison would have been comfortable with, and we’ve turned SnoPUD into a 21st century utility over the past seven or eight years,” he says.

A vital element of that transformation was creating a smart grid, something that required both engineering and computing expertise. And that meant putting careful thought into how responsibilities and tasks would be shared. “In all our substations, we have gateway computers that run a Microsoft platform,” Heimgartner says. “At heart, they’re PCs. Do I want my engineering people to be responsible for patching those?”

The answer to that question is no. On the other hand, he asks, “Do I want IT people going into substations to do those patches?” Again, the answer is no, because a substation is a dangerous high-voltage environment that requires special training to navigate safely. The solution — a networked system that lets IT patch substation computers remotely — is just one example of a project that required IT and operations to work effectively together.

When it came to creating, deploying and maintaining the smart grid, Beberness and Heimgartner both recognized that they needed a clear breakdown of responsibilities between IT and operations. “When I got here four years ago, the IT organization did not support any operational technology,” Beberness says. “Chris and I talked about what made sense and where the line is.” In SnoPUD’s case, the line falls between networks, servers and operating system patches, which IT manages; and physical stations, communications technology and SCADA software, which operations manages.

To keep everything straight, and to make sure no one was left uninformed, the two leaders created a RACI diagram — for responsible, accountable, consulted and informed — to settle just which members of each of their teams needed to be consulted on and/or kept in the loop about which decisions. “Consulted and informed are where most of the issues arise,” Beberness notes.

“We ended up getting everyone in a room who was responsible for their piece of it, and we hammered it out through the RACI diagram, and it was so effective that the silo barriers we had around those issues kind of disappeared,” Heimgartner says. One reason this worked so well is that he and Beberness discuss everything and are very aligned in their own views of how IT and operations work together. “People get that they won’t catch Benjamin and me on different pages,” he says.

The new smart grid got its first big test in late August, when a huge windstorm swept through the region, knocking down branches across Snohomish County and cutting off power to 175,000 SnoPUD customers — more than half of the utility’s total customer base of 335,000. The outage took five days to completely resolve, Heimgartner says, whereas a lesser outage in 2006 took seven and a half days to fix, and a comparable outage in 1993 took 10 days.

“Some of it was the really great people who flew at the problem,” Heimgartner says. “Some of it was our processes — and some of it was the visibility we now have with the smart grid.”