How updating an outdated industrial control system can work with fog computing

It’s the classic Industrial IoT problem – a 40-plant network of old-school manufacturing and production lines, run digitally by 9,000 outdated programmable logic controllers running on legacy Windows industrial PCs, was having difficulty in minimizing downtime.

According to fog computing and automation startup Nebbiolo Technologies – which declined to name the client directly, saying only that it’s a “global” company – the failure of one of those Windows IPCs could result in up to six hours of downtime for said client. They wanted that time cut down to minutes.

It’s a tricky issue. If those 9,000 machines were all in a data center, you could simply virtualize the whole thing and call it a day, according to Nebbiolo’s vice president of product management, Hugo Vliegen. But it’s a heterogeneous environment, with the aging computers running critical control applications for the production lines – their connections to the equipment can’t simply be abstracted into the cloud or a data center.

Architecturally, however, the system is a bit simpler. Sure, there are a lot of computers, but they’re all managed remotely. The chief problem is visibility and failover, Vliegen said.

“If they fail, they’re looking at six hours downtime,” he said on Tuesday in a presentation at the Fog World Congress in San Francisco. “Thousands of dollars are being wasted because they’re trying to troubleshoot it, trying to find a replacement [industrial PC].”

The plan of attack was relatively straightforward: Virtualize and converge all the compute nodes on the shop floor, making the whole system more visible, configurable, and manageable from a centralized control pane.

Nebbiolo used its own nodes to replace the elderly IPCs, but that’s only part of the problem solved. The network layer and the software itself had to be brought up to date and made part of the new system.

“There’s a huge brownfield of the things you have to do. You have to figure out how to clone all these Windows-based PLC applications,” said Vliegen. “So how do you mechanize that? How can you go to 40 plants, 9,000 machines and make it cookie-cutter?”

Another problem, he said, was the connectivity between the 9,000 PLCs. Connected manually, via serial, the system worked just fine, but virtualizing the networking element introduced problems in terms of tuning and timing – faster processors and network connections throw off carefully calibrated systems that have been working one way for decades.

Nevertheless, Vliegen said the company worked hard to make the new system highly resilient and efficient. Synthesizing machine data that’s moving across the newly Ethernet-enabled network in a bunch of different languages is what Nebbiolo’s FogOS software is deisgned for. They’re in two of the client’s facilities so far, and they are gearing up for a wider-scale deployment. 

Lots of minor issues remain, he noted. The older IPCs were used to having a one-to-one connection between the applications they run and a remote terminal, so a more centralized system (with multiple thin clients connecting to multiple terminals, and so on) required a lot of internal re-architecting of the system.

But this is the groundwork, according to Vliegen, and success on the initial trial should make more widespread deployment that much easier.