Network engineering teams struggle to establish an authoritative repository of network data.
As IT organizations mature their network automation strategies, it’s becoming clear that network intent data is an essential foundation. They need reliable documentation of network inventory, IP address space, topology and connectivity, policies, and more. This requirement often kicks off a network source of truth (NSoT) project, which involves network teams discovering, validating, and consolidating disparate data in a tool that can model network intent and provide programmatic access to data for network automation tools and other systems.
Enterprise Management Associates (EMA) recently published research on NSoT strategies based on in-depth interviews with 17 network engineering professionals. Our report explored how network teams establish a NSoT. It’s about more than selecting and implementing a tool. NSoT projects can be long and painful endeavors, our research has found.
With that in mind, we’ve identified several challenges that a network team might encounter as they build their NSoT as well as some advice on how to mitigate them.
Challenge: Management buy-in
Many network teams use open-source tools for their NSoT. Part of this is a cultural preference, given that homegrown and open-source technology dominate network automation toolsets. However, it is also reflective of a lack of executive support. IT leaders do not understand the value of NSoT solutions. The data is already available, although it’s scattered and of dubious quality. Why should we spend money on a product or even extra engineers to consolidate it?
“Part of the issue is that we’ve got leadership that are not infrastructure people,” said a network engineer with a global automobile manufacturer. “It’s kind of a heavy lift to get them to buy into it, because they see that applications are running fine over the network. ‘Why do I need to spend money on this is?’ And we tell them that the network is running fine, but there will be failures at some point and it’s worth preventing that.”
“It’s very difficult to have a business justification for spending $60,000 or $70,000 on a source of truth when we also have a fairly stacked engineering team,” a network automation engineer with a large university said.
Solution: Proof-of-concept mindset
Smaller companies and deep network engineering teams can often succeed with a homegrown or open-source NSoT solution, at least at in the early stages of a network automation strategy. As complexity, scale, and diversity of use cases grow, many network teams will need to partner with a NSoT vendor. Thus, network teams should approach these projects with a NSoT mindset. They should document the value of such a tool as much as possible and prepare a business case for how a vendor could amplify this success.
Challenge: Data pain
NSoT isn’t a magic bullet for solving the problems IT organizations have with poor network documentation and scattered operational data. Network engineering teams will need to discover, validate, reconcile, and import data from multiple repositories. This process can be challenging and time-consuming. Some of this data will difficult to find. Once the data is found, engineers need to validate the data is good. For instance, is the spreadsheet for a network’s IP address plan accurate?
“In brownfield networks, IP addresses are in one IPAM, VLANs are in another system. Some people have spreadsheet and some use MongoDB,” said a network automation engineer at a Fortune 500 bank. “The sources of data are everywhere, and everyone is doing their own thing.”
“There was so much information for the operational team,” said network engineer with a large energy company. “From circuits to IPs to devices, all that was in so many different Excel sheets and different spaces, and each team had its own version.”
Solution: Discovery tools, cooperation
There is no quick fix for solving data problems during NSoT projects, but network discovery tools can help. Network observability solutions can typically discovery a wide variety of relevant data on the network. IP address management tools also offer discovery. NSoT specialist vendors have also recently introduced discovery features. There will still be some manual data gathering and validation required, since not all NSoT data is online. Rack locations, patch panels, customer service contacts for WAN circuits may all require manual fact-finding, for instance.
Another key element of success is management buy-in. In this case, it’s not about budget. It’s about a getting support from mid- and -upper management. These authority figures can issue directives that require various engineering teams to cooperate with the NSoT project team. This is especially important if there are certain people in the organization who refuse to share or relinquish control of data.
Challenge: User uptake and compliance
Once a NSoT is fully established, it should be an accurate representation of network intent. In theory, that intent should match up with network state, as long as the NSoT is properly updated whenever a change is made to the production network.
If a NSoT solution is integrated with the automation toolset, then most changes will trigger updates to the NSoT tool. Unfortunately, network engineers have a long history of making manual changes to the network via command line interface (CLI) and various other tools. When this happens, the state of the network will drift from the NSoT.
“You need to get everybody to operate under the assumption that this is the centralized source of truth and that you shouldn’t go to all these other systems for information,” said a network engineer a Fortune 500 financial services company. “That’s easier said than done. It’s a culture shift. It’s a political thing in some places.”
Solution: Integration, evangelism, control
NSoT project teams should integrate the NSoT with all relevant systems to ensure that network changes are properly documented in the NSoT. For example, most NSoTs have IPAM functionality, but the network team often uses a specialized IPAM tool for primary IP space management. The NSoT federates with this IPAM tool to ensure it has an accurate and up-to-date record of IP addressing.
Network engineers appreciate a good source of data. The NSoT team should demonstrate to them the power of having so much data consolidated and curated in one location, with deep integrations with their preferred tools.
Role-based access controls and restrictions on CLI access on the network are a couple of ways to impose control over unauthorized changes that can cause NSoT drift. The NSoT team should seek out ways to protect the quality of operational data from engineers who refused to by into a modern approach.
Final thoughts
This column is absolutely not the final word on the challenges that people will encounter with NSoT projects. For example, technical debt associated with an open-source or homegrown NSoT can cause pain down the road. Problems specific to NSoT tools are also going to crop up. Engineers might find that a tool’s data schema isn’t flexible enough, or that the tool’s APIs have limitations.
This column addresses the challenges that most NSoT project teams can expect to encounter with some suggestions on how to meet them. To learn more about how to execute your NSoT successfully, you can check out EMA’s new NSoT report and follow our ongoing research into network automation and NetDevOps strategies.
