Tired of chasing ghost outages? This team turned messy network logs into AI-powered insights that spot problems before users even notice.
The problem
How many times in a year do we get complaints from our end users that the network is slow…and our monitoring tool has not yet alerted us to anything critical? Sometimes Zoom lags, and other times Jira takes a lifetime to load — and you don’t know whether it’s a circuit issue or a Wi-Fi issue or a DNS resolution problem, each of which is maintained by different teams and different tools. You end up desperately toggling between different screens and your CLI.
These slowness issues and complaints have sometimes come up in my organization and executive directors are never happy when these escalations come to them.
For us, once the ticket opens, the usual troubleshooting that every network engineer must know by heart starts — at least for the first three steps: Ping the destination, perform a traceroute and check physical links for any drops. Let’s agree that it’s a very time-consuming and drab process in 2025.
And for most use cases, ping times are normal, traceroute shows regular latency, and of course, your interface has zero drops, which is when we go into the CLI device by device and that takes another hour and is a real strain on the eyes.
We are living in an era where AI is supposed to do the repetitive tasks as well as heavy lifting and help us troubleshoot such issues at hand, or even better, alert us in advance.
But the questions arise, where do we even start? Thankfully, our network devices have also evolved with time, and they can support open telemetry in some cases or almost every time, logging based on all events occurring, but in our use case, most of our vendors do not support open telemetry, so we went ahead with logs.
The solution and challenges
For AI to work for us and alert us in advance, it should have good quality, reliable data over time, and this data can be retrieved from our classic logs when any event is triggered. Ping and SNMP would only provide data in polling time intervals of two or three minutes, and it seems like a blurred reality; they won’t tell us the current state or projected states on trends.
So the research began: What level of information logs should we be collecting? Information level. We were collecting logs from around 2,500 global devices, and so we need to scale for capacity servers, which is not a problem in a large organization.
We were now collecting every informational level log from our SD-WAN routers, which included SLA violations, CPU spikes on hardware, bandwidth threshold increases, logging configuration changes every second and even collecting netflow…because let’s just agree brownouts usually hide between “user” and “app,” not inside a single device.
SD-WAN routers have SLA monitors configured for DNS, HTTPS and SaaS application monitors, which worked as our synthetic emulators and created a log whenever SLA breached for a layer 7 service or when any website is “slow,” which would help us monitor layer 7 protocols from a router.
From our radius/TACACS servers, we were receiving logs on security violations on layer two ports and MAC flooding(occasionally). Not just that, we even collected granular data like signal strength, SSID, channel bandwidth, and number of clients on the access point on our wireless infrastructure, all thanks to a vendor API that made quick work of this. Similarly, for our switches, we were collecting data from layer two VLAN changes to OSPF convergence, from radius server health to interface statistics.
After all the heavy lifting, we were able to get all this data into a data lake, but it turned out to be more like a swamp, as the data had 10 different timestamps and it was not labeled correctly. And AI without labels is wishful thinking.
We couldn’t differentiate whether the log was coming from a router or a switch, and then we had to tweak our configurations again to log specifically on a unique UDP port for each device to filter/parse data from them. Once we finished that exercise, we were able to collect logs by each device type and collect them in unique buckets. Furthermore, we decided to configure all our devices to UTC timezone rather than multiple timezones for better standard governance.
We defined schemas, owners and retention tiers. Hot data (seven days) lived on fast storage for streaming analytics; warm (30-90 days) moved to columnar stores for trends; cold (beyond 90 days) went to object storage with life-cycle rules. A small catalog documented each table, how to join it and sample queries. It felt bureaucratic, but prevented fragile joins and mystery dashboards.
What we built and how it helped us
Once we were confident that our validations that we were collecting clean data and labeling it, it was time to create dashboards for correlation and anomaly detection features. One of our first wins came from an anomaly detection feature we implemented for CPU spikes on one of the routers.
We were aware that on one of the routers, the CPU would go high, making it unusable, and that would sometimes cause packet drops with other sites…but we never knew what was causing that CPU spike. Since we were doing anomaly detection, we were alerted in advance when the CPU would go high the next time and when we drilled down even further in the logs, we found out from netflow data that file transfers of huge capacity were making our router work harder and spiking the CPU, leading to packet drops. In most traditional monitoring solutions by vendors, this analysis is hard to get.
Another win came from solving what everyone thought were Wi-Fi problems. People kept saying, “The wireless is bad,” but the real issue wasn’t the Wi-Fi at all. Before AI, we only saw the basic “up/down” status from Wi-Fi controllers, and every ticket looked the same. Users simply said, “The wireless is bad.” Engineers had to dig through logs manually, often spending hours or even days trying to spot patterns.
With AI observability, the process became much smarter. AI mapped the entire client journey — from connecting and authenticating to getting an IP address to our radius servers and reaching an app — and quickly flagged that failures were happening at the authentication stage as logs from our radius servers indicated “expired certificate.” Instead of treating each ticket as unrelated, it recognized dozens of devices failing in the same way and correlated the spike with an expired certificate.
Rather than giving vague alerts like “Wi-Fi is failing,” the system highlighted, “Authentication failures increased 300% after August 20, 2025.” Possible cause? “Certificate.” This pointed directly to the root cause, allowing the team to fix the issue in minutes and put an end to chasing ghosts in access points or wrongly blaming the wireless network.
A good start…
We stopped chasing fancy vendors that would promise all these services one day without much effort, and we took ownership of collecting clean, labeled data with honest math and feeding it to our open source observability tools. We used our own experience to frame data with the right context in order to get the visibility we were striving for.
We still have a long way to go, but this is a good start from where we began.
Models are only as good as the intent we feed them. The silver lining is that once you build an observability fabric that speaks in user journeys and changes, the models have a language to learn.
Today, when someone asks, “Is the network okay?” I don’t just say yes or no. I can explain it clearly with details. For example, “Almost everyone can use the payroll app without problems, but one site is having some delays because of a connection issue. The system has already rerouted traffic, and things should be back to normal in about five minutes.”
This isn’t guesswork, it’s backed by real data. Because of this, leaders no longer just want to see a green “all good” light on a dashboard; instead, they want clear service-level objectives (SLOs). And for us engineers, it means we spend less time defending ourselves and more time making the network better.
If you’re beyond ping and SNMP and wondering what’s next, build for questions, not graphs; for paths, not just nodes; for streaming, not snapshots; and for action, not admiration. Start small, label everything, keep your data tidy, and give your models the context they deserve. The reward is an observability framework that’s genuinely AI-ready and a network that can run the marathon your business demands.
This article is published as part of the Foundry Expert Contributor Network.
Want to join?
