Exhibitors want isolated booth networks, attendees want more bandwidth, and nobody has much money. Here's how VLAN and QoS support on Linux routers helped keep a conference network up and running
Running a network is hard. But assembling one in a day, letting a motley assortment of virus-infected Microsoft Windows PCs, rogue DHCP servers, and beaconing boxes use it — and still maintaining full network functionality — is more than challenging.
Running a network is hard. But assembling one in a day, letting a motley assortment of virus-infected Microsoft Windows PCs, rogue DHCP servers, and beaconing boxes use it — and still maintaining full network functionality — is more than challenging.
The Southern California Linux Expo is in its fifth year, and was held this year at the Westin LAX Hotel. Like other tech events, the Expo provides a network for exhibitors, speakers and guests. Stu Sheldon, the Tech Committee chair for SCALE, not only expects things to go wrong when setting up the Expo network, he plans for it. Stu has been Tech Committee chair for the last four Expos. I sat down with Sheldon after the dust had settled from SCALE 5x and asked him about the challenges of planning, installing, and operating a network under such difficult conditions.
Explain the network architecture you designed for this year’s SoCal Linux Expo. What are the design criteria?
With SCALE, the design criteria is simple, Provide stable and balanced Internet access for both exhibitors and guests. That sounds easy doesn’t it? Oh, one other thing — I needed to make it so I could pick the entire network up and rearrange it every year. This has been my task since the very first SCALE. I officially took over the Tech Committee chair position shortly after SCALE 1, and now host and maintain the three SCALE public servers year-round in my colocation facility in Thousand Oaks, Calif.
Did I mention budgets yet? The network is only used once a year, so we really couldn’t commit tons of money to it. I think the budget for SCALE 1 was $200 for cable. All the network equipment back then was loaned or donated.
This year’s network design incorporated several major security and reliability enhancements, as well as a complete re-thinking of what fair and equal network access means. I went all out to try to solve all the problems I had encountered over the past four years.
The design idea was all about isolation. I put every booth on its own virtual LAN (VLAN). I pushed other key services such as registration and wireless onto their own VLAN. Everyone had their own little island on the LAN. Now I could treat them as individual interfaces on the Debian Linux router I built for the show.
For the first time in SCALE history, one booth could not see any traffic from their neighbors, and wireless was an island of its own. This required some serious planning, but I think the benefits were clear.
The end results were impressive for a temporary network
* More than 2.5 miles of Cat 5E cable.
* 76 VLANs with 76 firewalled networks.
* 10 Wireless access points.
* Nine VoIP Phones throughout the venue.
* An Asterisk server, which received incoming calls from the SCALE 800 number and routed them to the proper phone at the venue.
* A central Debian-based firewall to manage and secure all the VLANs, as well as provide DNS, DHCP, NTP and so on to all the hosts on the network. It also managed all the QOS for the show.
* A management PC to monitor the whole thing in real time.
What was your incentive for this design? Why was it designed in this manner?
My main incentive was reliability. Setting up a network for an event this size is like standing in the middle of a circle of people and telling them all to throw knives at you. You have hundreds or maybe thousands of network devices that you have never seen before being put on the network, and you have no idea of their health or configuration. You also might have people that for whatever reason want to try to attack or monitor your network with the goal of doing bad things.
Actually, most real network problems are due to stupid mistakes, not deliberate acts. You haven’t lived until someone brings a server from their office and plugs it into their booth network port, with it still running DHCPD and giving out IP addresses to everyone on the network. Or someone plugging a virus-infected laptop into the show network and proceeding to DOS everyone on the LAN.
There is also the issue of available bandwidth. A good friend once said, “Stu, if you gave them an OC3, they’d still use it all.”. Truer words have never been spoken. And yes, we ran our Internet access at full tilt for the entire show. That’s where the QoS settings came in. By setting up several weighted classful queues, and balancing the classes through carefully designed filters, I was able to provide the fairest network access ever for SCALE. This combined with VLAN separation, delivered unprecedented network performance and stability. Other than a couple of defective booth cables Saturday morning, the network delivered 100% uptime to all its users until the network was brought down Sunday night.
Explain how you implemented the VLANs; what was involved and what were your criteria?
Using managed switches, and the Debian firewall, I assigned each switch port to the VLAN I wanted it to talk on, then configured the Debian firewall to function as a VLAN router. I split the VLANS in half, 40 on one firewall port, and 40 on another.
Configuring VLANS is fairly simple, once you decode the switch interface and figure out what the switch manufacturer thought would be fun to try as an OS. Configuring VLANS in Linux is fairly easy. I installed the ‘vlan’ package from the Debian repository, and away we went. Debian has done a very nice job of incorporating VLAN configuration into the interfaces file which is used by Debian to configure all interfaces on the system. The net result was 80 separate networks that I could control via iptables.
I now had 80 networks I needed to provide such services as DHCP, DNS, NTP and other key services to. After thinking about the best way to configure the Debian box, I decided to create a series of scripts to create the configuration files. I had one to create the interface, one to create all the BIND 9 files for DNS, one to create the dhcpd config file, and I hand wrote the firewall and QoS files.
You implemented QOS on the SCALE 5x LAN; How did you do that and what tools did you use?
I’ve always considered QOS more like black magic than science, but it does a good job if you need to make things even or keep some bandwidth in reserve for a particular host or service. I use the iproute2 tool set to set up Linux Kernel-based QoS. I used HTB and SFQ queuing with a IFB interface to do egress shaping on inbound traffic. Then I setup tc filters to place the traffic where I wanted to be.
The lack of an SFQ queue under the VoIP Queue is no mistake. SFQ tends to reorder packets, and that’s bad for VoIP traffic. The VoIP qQueue also has the highest priority, so is emptied first. Only 80% of the actual bandwidth is available to the data, therefore, we need to keep the queues on the upstream routers as empty as we can to insure real-tim traffic has a fighting chance. Remember, these were DSL lines we were dealing with here. Also, The cushion at the top is for those nasty little bursts that pop the top of the limit on large packets.
A good example of this stuff can be found in Wonder Shaper. These guys know way more about this stuff than I could ever learn.
What skills are required for this design? How did you acquire those skills? What do you do for a regular job?
In real life, I’m president of my company, ACT USA. I’m a network engineer and security specialist. ACT USA provides Linux and open source solutions to small and medium-size businesses that are looking for low-cost alternatives to traditional business software and networking solutions. This allows me to create some pretty cool test environments. Working with open source has allowed me to get under the hood of technologies that a normal network administrator would seldom see.
To do what I did for SCALE 5x, you would probably want the following experience under your belt:
1) Experience in network design for hostile environments.
2) A strong working knowledge of routing and switching technologies, and how they relate to routing and security devices.
3) A working knowledge of the iproute2 tool suite, and a practical knowledge of how packets traverse routers and firewalls.
4) A working knowledge of iptables, and protection strategies in hostile environments.
5) A working knowledge of DNS, DHCP, NTP and other core network services as they relate to LAN management.
6) Knowledge of operating system diagnostics as related to end node network attachment and access.
7) A working knowledge of Asterisk, and Session Initiation Protocol configuration.
Or you could wing it, because let’s face it, Open Source software empowers anyone to do incredible things.
Who helped you install the network?
Because of the show size, I separated the wiring and hardware configuration tasks into individual jobs. I took the configuration part, which was programing all the switches, routers, servers, phones and firewalls. The Tech Committee lead, Mike Maki, as well as Nick Sheldon and Mason Maki set up all of the cable assemblies prior to the Expo.
Floor setup was managed by myself and Mike Maki, and made a reality by the rest of the tech committee members: Nick Sheldon, Steve Glasse and Jeff Deifik. These folks worked the entire show, and made SCALE a technical success. There were other volunteers on Friday who just showed to lay wire. I wish I could name them all here.
What equipment is involved and how did the Linux Expo acquire it?
* The border router, a Cisco 1800 serie and was provided by Digilink, the ISP Sponsor for the show.
* The firewall was a Solar PC was provided for the show by ACT USA.
* The switch core consisted of two 4500 50 port 3Com switches which SCALE purchased for this and future shows.
* Three 2900 series Cisco Switches were used as trunking switches and were donated by Simi Valley Hospital.
* The Asterisk server, a Solar PC, was provided for the show by ACT USA .
* The management server, a Penguinizer Elite, was provided for the show by ACT USA.
* The wireless access points were 3Com 7760s which SCALE purchased for this and future shows.
* The SIP phones used for the event were provided for the show by ACT USA or brought by the SCALE chairs for their use.
* Various switching hardware, and everything else we needed, was provided for the show by ACT USA.
What kind of Internet connectivity did you offer at the show?
Digilink Network Services provided two DSL lines for the show. They provided the Cisco 1800, and configured them as a bonded link. This gave us about 2.4Mbps in and 512kbps out. Bob Atkins, President of Digilink, has committed to doing SCALE 6x as the ISP Sponsor, and assures me that we will have at least double the bandwidth next year. Hopefully, nothing will change that plan; SCALE could use the bandwidth.
What areas besides the Expo floor were served by the SCALE network?
This year, we provided wireless to all of the speaker rooms, the downstairs bar, the e-mail garden, and the show floor. And of course, we supplied Internet access to the e-mail garden itself.
What if anything, do you plan on changing in the future?
That’s hard to say. I probably won’t do an Asterisk server on the show floor next year. Although it worked, it didn’t accomplish my main goal, which was to improve the communications between staff members. Although it was kind of cool, it just didn’t serve the purpose as well as the two way radios it replaced.
I’d also like to have better physical monitoring of the e-mail garden next year. That’s physical as in watched by a security guard. We didn’t have any problems, but I didn’t feel comfortable with it being in a room off to the side.
One thing I’m considering is not having an actual tech booth next year. As we get bigger, booth space is becoming more and more valuable. If I can swing it, I’d like to give the booth back to the show organizers to fill with an exhibitor.
There will most likely be many changes due to changes in floor plans, speaker rooms, and even the possible requirement of a complete re-design of the entire network due to venue change. Just like in real life, my committee will be tasked with looking at all the options that are available. For us, it’s all about creating a network that works the best, and costs the least for the show.
