When Apple's iPhone starting messing up parts of Duke University's wireless LAN, readers and bloggrer took note and gave voice.
The iPhone Wi-Fi Puzzle at Duke University this week triggered a cyber tidal wave of commentary, complaint, confusion and even some comic relief at NetworkWorld.com and other Web technology sites.
The problem is that a few iPhones on the Blue Devils' campus-wide wireless LAN (WLAN) sometimes trigger a flood of Address Resolution Protocol (ARP) requests, as many as 18,000 per second. The flood overwhelms groups of Cisco WLAN access points, anywhere from a dozen to 30 at a time, taking them offline for 10 to 15 minutes.
As of end of day Tuesday 17 July, Duke recorded 9 iPhone ARP events. The university did not reply to an e-mail requesting an update on the number of incidents since then
By midweek, Network World readers logged about 90 posts on the problem, and scores of readers at dozens of other tech sites did the same. One anonymous reader suggested a simple explanation: "Everybody loves a good mystery."
But plenty of posters thought they sorted it out quickly. It was the fault of Apple. Or Cisco. Or Duke IT. Or Applehaters. Someone even suggested that Michael Nifong, the North Carolina county district attorney who was found guilty of numerous ethical violations in his handling of the notorious Duke lacrosse team case, was behind the problem.
"My take? Apple knows the problem but is afraid to own up to it since they are riding the glory train of success with the runaway whoop-dee-doo of the jesus-phone...I mean the iPhone," wrote Ed web/gadget guru. on the Network World comments thread.
Pure jealously, according to one Apple fan. "Come on guys, it's the network, not the iPhones. Otherwise this result would be worldwide," wrote ajhill at AppleInsider.com [registration required]. "Just another fruitless attack on the iPhone. Oh, they can be so jealous, can't they?
"Duke's IT staff should send a delegation to the nearest Starbucks or Panera Bread outlet and ask the girl behind the espresso machine how to configure a wireless router," wrote zanely, at CNet's forum.
Comments such as these prompted a Duke faculty member, John Madden, to reply on the Network World thread. "The good news is, from the perspective of us Duke users, our network is behaving just fine, thank you very much," he wrote. "The bad news is, I'm mortified by the amount of hostility evident in certain posts here from Network Administrators towards both network users and device manufacturers."
One anonymous Network World poster introduced a welcome note: a YouTube link to a song "written about this very topic," the Osmond Brothers' "One Bad Apple", which topped the singles chart in February 1971.
As of Monday, the Duke troubleshooters were pretty sure that it is the iPhone, and not the university's Cisco infrastructure, that is causing the disruptions. The phones are calling for the media access control (MAC) address of an invalid router, and in effect, won't shut up.
The speculation is that the flood may be triggered when the iPhone user moves from one part of the campus WLAN to another, disconnecting along the way. When the iPhone tries to re-associate at the new location, it may be calling for an address that previously worked: the wireless router in the user's home. It's not a valid device on the Duke WLAN, so the iPhone inexplicably keeps jabbering for it.
But others sites, such as University of Wisconsin at Madison, also have iPhones showing up on a Cisco WLAN, but are experiencing no problems. "As far as we can tell, they are acting like any wireless client should and we haven't noticed any strange behavior," says Dave Schroeder, senior systems engineer with the school's Division of Information Technology.
"One of our network engineers has been in touch with Duke, and one theory may be the particular way Duke's wireless network is configured, which is one wireless SSID for the entire campus on different L3 network segments," he says. By contrast, University of Wisconsin uses a unique SSID for each subnet. Another possibility being floated is that the iPhone is trying to associate with as many available access points as possible for roaming purposes, creating confusion and a cascading effect on the WLAN. Finally, Schroeder notes, there could be an issue in how the iPhone handles roaming.
There's no shortage of other explanations from tech-savvy (and some perhaps not so tech savvy) readers on several sites and listservs. You can contribute to the networkworld.com discussion here, and cast your vote on who's to blame in our online poll. Last Thursday evening, Apple was the choice of 39% (a decrease from 54% earlier in the week), Cisco was in second place 36%, with Duke's IT group the choice of 16%, a decrease from 24%.
The most interesting posts were by those who read the initial story closely, did some research and applied their own IT expertise.
One anonymous reader posted a succinct but detailed review of the known evidence based in part on a partial traffic trace posted by Duke's Kevin Miller. This reader summarizes four points. First, a single iPhone is "ARPing" at 30,000 requests per second; "very unfriendly, to say the least." Also, that the destination MAC address for the ARPs is an improper unicast address instead of broadcast address; "it probably compounds the issue." The iPhone also claims an odd IP address, which Miller wrote isn't on Duke's network.
"While there's room for improvement in the Cisco gear (maybe better rate limiting or traffic prioritization in the configuration, or perhaps a fix in software), it's pretty clear the culprit here is the iPhone," he writes. "And security experience says it's an iPhone defect, not an exploited iPhone."
Another reader contends the problem is caused by a poor Apple implementation of a proposed IETF standard, a set of steps called Detecting Network Attachment for IPv4 (DVAv4). From the Request for Comments document, "DNAv4 optimizes the (common) case of reattachment to a network that one has been connected to previously by attempting to re-use a previous (but still valid) configuration, reducing the re-attachment time on LANs to a few milliseconds." This reader posted this excerpt from Section 2.1 of the document, seeming to suggest that the iPhone is "aggressively retransmitting" when it should adopt an alternative strategy (also specified in the RFC document) to minimize competition with parallel DHCP retransmissions: "Where the reachability test does not return an answer, this is typically because the host is not attached to the network whose configuration is being tested. In such circumstances, there is typically little value in aggressively retransmitting reachability tests that do not elicit a response. "Where DNAv4 and DHCP are tried in parallel, one strategy is to forsake reachability test retransmissions and to allow only the DHCP client to retransmit. In order to reduce competition between DNAv4 and DHCP retransmissions, a DNAv4 implementation that retransmits may utilize the retransmission strategy described in Section 4.1 of the DHCP specification [RFC2131], scheduling DNAv4 retransmission between DHCP retransmissions." Over at AppleInsider, another detailed analysis reached a similar conclusion. The writer is djdj, a AppleInsider forum member. "If the explanation in the article is correct, which it seems to be, it's the iPhone that is generating the traffic that is causing the problem," he writes. "I'm an IT guy, and the explanation is perfectly plausible and credible. From their description, it truly looks like Apple made a mistake. And it should be fixable." In a later post, he notes that the ARP flood seems to be caused by a "strange set of circumstances that trigger the problem. A given phone has to have been at three locations with similar configurations but different IP subnets successively (first at location A, then B, then C, though the specific locations of A, B and C do vary). At location C the iPhone tries to find a router that existed at location A, and sends out 30 ARP requests to find that router every millisecond, so each iPhone is sending out roughly 30,000 ARP requests per second." He also suggests says that "it appears that the default settings on the iPhone may not trigger the problem, that the user has to have changed the Wi-Fi configuration to some degree, though to what degree that is remains unclear." Finally, he defends Duke's network configuration: "The Duke network isn't flat as some have suggested. It appears to have been well organized and implemented, and this is (ironically) actually contributing to the iPhone's behavior." His conclusion: "It's a bug in the iPhone, and Apple will fix it." The mystery-loving reader, whose quote began this story, has the last word. "It will be interesting to hear the cause," he writes. "Until then I say it was - the butler, in the library, with the access point."