From Friday, Aug. 15 until the morning of Aug. 18, Linkedin was not available to some of its wired Internet users; though the outage didn’t affect Android and iOS mobile users. The analysis of the GEO tags of about a 100 English language tweets revealed the locations of users who experienced the outage. The map above approximates the locations affected.
The first tweeted report appeared in the twitter stream on Friday morning Aug. 15 was sent to @Linkedinhelp. Based on Linkedin’s recommendation to try a different browser, the company seemed unaware of the problem.
@fancymelissa What happens when you try? Are you able to try an alternate browser to see if it is browser specific? -AML— LinkedIn Help (@LinkedInHelp) August 15, 2014
Later on Friday morning, recognizing the problem @Linkedinhelp announced:
Some members may be experiencing issues with our site. Our team is working hard to resolve. Stay tuned.— LinkedIn Help (@LinkedInHelp) August 15, 2014
But after the problems persisted unabated for more than 48 hours, @Linkedinhelp tweeted apologetically:
@TimREven1 Sorry about that :( We've got all hands on deck, stay tuned! -AML— LinkedIn Help (@LinkedInHelp) August 17, 2014
A Linkedin spokesperson responded to Network World’s request for more information Aug. 18 and said:
“We are looking into the issue - it currently appears to be affecting people who have Verizon FiOS at this time.”
The most tweets reporting the outage were made by Verizon FiOS users. Comcast and AT&T Uverse customers experienced problems, but there weren’t as many tweets attributable to users of these networks. But the problem was not limited to just wired consumer Internet access. iWeb located in a collocation facility in Montreal connected by multiple tier one networks experienced the outage.
The outage was also not limited to North America either. In Europe, Linkedin users in Germany, Hungary and the United Kingdom reported problems. The problem was reported from Japan too. Tweets under the hashtag #linkedindown in Russian and Italian without geotags may indicate that users in Russia and Italy also experienced the outage.
The cause of the problem is difficult to pinpoint. The usual suspect was a Distributed Denial of Service (DDoS) attack which is frequently the cause of such outages. Arbor Networks Worldwide 9th Infrastructure Security Report pointed out that 70% of the data centers surveyed had experienced DDoS attacks. But DDoS was easily eliminated because the outage was not pervasive in a single network or all networks. IPv4 address depletion was speculated by a number of experts because there have been many recent reports that older routers are strained to route traffic as they reach TCAM limits. A Linkedin spokesperson denied IPv4 address depletion was the problem.
The root cause of the problem is still unknown and information from Linkedin is scarce. On Aug. 18 a a Linkedin spokesperson offered this explanation:
“they [network operations team] were able to explain that while conducting regular site maintenance, we ran in to an issue where particular client IP addresses were not routed correctly. The issue has been resolved.”
Some Linkedin users concerned about another data breach, like the one in 2012 in which 6 million lightly encrypted passwords were stolen, could apparently sigh with relief. @LinkedinHelp replied to a user with privacy concerns stating:
@LinkedInHelp what is the problem they are working on ? do I need to be worried about any of my personal information ?— Jerry Speyer (@Jerry_Speyer) August 17, 2014
@Jerry_Speyer No need to worry. Just having a little hangup with your ISP. Hope to have it resolved soon. Thanks for staying patient. -DH— LinkedIn Help (@LinkedInHelp) August 18, 2014
Guessing the root cause of the outage is difficult. Presented with this limited information a network operations manager of a large network services provider who did not want to be identified speculated:
“It sounds like they may have forgotten to include IPs in their export rules on some of the edge routers or peering routers for some specific BGP neighbors. That is only a guess without knowing their network topology.”
The number of users affected by the outage is still unknown.
Looking at the bright side, a search consultant with a positive attitude who realized his Saturday afternoon was freed from searching Linkedin for candidates by the outage tweeted: