Even the government shudders when someone says they're from the government and they're here to help.
Case in point: A hacker's diversion of traffic from a California county government Web site to a porn purveyor spiraled into IT chaos yesterday after a countermeasure applied from Washington essentially "deleted the ca.gov domain."
Order was restored only after seven hours of frenzied coast-to-coast communications and a "forced propagation" of ca.gov network systems, according to Jim Hanacek, public information officer for the California Department of Technology Services.
"We don't for sure have the whole picture, but as we understand it, there was some event at the Transportation Authority of Marin County where their site got hacked," Hanacek told me this afternoon. Traffic was being redirected from that site to one featuring pornography.
A department within the U.S. General Services Administration in Washington oversees and polices the .gov domain.
"The federal government saw this incorrect use of ca.gov and they made a change at a much more global level than probably was necessary and it started taking down all of our ca.gov domain," says Hanacek. "That impacted Web access and e-mail services."
A Network World reader whose brother works for a California state agency forwarded me an e-mail alert that his brother received: "The Department of Technology Services (DTS) has notified us that the Federal Government inadvertently deleted the CA.GOV domain. As the evening progresses you may experience an impact in your ability to access some Web sites and exchange e-mail. DTS is working with their federal counterparts to restore service as quickly as possible but service may not be restored until tomorrow morning."
The change from Washington was made around noontime yesterday on the West Coast ... and things quickly got worse.
"Unfortunately there was no prior notification, they just made the change and sent us an e-mail to one of our administrators who wouldn't be a normal contact," Hanacek says. "Once that person saw the e-mail and started looking we determined how serious this could be and we opened our emergency operations center. Unfortunately that was about 3 in the afternoon and folks back East were already going home, so it took us some time to get hold of the right people in the General Services Administration to get this address reinstated."
Those corrections began between 4 and 5 p.m. PDT but didn't restore full normalcy until about 7:30 p.m.
Hanacek indicated that California's IT people will be having a chat with their Washington counterparts: "We'll certainly be discussing how we should be notified of a change of this magnitude."
Welcome regulars and passersby. Here are a few more recent Buzzblog items. And, if you'd like to receive Buzzblog via e-mail newsletter, here's where to sign up.
This year's "25 Geekiest 25th Anniversaries."
A burning question: What's up with all the fires?
Verizon FiOS installation sparks ... nothing?
MIT tries to put the prank back in "hacking."
A defense against Photoshop funny business.
How much does the computer store owe this PC buyer?
The next 5 items that Google might buy from NASA.
'Hello, you have reached my iPhone.'
When not blogging, I am a Network World news editor and write the 'Net Buzz column.
The opinions expressed in this Weblog are those of the writer and may not represent the opinions of Network World.
|
|
I love how states always
I love how states always seek to blame the fed when things go wrong :)
Seems they hold a large piece of the problem here.
They failed to secure their systems and were unaware of the porn redirect. GSA was notified of the abuse and like all registrar's notified the technical and administrative contact.
Ca.gov says they contacted the wrong employee but wait a min. The reason you have technical and administrative points of contact is so that the registrar can contact the right people, right? Is that not the purpose?
So CA.gov failed to update thier own records, GSA got no reply and took the prescribed action. Delete. Its a government domain that forbids porn.
Hash perhaps but i bet ca.gov will take action to keep their records up to date and increase their security program.
Exactly! That's why porn on
Exactly! That's why porn on the Internet should cause the whole .com to be deleted!!!!!!
Urm.
Urm. Porn on a .gov domain will get you deleted if and when you fail to take action to correct it.
The blame resides with those responsible for ca.gov. They failed to keep their domain records up to date and as a result, the wrong employee was being notified to fix the problem.
Had they kept their poc's up to date, they would have been notified of the hack as well as the pending domain deletion.
Just like a government org, they try to spin themselves out of responsibility....
This was something.something.ca.gov that got hijacked
Sure, the individual machine that got hijacked probably should have been taken down until they could clean it up, though even that's not necessarily true depending on whether there were critical functions that could still be accessed even though other parts were scribbled. But that doesn't mean that somebody should take down the DNS for the entire state government without extensive discussions first
And yes, the state and counties should keep their whois data and the emails that it points to up to date - that is the kind of thing it's for. But the Feds who decided to shut down the whole state because of one machine were also seriously irresponsible. Also, I couldn't tell from the article, which said the admin who received the mail "wasn't the normal contact", whether the Fed sent mail to the listed contact or not, and whether it was the listed contact for the whole state, or the county, or the county transport agency where the problem actually was.
Few things to remember: The
Few things to remember:
The state of California is one of the larger governments bodies *In the World*.
The people who had thier machine hacked were part of a county government (Marin, where Yuppies came from). County govt's are semi-autotonomous from state govt entities.
Instead of taking the DNS record out for the server/organization in quesiton, the feds blew out the DNS record for the entire *.CA.GOV. Basiclly using a shotgun to kill an ant.
The change had the potential to affect every organization in the state (Yes, you wouldn't have even been able to e-mail Arnie to complain!). For some agencies, this effect was even a public saftey issue.. (Its still summer..fire season.. Do we really want our Dept of Forestry not to be able to e-mail? Highway Patrol?
It is a wake-up call though to ensure that emergency contact lists are kept up to date.. Including ones that go outside a single organization (or even government body!)
Just remember you're wrong *if*...
If there's a disaster in a liberal state, and the Republican federal government either hesitates to take action (NOLA), or in the case of *.CA.GOV takes drastic, overreaching action...
It's the liberal area's fault. Don't bother with nuancey definitions like a little server in Marin or something.something -- it's yer own damn fault! Even (especially) if it isn't... because nothing works better after a screwup of great magnitude than misdirecting the blame.
Here in the internet age, much business is conducted online. For the Federal government to take down the entire *.CA.GOV without calling to discuss, to negotiate, without *speaking* to the humans using that many-tiered domain on the other side of the country -- that's just plain stupid.
It means they didn't understand the ramifications of deleting the entire domain. The person who made that decision shouldn't have been in charge of making that kind of decision. If they had understood what they were about to do, they wouldn't have done it. Period.
As a unix sysadmin, I can tell you, it's a simple matter to delete a *subdomain* of a domain such as MARIN.CA.GOV. It is also a minor effort to change the NAMESERVERS of that subdomain, such that it is no longer hijacked. Really.
Politics
Well, the article doesn't clearly tell us if in fact the whois info was correctly up to date, or if the feds didn't contact the person(s) named. But still, I doubt that the records for a county department and the State of California would be out of date in such a way, and that the Feds couldn't think of ANYONE in Calif to call when their email went unanswered.
For those of you who believe in the Warren Commission report, no doubt the answer is in a whois registration. For those of us who don't subscribe to the lone gunman theory - this is two political bodies, approaching an election year, although I thought W and the Governator were on good terms untill the healthcare for children debacle last month... Or, it's two massively large, budget-challenged groups of civil servants... Hummm
This is why lots o gov is moving to other TLDs
This is one reason why so much government (especially at muni and state levels) is moving to non gov TLDs. I had to promote such a solution to avoid a similar situation at a state level. There are other issues and other responsibilities, but at least you don't have someone who is in no way responsible or beholden to you turning the lights off because of a bit of porn..... We are worse prudes than the Victorians, and instead of pulling the plug Uncle should have asked whether or not the porn was the authorized content of the County, as porn is a major industry in the valley after all.... that is unless you, like Felix, can't describe it but know it when you see it......
congrats feds
Feds are doing really gr8 job shutting down websites and irritating ppl with their wise knowledge... I don't understand why Feds are so irresponsible this much...
Eliena Andrews
Government I.T. Security Procedures?
I really have to wonder what the Government I.T. Security procedures are, after this latest gaff. Isn't the first rule of administrating anything in I.T., before making a change, make a backup copy in case you screw up whatever your changing. And the second rule, know how to restore from your backup.
The Fed employee evidently did not know one of those rules since the reason it took 7 hours to recover, because it happened at the end of a work day, it took that long to find someone who knew how to recover from the error.
Last week, in response to news of Chinese hackers had broken into Defense Department computers, I said:
The U.S. Government needs to wakeup the severity of the security problems we are currently having and be made aware just how terrible they could get. Then government needs to get serious in mandating information security protocols on sensitive material both public and private.
Until they do, I know I'm going to sleep a little less secure at night, how about you?
How much worse to these accidents need to get, than taking down all the government websites of the countries largest state, and no, I'm still not sleeping very well.