Hey, dummy!

What you don't understand is the nature of the patch. SQL server patch 3 is around 50 megabytes, and can take 2 to 6 hours to apply, with the database down for that long because you can't afford a stand-by. And, many people didn't apply patch 2 because it broke some processes. So, if 2 is bad did you want to load 3? It is no surprise to me that so many servers were unpatched.

Good God, I obviously had no idea. So Microsoft needs to come up with a better way to apply patches, then?

Many of the SQL application vendors had not yet announced support for the SP3 patch. If we had applied SP3 we would have not been able to get support.

The patch to fix the buffer overflow vulnerablity was addressed in MSSS02-039. The actual file size of the patch is only 164KB and doesn't require that you reboot the server, only stop and restart the SQL service.

The file sizes mentioned in the previous comments are for the entire SQL Service Pack 3 download. SQL SP3 covers over 90 fixes!

Everyone knows it is important to keep up on the latest issues. If I find my router/switch has a security problem, I take steps to remedy it. Not taking steps can lead to a very localized outage. Not patching this SQL issue caused a Global problem for millions of users and untold thousands if IT staff for many, many days.
Then you have to go fix your software issue anyway.
As the U.S. Military would say, I Don't Want Any Excuses - Just Actions.
Tom

Adam -

Even if everyone had patched their server, it was te aggressive nature of the search for servers that brought down networks - so don't blame the poor, over-worked SysAdmins!

-dave

To those who complain about it taking a while to apply the patch ("the database would be down too long"), a couple of questions: How long was your database down from the attack? What was the CUMULATIVE downtime of all the banks and other organizations because people didn't apply the patch? I suspect the answer sounds like "a LOT longer than it would have taken us to apply the patch to our systems."

This was not just sysadmins who did not patch their SQL servers. We had an SQL serever here, it was patched and was not hit. The two machines we had that were hit were 2000 workstation running visual studio. This effected practically all the MS development environmrnts as well. Now the owners of these machines ran windows update regularly, and had all the critical updates, and most of the others installed. I would say that this was a failure of the MS update system. I know that visual studio is an add on product, and windows update is only for the OS, but when an MS product changes the basic networking behavior of the system, the update site should at least tell you that you need to go get a patch from somewhere else. After all, who has ever heard of a C compiler that opens up network ports to the world?

As I sit and think about it and try as I may to find blame for this I can only think of one thing. We(The IT community) are in trouble for the rest of our careers if we don't take our job serious and do whatever it takes to keep up with Johnny hacker and Gary worm writer and stop blaming it on everyone else (Microsoft patches Ver. Blah Blah Blah). Because who ever they are and whatever their purpose ( most american kids) we all suffer so get off the pot or P***.

None of us knows the criticality and risks of the other's business systems. A smart company has a predifined strategy for patching their systems. This is done for a reason. These strategies range from conservative to "install anything as soon as you know about it". Some businesses cannot risk installing patches when they are first released. Some businesses are restricted to testing and installing patches that are included in bundles (e.g. SP3). Some businesses must wait for their application software to support the fixes in patches. Some businesses have a mandated "burn in", or test period for patches before they are allowed in production. If you can risk your business to installing whatever patches or fixes Microsoft (or any vendor) puts out on a whim, then go ahead. Of course, if you do that then you spend all of your time installing Microsoft patches (with as many as they put out on a regular basis).

Placing blame will not solve anything. Let's get off of each other's backs and see if we can use our energy to develop a better solution that simply placing blame.

Patches? First of all.. they don't appear in critical updates (right ok I understand.. that's just the O/S)... do you realize though how long it would take to install every patch for every piece of software on our network.. and to CHECK for that every day? Also, M$ can not verify that a patched system was neccessarily safe from the attack.

In addition, why were patches not installed? Because usually the patches require a reboot. Try Linux.. you can patch and eveything is hunky doory.. but on a W2K system.. you patch and ooops we gotta reboot.. that's just not something you can do in a production enviroment. I hold no one but M$ responsible for what happened. It was gross neglegancy on their part to release software with this kind of an exploit in it (time and time again!)

As a fair followup, we have many MS SQL servers, mostly unpatched. We have a big Internet pipe. We were not affected because of sufficient network security design. Also, we are working on the patches.

Do any of you following this thread have an example of what your security policy states with respect to patching your systems? I imagine there must be some guidelines provided - assuming there is a security policy. However, this sounds like something easier said than done.

Anyone exposing an unpatched server to the internet is got to be asking for trouble. Like most busy IT people, we don't stay totally up-to-date with the latest patches. But we DO ensure our firewalls have only the bare minimum of ports open -- and the MS SQL Monitor port certainly isn't one of them. The only thing we knew of the worm was lots of blocked packet messages in our firewall log.

The Slammer worm is Reason #65536 to NOT use Micro$oft
products for your mission-critical systems. How
many more reasons do you need?

And if the theory "Micro$oft is the most popular,
therefore its the most hacked, and if we switched
we'd just get hacked on that" were true, then
there'd be scads more Apache hacks than there are for
IIS - guess which one is the most-hacked webserver?
Hint: Its not Apache.

(Note: I didn't say Apache never got hacked, I said that
IIS gets hacked a lot more).

The only thing that amazes me about Slammer is how
many people seem to enjoy getting reamed every time
the latest iteration (Nimda, Code Red, Slammer....)
of attack on Micro$oft's crappy software comes out.
Admins of all-M$ environments have got to be the
world's biggest masochists. I'm surprised ya'll
don't install stun-guns in your chair seats to zap
you every time a box blue screens or a service dies
unexpectedly.

That any corporation worried about its
bottom line would continue to depend on Redmond
for mission-critical software is nothing short
of incredible. Rapacious licensing costs, EULA's
hidden in patches that radically alter software
licensing terms, buggy software released as
production code, lies about products and competitors,
bloatware and vaporware: why do business with
Billy when you don't have to?

Much has been said about the patches and considering the time lag (six months) there has been ample time to verify that closing the port didn't break anything else. What is really unconsionable is having the port open to the internet in the first place. This is a back-end app that should have been protected by a firewall. Even a cheap lowly packet filter would have prevent ingress and the egress of the worm effects. Can you say 486 door-stop running ip-chains. Cost $20.00

Let's not let Microsoft off the hook too lightly. See this article as an example of why not:

*** snip ***

Microsoft fails Slammer's security test
By Robert Lemos Staff Writer, CNET News.com
January 27, 2003, 4:27 PM PT
http://news.com.com/2100-1001-982305.html

Internal memos show that the software giant hadn't patched its own network against the Slammer worm, causing many of its services to fail.

From the article itself:

Microsoft's policy of relying on software patches to fix major security flaws was questioned Monday after a series of internal e-mails revealed that the software giant's own network wasn't immune from a worm that struck the Internet last weekend.

*** snip ***

Kind regards,

Russell

There's a ton of reasons this didn't happen:

1. Support - Many applications only support a particular version of SQL Server and you void your support if you apply a patch they haven't certified.

2. Overworked SQL Admins - We have dozens of SQL Servers, but only 2 SQL Admins. If they applied every patch MS came out with, that's all they'd be doing. Do you apply every patch or try to get by until the next major service pack?

3. Testing - We don't apply a patch ad hoc to our PROD SQL Servers before it runs through testing in our DVLP and DEMO environments. We can't afford to have a patch break our PROD SQL Servers and cause downtime experienced by our customers, both internal employees and external customers. This takes time. If you did it for every patch and service pack, you'd be here 24 x 7 x 365.

4. Availability - We can only take PROD servers down on the weekend evenings for maintenance. With 2 SQL Admins and them having lives outside of work, being here every Saturday night is not practical.

5. Network Security - We keep our internal network locked down pretty well, so our servers were protected from the virus by our firewalls. We'll still be applying SQL SP3 to cover all the bases, but we were minimally affected by the worm, even the internet slow down.

I put most of the blame on network security - almost none of our SQL servers were patched but we had no problems at least partly because we don't have our back end servers on the 'net. And if I understand correctly rebooting the machine is sufficient to stop the worm, even bringing an "infected" laptop inside the firewall wouldn't be an issue unless it was brought in running.

So please correct me if I'm wrong but if one's behind a firewall unless 1434 is poked thru to a SQL machine, there's no way for the worm to infect?

Following up on Jerry's comment, I saw on one newsgroup where a site got infected when one of their user's with a laptop and MSDE installed VPN'ed into their network. The worm went off the 'net, onto the laptop infecting MSDE, ran down the tunnel and onto the back-end SQL servers. Makes you think, doesn't it?

One of the major problems with applying these patches in my network is that it takes too long to find affected computers in the network of thousands of computers, then get to them, and get the patch run. May be less of an issue with SQL in particular (would assume that admins know about most SQL Servers in the network, so finding affected machines should be easier), but remember that MSDE crops up pretty frequently on user's computers that you don't expect.

The only solution that I've seen that works for this problem is a patch management solution that detects the computers in the network that need vulnerability remediation, and fixes them from a central console. Using one of these solutions (Bigfix Enterprise in my case) has worked well for us so far.

The SLAMMER amplifies the root cause of the SLAMMER.

It is not MICROSOFT, not IT, not ISPs nor VP of anything.

It is the general all around lack of QUALITY in the Internet.

MS has a flaw caused by the lack of QUALITY, the patch is very difficult to apply because the lack of QUALITY, the IT does not apply the patch because of the lack of QUALITY, the ISPs do not apply the patch because of the lack of QUALITY and the VPs do not manage for QUALITY from its employees or FOR its customers.

Yet, so many of the various players are ISO xxxx certified for: QUALITY.

If the players had been proactive and QUALITY focused, the Planet Earth Internet would be largely safe from 'data terrorists' and mis-management.

An analogy; you cannot make a QUALITY car by enforcing QUALITY at the end of the assembly line. QUALITY starts at the top (are we going to make a QUALITY car?) and QUALITY is obvious in every department all the way thru the floor sweeper.

QUALITY pays, QUALITY is not overhead to be reduced. QUALITY reduces overhead.

Who is minding the Internet store?

Gene Thomas

This message is in reference to Gene Thomas "Quality message"....
what do you propose ?

First, my systems were patched, so I did not contribute to the problem this time.

However, I am getting sick of people saying that 'lazy admins' are the real problem, or 'Don't blame the vendor cause the fix was out there'.

The problem is not lazy admins - it's exhausted admins. Keeping on top of patches (and not just Microsoft - but every device on our network from the router to the Nic card on the printers), plus planning and testing for the future, plus keeping a lid on our user community is one heck of a balancing act. Add in tough economic times and working with decreased resources and it's no wonder this sort of problem doesn't happen more often!!

Instead of placing blame, why don't vendors (and magazine/newspaper writers) invest some energy in making the lives of system administrators a bit easier. A good place to start would be creating an industry-wide standardized patch delivery protocol and announcement system. If I had one or two sites to check for patch information (instead of the 46 I do now), would be a good first step.

Who agrees with me?

In response to Dave Kearns article on Microsoft's responsibities and the Slammer worm -

I just finished reading Daves article about Microsoft and the Slammer worm on Tech Republic, and I want to thank him for not joining the long list of people eager to point fingers towards Redmond when something goes wrong.
I am an MCSE and also carry several other MS certifications, as well as tickets from CompTIA and Cisco. I am not in awe of Microsoft, or partial only to MS products, but I am very much aware of what Microsoft's products mean to my career. I get a little tired of people pushing the blame for every security breach or design failure on Microsoft's shoulders when there are many other people with equal responsibility for what happens to their equipment.
Any network is only as good as the people who are responsible for the maintenance and repairs required for it's smooth operation. I know what it takes to keep up with the task, but that's part of what I get paid to do. If you are unable to get what you want from your operating environment, then by all means you should migrate to another product, such as Linux or Unix. However, as Dave said, I guarantee that if those OS's were to become the most widely used, they would also become the number one targets for security breaches and denial of service attacks. There will then be an equal amount of work installing upgrades and patches to those operating systems, and an equal responsibility on administrators to keep up with them.
Yes, Microsoft slipped up with SQL 2000 and the difficulty of patching the software, but MS has always been quick to step up and face the music. If they need to patch something, they patch something, making that patch widely available, and usually easy to install.
Microsoft also continually issues upgrades to their operating systems that are not integral to their functionality or security, but merely enhancements to the end users environment. I don't know about you, but I believe I would faint dead away if Chevrolet called me and said they had come out with a better CD player and would like to install it in my truck at no cost to me.
There are several areas where I use Linux products to do the job because they are better suited, and I am always open to new products that will make my job easier or more efficient, or will make my users jobs more enjoyable. However, at this moment in time, Microsoft appears to have that niche.
If someone develops a better product, I will certainly be open to adopting it, and possibly even replacing Redmond’s software completely if it is the most cost effective, secure, and efficient option, but until that day, people should spread the blame equally across the board. That means standing up and admitting the problem when lax administrative practices were to blame for server downtime if a patch advised by any manufacturer has not been applied. It also means that that the responsibility for avoiding any kind of attack with such far-reaching implications should fall on the shoulders of everyone who influences the development, maintenance, and monitoring of networking software and equipment.

Chuck Warren

Sorry, the article was on Network World. I read too many things back to back that morning......

Doesn't anyone know of a product or a realiable way of checking multiple servers for the "latest and greatest" O/S patches? It seems that we should be able to automate this task..... How about application patches, does anyone know of a product that checks here for current patches?

Post a comment