Northrop Grumman today said it will pay almost $5 million to cover the costs of network outages that slammed 26 Virginia agencies in August 2010.
The financial package to be paid by Northrop includes monthly invoice credits over the next 24 months totaling $1.908 million; $2.09 million to make operational improvements to database backup, storage systems, system monitoring, database monitoring, network monitoring and capacity management; $750,000 for "point in time" technology to protect the data held by the state; and $250,000 for the cost of the independent third-party audit performed by Agilysys after the outage.
MORE ON CONTROLLING BIG COMPUTER PROJECTS: DARPA takes aim at building iron-clad cloud services
The major findings of that Agilysys review included:
• The outage resulted from the failure of a key data storage system (EMC DMX-3) and subsequent human error during its repair.
• Loss of data and the delay in restoring operations and data resulted from the failure to follow two key industry best practices. Also, there was an insufficient degree of self-governance toward continuous process improvement and the management of risk.
• Many components selected for the data center and its IT infrastructure meet or exceed industry best practices, but that implementation falls short in certain areas.
On Aug. 25, 2010, the failure of a key data storage system owned and operated by Northrop Grumman at the Virginia state data center -- and subsequent delays in restoring operations -- caused a significant impact to the operation of several state agencies. "The delays resulted in unexpected expense to state agencies and of course made most agencies unavailable to the citizenry. Outages related to the disruption impacted 13% of the Commonwealth's executive branch file servers and 26 of 89 executive branch agencies, including, most visibly, the Virginia Department of Motor Vehicles. Of those 26 agencies, 16 reported a financial impact due to the outage," according to a release be Virginia officials.
Northrop has a $2 billion, 10-year contract with Virginia that has never gone very well, with a track record of rampant delay and other problems.
Northrop last year said it would move its headquarters to Virginia.