Cisco: Faulty clock part could cause failure in some Nexus switches, ISR routers, ASA security appliances

Cisco this week issued a notice that faulty clock timing chips in some of its switches, routers and security appliances could fail after about 18 months of service – causing those devices to crash and not recover.

The notice includes some of the company’s most widely deployed products, from certain models of its Series 4000 Integrated Services Routers, Nexus 9000 Series switches, ASA security devices to Meraki Cloud Managed Switches. Clock components are critical to the synchronization of multiple levels of a given device.

+More on Network World: Cisco amps-up Tetration platform with better security, reduced footprint, AWS cloud option+

Cisco wrote: “In some units, we have seen the clock signal component degrade over time. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months.

Once the component has failed, the system will stop functioning, will not boot, and is not recoverable. 

This component is also used by other companies. We have identified all Cisco products that have this component and worked with the supplier to quickly put a fix in place. All products shipping currently do not have this issue.

To support our customers and partners, Cisco will proactively provide replacement products under warranty or covered by any valid services contract dated as of November 16, 2016, which have this component.”

+More on Network World: Has Cisco broken out of the network hardware box?+

Specific products [and the Cisco Field Notice Number (FN)] that could have the problem include:

FN – 64230 – NCS1K-CNTLR FN – 64231 – NCS5500 Line Cards FN – 64252 – IR809/IR829 Industrial Integrated Services Routers     FN – 64253 – ISR4331, ISR4321, ISR4351 and UCS-E120 FN – 64228 – ASA 5506, ASA 5506W, ASA 5506H, ASA 5508, and ASA 5516 FN – 64250 – Cisco ISA3000 Industrial Security Appliance FN – 64251 – Nexus 9000 Series N9K-C9504-FM-E/N9K-C9508-FM-E/N9K-X9732C-EX MX 84 (Meraki) MS350 Series (Meraki)

The problem is generating quite a bit of consternation on Reddit’s Networking subreddit site.

From Cisco’s site, here are a few of the FAQs about the problem. [A full version is available here]

When did you become aware of this issue?

Cisco learned about the scope of potential customer impacts due to this issue in late November 2016. Cisco and the supplier have been working as quickly as possible to identify the root cause, impact and scope of the issue, create and test product fixes, and put in place a remediation plan for our customers.

Is this a product recall?

No, this is not a product recall. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months. For that reason, Cisco is offering to proactively provide replacement products for those products under warranty or covered by any valid services contract dated as of Nov. 16, 2016, which have this component.

Is Cisco replacing only products that have failed?

No, Cisco is offering to provide replacement products for those products under warranty or covered by any valid services contract dated as of Nov. 16, 2016, which have this component, even if they have not failed. If you have a product that has failed, please contact Cisco TAC. If you’d like to proactively order replacement products, please see the ordering instructions.

How does Cisco plan to fix the affected products? Does Cisco intend to replace them?

To support our customers and partners, Cisco will provide replacement products for those products under warranty or covered by any valid services contract dated as of Nov. 16, 2016, which have this component.

Who supplies the impacted component?

As a matter of policy, Cisco stands behind the reputation of our products. We do not intend to publicly name the supplier.

Is this issue Cisco-specific?

No, other companies also use this component from the supplier in their products.

What other companies are seeing this issue?

Cisco does not comment on other companies.

Will you be replacing products, repairing products, or offering discounted upgrades?

To support our customers and partners, Cisco will provide replacement products for those products under warranty or covered by any valid services contract dated as of Nov. 16, 2016, which have this component. Customers with these products should review the information within the field notices to replace these products. Due to the age-based nature of the failure and the volume of replacements, we will be prioritizing orders based on the products’ time in operation.

How long will replacements take?

Due to the age-based nature of the failure and the volume of replacements, Cisco will be prioritizing requests based on the products’ time in operation. Therefore, fulfillment requests may be delayed and your request may be fulfilled in multiple shipments.