Counterfeit network gear has appeared in the channel in increasing numbers and could be in your organization.
Nor did he get an answer to the most important question: How did a registered Cisco reseller (also a platinum Network Appliance partner and gold partner to Microsoft and Symantec) acquire the counterfeit WICs in the first place?
What he didn't know was that phony network equipment had been quietly creeping into sales and distribution channels since early 2004, when manufacturers began seeing more returns, faster mean-time between failures and higher failure rates, says Nick Tidd, vice president of North American channels for 3Com and president of the Alliance for Gray Market and Counterfeit Abatement (AGMA).
Counterfeit gear has become a big problem that could put networks - and health and safety - at risk. "Nobody wants to say they've got counterfeit gear inside their enterprises that can all of a sudden stop working. But it's all over the place, just like pirated software is everywhere," says Sharon Mills, director of IT procurement organization Caucus.
How to recognize the real deal
Because clones and packaging are getting more realistic, many people don't realize they have counterfeit network equipment until it's installed and begins acting quirky. Outages and failures are often the tip-off that the gear is fake.
* Don't shop on eBay for deeply discounted gear, particularly from sellers in China.
* Don't go outside your trusted channel to buy critical network components.
* If you're in the market for refurbished gear, the safest bet is to purchase certified products through the manufacturer.
* Check serial numbers against the vendor database.
* Check the packaging carefully, inspecting for anything out of the ordinary in the logo, size and type of packaging materials by comparing them with others in the same shipment.
* Closely examine the gear and compare holograms and chip sets.
10% counterfeit suspected
There are no statistics specific to network hardware counterfeit rates. But according to a white paper by AGMA and consulting company KPMG, counterfeit products account for nearly 10% of the overall IT products market.
"That's $100 billion in fake memory sticks, drives, monitors, networking gear and other IT products floating around out there in black and gray market channels. This has huge implications for the enterprise," says Tidd, who became involved in his first counterfeit case in 2001. That case led him to a Canadian reseller who also was under investigation by HP. Out of that case, 3Com and HP, along with Cisco and Nortel, founded AGMA.
Vendors and resellers started seeing counterfeit in the gray-market channel where used and refurbished products are sold, says Phillip Wright, director of worldwide brand protection for Cisco, which is the most counterfeited brand. That's when the supply of out-of-box secondhand equipment from the dot-com fallout dried up.
"Users got a taste for new used equipment at bargain prices. So counterfeiters moved in to meet the demand," Wright says. "It didn't help that some resellers turned a blind eye to possible counterfeit so they could keep their own revenue streams going."
The vast majority is still being purchased from gray market, uncertified resellers who unload their goods on eBay at extremely low prices, says Scott Augenbaum, supervisory special agent for the FBI Cybercrime Fraud unit in Washington, D.C.
These parts sometimes move sideways into the hands of legitimate resellers and integrators.
"Recently, I did some voice over IP integration for a client in Huntsville, and the engineer there asked if he could pay me with five extra VoIP network cards he had left over from the project," says Neal Rauhauser, founder of Layer 3 Arts, a system integrator in Omaha. "I got four cards I could use, and one that was counterfeit."
Fortunately, Rauhauser never installs anything before checking it first. He's wise to counterfeits, having had his first run-in with such products in 2004, when two of six new Cisco 1721 routers started acting up at one of his client sites, a large auto manufacturer in Michigan. They turned out to be counterfeit, and he has since been campaigning against counterfeit products.
There were visible differences between the counterfeit and the real gear, he says, but only after close inspection. The counterfeit VoIP card had a brand-new box even though the card was 4 years old. He also noticed discrepancies in packaging and labeling.
"The printing on the bar-code label was fuzzy like it'd been printed off a low-quality printer instead of a laser. And its internal packaging was a plastic bag instead of a plastic box like the others," Rauhauser says.
He contacted the customer who gave him the product, and the customer admitted he bought the cards off eBay. The four good cards came from a reputable seller. The bad card came from TFS Systems, which claims to be a Cisco registered reseller that buys only from Cisco's top-tier distributors. Rauhauser took pictures of the differences in products and called TFS to find how they wound up selling counterfeit product to his client.
"They were ready to pull my leg and tell me I was wrong. So I told them I was going to the FBI," Rauhauser says. "Then they asked me to box it up again, keep it pristine and they'll get me my money. I'm sure they sold it again on eBay right after they got it."
In the MortgageIT case, Bruner figures his representative at Atec got burned when she went outside her normal supplier to purchase the cards in late 2004.
"We were notoriously cheap with our equipment purchases, so she might have bought from someone besides Ingram, her usual supplier, to get us a better bargain," says Bruner, who left MortgageIT in July, shortly after Deutsche Bank signed an agreement to buy the company.
How Atec came into possession of the counterfeit WAN interface cards can only be hypothesized because repeated calls and e-mail to Bruner's former representative at Atec, and to the company vice president, were not returned. The company's operations manager says MortgageIT was a big client, and sales representatives don't see the gear that's being shipped to their clients.
Gambling on quality
No matter how the counterfeits got into MortgateIT's authorized channel, such slippages highlight the complexities of dealing with this problem - not just in the sales and distribution channels, but also in the manufacturing supply chain, says Pete van de Gohm, director of IT security and quality at Bayer.
AGMA's Tidd acknowledges this, adding, "In some geographies, you've got resellers and distributors blending their inventories, which is why a single shipment might contain five good and five counterfeit parts."
It's difficult to control past the distributor layer, Tidd says, especially when Cisco has 28,000 registered resellers, 3Com has 3,000 and so on.
That means organizations face loss of equipment that vendors may or may not support (Cisco handles on a case-by-case basis). They also could experience critical network outages that, in the right circumstances, could affect human health and safety.
"What if it wasn't a bank subnet that went offline because of a faulty card in the router? What if it were an air-traffic control network instead?" van de Gohm asks. "This is no different than counterfeit medicine in the pharmaceutical industry. And it's potentially just as life-threatening."
Such concerns also grip the network vendors whose reputations and brands are at stake if they can't stop the dumping of counterfeit parts into the channel. "We worry about things like wiring in the motherboard overheating and the potential for network outages that would impact personal health and safety," Wright says.
Efforts to fend off fakes
Manufacturers are working on ways to make their products harder to clone through use of packaging labels, logos and three-dimensional holograms. Vendors such as 3Com are working on tagging systems, and cryptographic machine authentication is a viable option to help devices call home.
For the past few years, Cisco and 3Com have been building anticounterfeit culture into every level of their product-to-market channels, educating suppliers and distributors about what they need to do to protect their own channels, while building international investigative teams to help law enforcement agencies shut counterfeiters down.
Cisco's 30 investigators stationed worldwide are dedicated to 200 active counterfeit cases at any given time. From the Mandarin characters on the back of his business card, it's clear that Wright spends a lot of time in China. And a whiteboard behind Wright's desk has a hand-drawn diagram titled "Stopping the counterfeit flow," which contains multiple loops back to Chinese distribution and law enforcement intervention points.
According to Wright and Tidd, China is the source for most counterfeit gear. Tidd toured multiple floors of counterfeit consumer electronics and network gear last year at a public shopping mall in Shenzhen, China.
"3Com has done raids in China, cooperating with local law enforcement who've shut down factories and seized counterfeit goods. Once they've done the seizure, we go in and try to figure out how many boxed products went out before they were shut down," Tidd says. "Unfortunately, as fast as you shut the factories down, other factories go back up."
According to the AGMA study, the United States is the second major point of origin for counterfeit goods - California, in particular, say Rauhauser and Dana Andrews, owner of Digitial Surplus in Boston. They point to the port of Los Angeles as a big dumping ground for Chinese counterfeit parts and to Silicon Valley as a place of production.
Since 1994, Andrews has made a pastime of helping federal agents catch criminals he says are polluting the reseller channel and costing him his business. In February, he helped the FBI catch a fraudulent buyer who had set up a phony escrow company and tried to scam Andrews out of half a million dollars worth of Cisco gear.
While several law enforcement agencies contacted for this story won't talk about specific cases, a June raid on Sun Valley Technical Repair in Morgan Hill, Calif., could turn out to be a big case of counterfeit in Silicon Valley.
Reports in the San Francisco Chronicle made it appear at first like an immigration raid, as 12 illegal immigrants (11 from Mexico and one from Colombia) were taken away. But that wouldn't explain the presence of so many agencies, including the FBI, the U.S. Immigration and Customs Enforcement, the U.S. Postal Service and the Rapid Enforcement Allied Computer Team, which investigates large-scale, high-tech piracy and counterfeit cases.
Law enforcement efforts are helping to shut these factories down, say Tidd and Wright, who is active with Business Action to Stop Counterfeit and Piracy, which is sponsored by the International Chamber of Commerce.
H.R. 32, the Stop Counterfeiting in Manufactured Goods Act, signed by President Bush in March, should also make a strong deterrent, experts say. The act sets prison terms of as many as 20 years and fines of as much as $15 million for counterfeiting in what the International Anticounterfeiting Coalition praises as a direct response to "dangerous international counterfeiting problem that is threatening the U.S. economy, costing U.S. jobs and harming citizens."
Industry leaders need to do more to keep counterfeit out of the distribution channel, resellers and users say, before it affects public safety.
"The networking industry should reach out to other industries that have problems with counterfeit parts," van de Gohm adds. "The industry should apply the best practices already learned in the auto, pharmaceutical, airplane and other industries where counterfeit parts could result in loss of life."
Radcliff is a freelance writer. She can be reached at deb@radcliff.com.