Criminal negligence

U.S. Border Patrol agents intercept a man trying to enter the U.S. illegally from Mexico. Unaware that he is wanted by the FBI for three murders, they return him to Mexico. The man returns to the U.S. and murders several more people before being caught.

A team of investigators works for 20 years to bring down an international drug-trafficking organization. Had they known about related information in other law enforcement databases scattered across the U.S., the case might have been closed in three.

True stories like these have highlighted the critical need to improve information sharing among law enforcement organizations, but it wasn't until the 9/11 attacks, the subsequent 9/11 Commission Report and a presidential mandate that better information sharing became a top priority.

The initiatives that arose from that mandate are finally beginning to open up stovepiped data repositories by transforming how law enforcement agencies at the federal, state and local levels capture, store and share data.

The biggest changes have come in two areas: how law enforcement identifies bad guys, and how investigators gain access to incident reports documented by more than 20 federal agencies and 20,000 state, county, local and tribal law enforcement organizations nationwide. "You'll be able to search data that you never had access to before," says Tom Bush, assistant director in the FBI's Criminal Justice Information Services (CJIS) division.

Most of the improvements in data sharing flow from the development of the Global Justice XML Data Model, a standard that provides a common vocabulary and structure for the exchange of data among law enforcement databases. Initiated by the U.S. Department of Justice , GJXDM was released in 2003. "By 2004, there were projects all across the country using it," says Paul Wormeli, executive director of the Integrated Justice Information System Institute, a public-private partnership that helped develop the standard.

In 2005, CIOs at the DOJ and the U.S. Department of Homeland Security agreed to build the National Information Exchange Model (NIEM), an extension of GJXDM that facilitates data sharing beyond law enforcement to the areas of justice, public safety, intelligence, homeland security, and emergency and disaster management. Work is also beginning on direct computer-to-computer data exchanges using Web services. "This field is waking up to service-oriented architectures," says Wormeli, noting that some reference architectures are already in place.

These standards are designed to solve the problem of proprietary and incompatible law enforcement record management systems without requiring every organization to throw out what they have and start over. "The beauty of NIEM is that it preserves the legacy systems. We're building middleware," says Wormeli.

Most of the identity databases at the federal level aren't yet NIEM-compliant, but agencies are planning upgrades to those systems and have already taken steps to facilitate data sharing. Although federal agencies use many databases for law enforcement, the three primary identity databases are the FBI's Integrated Automated Fingerprint ID System (IAFIS); the DHS's IDENT fingerprint database of 90 million foreign nationals, gathered from visa applications and used at all points of entry; and the U.S. Department of Defense's Automated Biometric Identification System (ABIS), currently used to monitor foreign nationals entering and leaving U.S. military bases in Iraq and Afghanistan.

IDENT, IAFIS and ABIS are all capable of some data exchanges by way of GJXDM today, but each is being reworked to natively support the NIEM standard and allow data exchanges with databases in fields outside of law enforcement, such as emergency management. IDENT is in the process of being updated, and contracts to develop the next generations of IAFIS and ABIS, which will add facial and iris image-recognition capabilities, were awarded in February.

Broken Records

The second half of law enforcement's silo problem is the inability to access incident reports.

Agencies share information on criminals and arrest records with the FBI, but the incident reports, which detail the crimes, remain isolated in thousands of federal, state, county and local record management systems. Those records, consisting of structured and unstructured data, are the lifeblood of investigations, says Maj. Chris Brown of the Oregon State Police.

Although 75% of police agencies use automated systems to store those records, less than 25% of those systems are capable of sharing that information, says Dan Hawkins, director of public safety programs at Search, a national consortium of state agencies that promotes information sharing.

Regional data-sharing networks have sprung up around several metro areas, but there is currently no way for investigators to access all of the disparate record management systems across the country. That ability to "connect the dots" is important not only for FBI trending and analysis, but also for wide-ranging investigations, such as Brown's ultimately successful 20-year pursuit of an international drug ring. In that case, he says, "the scope of the organization, the number of places involved and the distribution of people presented an incredible challenge to investigators."

So last March, the DOJ and the FBI's CJIS division began rolling out the National Data Exchange initiative (N-DEx), a NIEM-compliant database and data-sharing network. N-DEx was designed to gather and exchange incident and case reports, as well as arrest, incarceration and parole records, and other data with all NIEM-compatible systems in local, state, tribal and federal agencies.

Both the FBI and the DOJ wanted to have federated search capability across incident reports residing in state and local record management systems nationwide while allowing those records to be updated and maintained by their local owners. "The locals maintain possession, but we have visibility into their sharable information, and they have similar visibility into ours," said Vance Hitch, Justice Department CIO, in an e-mail exchange with Computerworld .

"Within the system, we'll do correlation of data, pull out entities [incident data] and provide the ability to search the data," says program manager Kevin Reid. Investigators can use the system to make connections among incidents that might help to identify and track down suspects, says Brown.

In the first phase of the $85 million project, N-DEx will incorporate about 100 million records, including records from federal agencies. Initially, records will come from case management systems at the FBI and the Air Force Office of Special Investigations, followed later by those of the Bureau of Alcohol, Tobacco and Firearms, the Drug Enforcement Administration, the U.S. Marshals Service, and the Bureau of Prisons, says Reid.

The regional data-sharing networks are also being connected. Initial deployments include networks in Delaware, Oregon, Nebraska, Texas, Ohio, San Diego and Los Angeles.

In this phase, 50,000 law enforcement users will have access to the N-DEx system. The next step will be to support Web services access and expand the user base to 100,000, says Reid. Ultimately, the system will have about 200,000 users and contain 250 million records. CJIS plans to add tools to enable investigators to work together on cases that cross borders. Investigators will be able to use N-DEx to create virtual regional information-sharing systems and form joint task forces on the fly, says Reid.

Brown was an early adopter of N-DEx and is a true believer in the system. If N-DEx had been at full capacity when he was working his drug investigation, he says, "we would have been able to do this in two to three years instead of 20."

Linda Rosenberg, director of the Pennsylvania Office of Criminal Justice Improvement, credits CJIS with doing "a tremendous job" with N-DEx. The state has 1,200 municipal police departments and no central department of public safety, so tying those disparate systems together has sometimes looked like an insurmountable challenge. "Now you don't have to go back and build these data warehouses and totally redo your entire infrastructure," Rosenberg says.

Chicken and Egg

For the system to work, the information needs to flow in both directions. "That's the challenge," says David Gavin, assistant chief of the administration division at the Texas Department of Public Safety, which runs a regional data-sharing network known as T-DEx. "How do you get all of the record management systems in the country to export in that format so that they can participate and not just access [N-DEx]?"

Regional law enforcement networks will want to tie in, but connecting multiple record-management systems will be challenging. To facilitate that, the Office of Justice Programs' Community Oriented Policing Services program at the DOJ last year awarded $159 million in technology grants, with one caveat: Any record management system project is required to be NIEM-compliant.

Moreover, several vendors of record management systems have been mapping law enforcement agency data to the NIEM-standard format free of charge in hopes of getting future upgrade contracts, says Reid.

The DOJ's objective is to have all 20,000 agencies online within three years, but Reid is more optimistic. "By 2009, I think we'll have the majority of the country participating," he says.

That may be enough time to get the major regional information-sharing systems linked up, but Hawkins thinks it will take much longer for the rest of law enforcement community to follow along.

And Brown isn't so sure that things will proceed smoothly. Most record management systems in use by law enforcement are so highly customized that they often can't even share information with other localities using the same software, he says. Integration is expensive.

The federal government set aside $85 million to complete the N-DEx back-end systems and allocated nearly twice as much in grants last year to help state and local agencies update and connect their record management systems. But state and local officials say the federal government needs to spend much more to get everyone's data connected -- a critical step to making N-DEx truly useful.

"I don't believe that there is the federal funding to make it happen," says Hawkins, noting that the $159 million in grant funding last year went to just 37 out of more than 20,000 agencies nationwide. Barring a major increase in federal funding, Hawkins says that it could be 10 years before the majority of agencies are online with N-DEx.

Rosenberg is also doubtful. Despite the $159 million, "the pot of money [from the DOJ] that's used by state and locals for information sharing has been cut by two-thirds," she says. Rosenberg says she worries that without more federal dollars, smaller agencies will simply forgo uploading their own data.

Hawkins also worries about unanticipated integration issues. "There's still a lot of testing to be done as to what NIEM-compliant means," he says.

But Reid says mapping data to GJXDM and validating the data isn't that complicated. "All they need is an XML mapping tool," he says.

Wormeli sees a bright future for data sharing in law enforcement. "We have the standards, we have the architectures, and for the first time, the president has created an information-sharing policy," he says. "There's a feeling of collaboration."

This story, "Criminal negligence" was originally published by Computerworld.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2008 IDG Communications, Inc.