Michael Cooney
Senior Editor

DARPA: Big data needs Big search, Big analytics

Opinion
May 27, 20144 mins

Defense Advanced Research Projects Agency programs to look at changing search, analytics technology

shutterstock 434825713 big and small light bulbs
Credit: patpitchaya / Shutterstock

How you might one day most effectively negotiate the enormous amounts of data and video around the Internet was on display recently as DARPA’s Information Innovation Office hosted Demo Day 2014 to highlight the agency’s ongoing contributions to what it called “preserving and expanding IT superiority.”

Among the over 100 products on display were a couple looking to address future issues such as vastly improving search functions and analyzing tons of data.

+More on Network World: DARPA wants to scrub scourge of counterfeit computer gear+

DARPA says the existing methods of searching the Internet for all manner of information just won’t cut it in the future as the “one-size-fits-all approach to indexing and search of web content limits use to the business case of web-scale commercial providers.”

To go beyond that DARPA is working on a program known as Memex which looks to devise “better methods for interacting with and sharing information, so users can quickly and thoroughly organize and search subsets of information relevant to their individual interests. The technologies developed in the program would provide the mechanisms for improved content discovery, information extraction, information retrieval, user collaboration and other key search functions.” DARPA says.

In its announcement of the program DARPA said it expects Memex to:

  • Develop of next-generation of search technologies to revolutionize the discovery, organization and presentation of domain-specific content
  • Create of a new domain-specific search paradigm to discover relevant content and organize it in ways that are more immediately useful to specific task
  • Extend of current search capabilities to the deep web and nontraditional content
  • Improve interfaces for military, government and commercial enterprises to find and organize publically available information on the Internet

While Memex technology will focus on search, DARPA’s Big Mechanism program aims to change the way tons of data are analyzed. 

DARPA Big Mechanism Program Manager Dr. Paul Cohen said the program looks to develop a system that could autonomously gather all exiting data about a particular topic, keep it up-to-date and develop new conclusions or research directions.

“We are looking to change the way people develop knowledge,” Cohen said.

For example, today researchers struggle to keep up with relentless streams of relevant publications. To stay current, a researcher must specialize, becoming expert in a small part of something much bigger. The vision for the Big Mechanism program is fundamentally different:  Every publication would immediately become part of a public, computer-maintained, causal model of a complicated system-a big mechanism-and every aspect of a big mechanism would be tied to the data that supports it or contradicts it.  To the extent that we can automate the construction of Big Mechanisms, we can change how science is done,” DARPA said.

In a nutshell the Big Mechanism program will develop technology to read research abstracts and papers to extract fragments of causal mechanisms, assemble these fragments into more complete causal models, and reason over these models to produce explanations.

DARPA said it will aim the Big Mechanism program at cancer research first, specifically cancer pathways or the molecular interactions that cause cells to become and remain cancerous.

The Big Mechanism program has three primary technical areas: Computers should read abstracts and papers in cancer biology to extract fragments of cancer pathways. Next, they should assemble these fragments into complete pathways of unprecedented scale and accuracy, and should figure out how pathways interact. Finally, computers should determine the causes and effects that might be manipulated, perhaps even to prevent or control cancer.

Check out these other hot stories: