DARPA: Big data needs Big search, Big analytics

Defense Advanced Research Projects Agency programs to look at changing search, analytics technology

How you might one day most effectively negotiate the enormous amounts of data and video around the Internet was on display recently as DARPA's Information Innovation Office hosted Demo Day 2014 to highlight the agency's ongoing contributions to what it called "preserving and expanding IT superiority."

Among the over 100 products on display were a couple looking to address future issues such as vastly improving search functions and analyzing tons of data.

+More on Network World: DARPA wants to scrub scourge of counterfeit computer gear+

DARPA says the existing methods of searching the Internet for all manner of information just won't cut it in the future as the "one-size-fits-all approach to indexing and search of web content limits use to the business case of web-scale commercial providers."

To go beyond that DARPA is working on a program known as Memex which looks to devise "better methods for interacting with and sharing information, so users can quickly and thoroughly organize and search subsets of information relevant to their individual interests. The technologies developed in the program would provide the mechanisms for improved content discovery, information extraction, information retrieval, user collaboration and other key search functions." DARPA says.

In its announcement of the program DARPA said it expects Memex to:

  • Develop of next-generation of search technologies to revolutionize the discovery, organization and presentation of domain-specific content
  • Create of a new domain-specific search paradigm to discover relevant content and organize it in ways that are more immediately useful to specific task
  • Extend of current search capabilities to the deep web and nontraditional content
  • Improve interfaces for military, government and commercial enterprises to find and organize publically available information on the Internet

While Memex technology will focus on search, DARPA's Big Mechanism program aims to change the way tons of data are analyzed. 

+More on Network World: Old electronics don't die, they pile up+

DARPA Big Mechanism Program Manager Dr. Paul Cohen said the program looks to develop a system that could autonomously gather all exiting data about a particular topic, keep it up-to-date and develop new conclusions or research directions.

"We are looking to change the way people develop knowledge," Cohen said.

For example, today researchers struggle to keep up with relentless streams of relevant publications. To stay current, a researcher must specialize, becoming expert in a small part of something much bigger. The vision for the Big Mechanism program is fundamentally different:  Every publication would immediately become part of a public, computer-maintained, causal model of a complicated system-a big mechanism-and every aspect of a big mechanism would be tied to the data that supports it or contradicts it.  To the extent that we can automate the construction of Big Mechanisms, we can change how science is done," DARPA said.

In a nutshell the Big Mechanism program will develop technology to read research abstracts and papers to extract fragments of causal mechanisms, assemble these fragments into more complete causal models, and reason over these models to produce explanations.

DARPA said it will aim the Big Mechanism program at cancer research first, specifically cancer pathways or the molecular interactions that cause cells to become and remain cancerous.

The Big Mechanism program has three primary technical areas: Computers should read abstracts and papers in cancer biology to extract fragments of cancer pathways. Next, they should assemble these fragments into complete pathways of unprecedented scale and accuracy, and should figure out how pathways interact. Finally, computers should determine the causes and effects that might be manipulated, perhaps even to prevent or control cancer.

Follow Michael Cooney on Twitter: nwwlayer8 and on Facebook

Check out these other hot stories:

Selfies in space: Boeing, Samsung team for cosmic mobility

Bell Labs offers $100,000 prize for game-changing information technology

NASA's broken planet-hunter spacecraft given second life

US Navy wants smart robots with morals, ethics

Federal car fleet to become test bed for high-tech safety gear

"Game of Thrones" author like DOS, hates spellcheck

Quick look: What's hot with 3D printers?

10 crucial issues around controlling orbital debris, directing space traffic

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.