Not enough data scientists? Use AI instead.

One of the really hard things about having Big Data is figuring what to do with it. There are obvious questions that can be asked such as "what’s the correlation between demographics and purchasing choices" but when it comes to complex inductive reasoning you need expertise and thus we’ve seen the rise of a new analytics role: Data scientist.

Data scientists were recently the topic of a Harvard Business Review article, Data Scientist: The Sexiest Job of the 21st Century (paywalled) but there’s a problem with this profession: There aren’t enough of them. And no matter how many universities can produce in the next few years there still won't be enough.

The answer is, of course, to set a computer onto the task of analyzing and deriving insights and conclusions. Unfortunately most of the available solutions are complex to use and require that you ask just the right question in a some sort of computer language. Enterra Solutions, a key competitor in the big data analytics market, has a solution that is completely different in that it can automatically mine data exhaustively and intelligently to draw conclusions based on natural language queries.

Enterra Solutions can ingest huge amounts of data and using natural language processing transform it into knowledge using a generalized ontology to discover the meanings of words in context along with the implicit rules and relationships as used by humans.

Then, when a question is asked in what is more or less natural language, the database of knowledge is accessed by Enterra's Hypothesis Engine. The Hypothesis Engine is an artificial intelligence system that applies common sense and and domain-specific ontologies to further structure the knowledge. Next, using Enterra's Rules-Based Inference System it can determine an objective and find the facts to support that objective (backward chaining) as well as using facts to determine objectives (forward chaining) as determined by the knowledge found and its significance. 

Other engines in the system weigh results, formulate database queries, and analyze assets and all of these components pass data back and forth between themselves based on rules and inferences to derive conclusions. 

As an example, a grocery chain might ask “Find what drives an uplift in sales for Cumin food flavorings in PA and map the results.” The already ingested data that now has structure is a passed to the Hypothesis Engine which will produce something like this:

Enterra map of PA

This is a map of Pennsylvania showing the result of that query: Enterra concluded the uplift is due to increased sales of barbecue sauces in conjunction with purchases of beef, chicken, and pork. The resulting map plots the locations and the individual increased lifts in sales and spending for each protein in each store. Armed with this data a marketing group could create individual promotions based on the most popular proteins and sauce for each store.

Enterra’s solution is being used by several major corporations including McCormick & Company, Inc., the 150-year old spice and ingredients company. McCormick has developed Flavorprint, a service they describe as "the food equivalent of Pandora’s Music Genome Project." Flavorprint offers recipes as well as spice and flavoring suggestions based on your culinary preferences and Enterra is a key component of the Big Data analytics that drives the service.

Artificial intelligence driven analytics will have a big impact not only in sales and marketing but also in healthcare (particularly epidemiology), education, farming, and supply chain management. Moreover as platform performance increases due to improved storage and processor performance you’ll see this type of analytics being done in real-time.

The application of artificial intelligence to Big Data analytics is one the hottest areas in data science and it’s ability to make up for the shortfall of human data scientists (which is likely to be a long term problem) means Enterra Solutions has a very rosy future.

Deduce your conclusion below then follow me on, and Facebook.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: 10 new UI features coming to Windows 10