Big Data scientists get 100 recruiter emails a day

A lot of people—myself included—have long been talking about how big data changes everything. But big data’s biggest disruption may be in the high-tech labor market. Forget mobile app developers—a front-page story in the Wall Street Journal over the weekend makes it clear that the most in-demand tech skill is now data science.

According to author Elizabeth Dwoskin, these elusive “unicorns,” with the skills to “extract and interpret the explosion of data from Internet clicks, machines and smartphones,” are being fought over with manic intensity. While some have questioned the reality of the data scientist shortage (see How many data scientists does the world really need?), Dwoskin quotes the head of Booz Allen Hamilton’s data science group claiming that “anyone with ‘data science in his or her job title is going to get ‘100 recruiter’ emails a day.” Even more striking, that’s only one part of the frantic search to fill up to 36,000 openings at up to 6,000 companies.

Offering salaries of $200,000 to $300,000 for data scientists with just a couple years of experience, tech recruiters are also going after academics with experience in areas like genome mapping and breast cancer research, dangling the big bucks to get them to help figure out what search terms people use and the impact of tiny changes in online ads.

See also: Data scientists: IT’s new rock stars

The Insight Data Science Fellows Program, in Silicon Valley and New York City, claims to be “your bridge to a career in data science,” offering an “intensive six-week post-doctoral training fellowship bridging the gap between academia and data science.” The programs’ website claims a 100% placement rate (duh) and notes fellows with doctoral backgrounds in astrophysics, biology, statistics, and so on. Academic budget tightening during the recession has helped fuel the trend as well. And Dwoskin quotes one Yelper who complains that “academia is slow and only a few people see your work.”

That’s all good if you’re one of the unicorns, or a tech company that can afford to pay big bucks to hire them. But what’s the effect of this trend on the larger world? If the smartest big data analysis is focusing on helping Task Rabbit do more efficient scheduling, what the prognosis for breast cancer research they’re no longer doing?

Last year, I interviewed IBM’s Grady Booch about the idea that with big data comes big responsibility. Booch was concerned about the misuse of data and its unintended consequences—and said “technology professionals have a responsibility to be cognizant of the possible effects of the data we collect and analyze to raise the awareness of the public and the lawmakers.” We didn’t even discuss the possibility that the best and brightest would be drawn away from the most important work to help.

The long-term solution to these issues no doubt lies in better automation of data analysis so data scientists aren’t needed for routine analyses. That’s a holy grail for many companies, but no matter how successful that becomes, it’s unlikely to temper the need for highly competent data scientists in the age of big data. The elite data scientists will just find harder and harder problems to work on. And we don’t seem to be running out of those.