back to Big Thinking arrow

Data scientists – do you have a future…?

Gary Parilis
Marketing Science & Insight Executive
Hall & Partners The Modellers

LinkedIn Twitter

Since the industrial revolution, technological development has led to hand-wringing about the obsolescence of certain occupations. In the mid-20th century, hundreds of thousands of switchboard operators were employed in the US to manually connect phone calls, and as AT&T moved to automatic switching equipment, this function became obsolete. Similarly, automation has reduced and changed factory jobs in recent decades.

With the rise of AI and machine learning, some people wonder whether analytics professionals are destined to be replaced. They’re not, but the job is certain to change – and to become more important, in a very real way. We’ll see higher stakes for the decisions that machines can’t make.

Advancements in data science have made it possible to mine vast amounts of data and automate the optimization of predictive models as never before. We now can generate and run huge numbers of functions simultaneously, and use complex algorithms to efficiently compare results, select optimal models, and integrate output into subsequent analyses. The massive increase in the amount of data being generated and stored actually necessitates this increased efficiency.

As technology advances and the cost of prediction declines, the value of human activities in performing those mundane calculations and basic programming will decline. It also increases the number of potential predictions as well as making the methodological decisions potentially more consequential – requiring greater human judgment to choose from among the many possible solutions, and to generally steer the prediction ship.

High-quality human judgment is always going to be valued

New sources of data are constantly being generated, and with each new type of data comes a whole new set of knowledge to learn. Simply understanding and preparing each dataset can be a demanding intellectual and programming exercise. Combining novel data sources requires investigation and a great deal of judgment and creativity, as well as tolerance for ambiguity – all of which place a premium on human analytic skill. Beyond the many complex steps in simply understanding and preparing each dataset, how can these datasets be connected? What unit of analysis lets us use all our sources? If connections are possible at an individual observation level, what kinds of assumptions, decisions and transformations are necessary in order to do that? And if not, what kinds of modelling or aggregations would facilitate linkages of some sort? Machines are nowhere close to being able to make those kinds of complex judgments.

As complicated machine learning algorithms become more common and highly specialised, and as the quantity and diversity of data grows, the opportunities for analysis increase, especially novel analysis requiring creative thinking. It’s already not enough to choose from among a set of textbook methods, or to fall back on how you handled the last situation. Most new problems are bespoke. And, even once the algorithms have been run, they don’t provide ‘answers’; they really just provide more data to interrogate.

In a world with more statistical programmers and automated algorithms, we’ll generate far more output than ever before. We’ll need leaders to translate problems into analytic plans, to guide and interpret the results, to plan successive rounds of modeling, and finally to communicate the results to the stakeholders.

With so much data and so many automated analyses, more now than ever before, many marketing and marketing science leaders aren’t relying on traditional data analysts to get technical pieces moving (i.e., coding and programming). However, strategic expertise is now essential to ascertain which data sets to use, and which problems to tackle. High-quality human judgment is always going to be valued. The specifics of analytics jobs will change dramatically, as will the specifics of the skills required and the tools needing to be mastered.

We’re very excited about the changes our industry is going through, and the opportunities this presents for us to find new ways for brands to minimize uncertainty in an uncertain world.


Are you distracted by algorithms?


FMCG: strategic pretesting built around Frame


How can businesses predict the unpredictable?