“Big data is the Enlightenment for the digital age”

1 February 2016

You say big data is more than a technology or tool. How do you define it?

“Humans have used data throughout history; that’s how we make sense of the world. But collecting and analysing data has always been costly and time consuming, so we tried to collect as little as possible and then squeeze the meaning out of it to answer our questions. By using vastly greater quantities of data, big data gives us insights into how the world works that we couldn’t have in the past. This is why I say it is more than a technology or a new technical tool; that it’s a new perspective on reality.”

Big data allows us to move beyond samples and educated guesses?

“With the cost of collecting data and the time needed to analyse it plummeting, the really interesting question we are now facing is whether we need to rethink how we deal with data? We are no longer living in a small data world and we no longer have to use the processes and institutions that were designed for a world of data poverty. We need to rethink the way we approach discovery, understanding and how we make sense of our world.”

In your book about big data, you talk about shifting from asking why to asking what – from causality to correlation. Should we not worry about why things happen anymore?

“Discarding our human quest for causality would be preposterous, utterly stupid. What we are saying [Mayer-Schönberger and co-author Kenneth Cukier] is that we now have to understand that asking ‘what’ can be an important first step to answering ‘why’. We can now use correlational analysis to discover interesting patterns and things that point us to relationships that are worth investigating, rather than banking on our human ingenuity to come up with the right questions and quests to investigate in causal terms. I’m not against causality at all, but often when we think we know the exact cause of things, we actually don’t. Knowing ‘what’ as a pre-stage to knowing ‘why’ may be a very pragmatic step forward, especially for very complex situations. It is a paradigm shift that continues the quest for enlightenment.”

You see the advent of the big data era as being as significant as the Enlightenment?

“Yes. It’s as about as important as the Enlightenment was for reason and rationality. We are continuing on this quest and big data is the version of the Enlightenment for the digital age.”

What are the big data opportunities? Where is it driving change the most?

“Big data helps in decision-making. All decision-making. As humans, we base decisions on prejudices and stereotypes and beliefs and ideology and religion. But with big data we can base our decisions on empirics, on facts – on data and the analysis of data. So the hope is that big data can boost our ability to make better decisions. This applies to many, many sectors. Does your doctor really know that taking an aspirin is the best way forward if you have a cold? Of course not. It’s a hunch and little more. Are teachers really selecting the most appropriate books for a particular class? Of course not. They often just pick the ones they know best. They make sub-optimal decisions. People do it all the time.”

And big data will help us make optimal decisions?

“Yes. Think about credit scoring. Credit scoring is a way of assigning and giving credit to those who can pay it back. But credit scoring is utterly stupid in many ways, because it relies on a handful of factors that, actually, do not predict an individual’s ability to repay a loan or mortgage very well. A big data approach, as some start-ups in the United States are pushing, is eminently more precise, eminently more accurate and therefore eminently more efficient.”

If big data is going to affect all decision-making, then the impact on society, on work, on so many things we take for granted is going to be huge.

“There’s a book called The Second Machine Age that predicts that up to 50% of white collar work will disappear in the next 20 years. I haven’t done the math, but we see it happening already. Look at how many people still work for the post office, in travel agencies, in photo labs, in music and record stores. Think of the closing of bank branches left, right and centre. The world is changing. If we switch to autonomous trucks [in which sensors and big data processing drive the vehicle], which we will very soon, then ‘truck driver’ is going to go the way of ‘coachman’. And the same is true for taxi drivers. But what is really different in the big data age is that it is not just the manual jobs that are going to go away, as happened with automation, but white collar jobs that were supposed to be immune, like banking clerks and doctors, are also going to disappear.”

Much of the middle class, on which society as we know it was built, is going to be wiped out?

“Yes, and it’s a challenge to know what people should do. A response would be to try to identify your unique selling proposition; your differentiator versus a machine. Is it your creativity or originality? Is it your irrationality? Is it your humanness?”

But not everybody can be a creative, off-the-wall original thinker. Does big data create new jobs? And enough of them?

“Of course big data creates new jobs, but enough – the jury is still out on that. But creativity comes in very different flavours. If I go and have a massage, the masseur or masseuse can use their skill to make my muscles relax. That’s not a cookie-cutter kind of thing. That’s a creative, personalised and original way of kneading my body. A caregiver can be creative and personal in how they care for a particular person. I think we will need to broaden our sense of what creativity means. But it is true that jobs will disappear. At the same time new jobs are appearing because big data offers opportunities for start-ups.”

And it’s easier to be a start-up because you no longer need to be a big, physical organisation?

“Indeed, and I think that’s the wonderful message to start-ups and entrepreneurs. A hundred years ago you needed to build a factory if you wanted to be a start-up. Ten years ago you needed to build a server farm if you wanted to be an internet start-up. Today you can just go out and get cloud services. The real and only thing you need today is a really good idea.”

Are you optimistic about big data and where it is taking society?

“No, I’m not. I’m not optimistic at all. I am convinced that big data is the next step to take if we want to continue our human path based on reason and rationality. But I’m not at all convinced that we, as a society, will rise to the challenge of putting in place the societal and ethical framework to ensure that big data is used appropriately, ethically, and that the dark sides of big data will be constrained.”

How worried should we be about big data ‘abuse’? that it will be used for making predictions, for example?

“The fundamental point is not so much that big data will be used for predictions, but that we will take predictions and assign individual responsibility based on predictions, whether it’s incarcerating people, but also things like giving or denying insurance to people. Suppose we could, even before somebody takes their driving exam, predict whether that person is going to be a good driver. Are we going to give him the driving licence, let him even take the driving exam, if he is predicted to be a lousy driver? And if we let him take it, who the heck is going to insure him? If we take ignorance out of individual risk propositions and start making individual risk predictions, what does that do to societal institutions, like risk pools? This will force us to rethink a lot of institutional structures and constraints in our society. Is this the end of free will? That’s a possibility. There is a real possibility that we will fall prey to believing more in predictions than anything else, and misinterpreting what they are telling us.”

Should we be trying to minimise the amount of data that is available about us?

“No, I don’t think so. I don’t think a general policy of ignorance is a good way forward. Not collecting the data on illnesses will mean that more people will die. That would push us all the way back to the superstitions and prejudices of the past. But we do need to be careful that data and data analytics are being used ethically – that we may use data for many purposes, but that some uses are prohibited and constrained, such as predicting when a person is going to die.”

What do you think will be the impact of big data on financial services?

“Financial services is a matching industry. It tries to match the supply of and demand for money, whether that’s for mortgages, for credit, for saving. Doing that matching more effectively and more efficiently will be a great boon. If we are able to give mortgages to those who really will be able to pay them back – that is, to assess the risk much better than we do today – the spread between the interest paid and the interest asked can be lower, because we won’t have to build in a risk margin. That means that more people who deserve money will get it, and those that don’t deserve money will not. And on the investor end, people will get greater returns. That’s a good thing. That drives out inefficiencies in the marketplace. So big data will bring more sophistication to what is fundamentally a matching business.”

And do you think these innovations will come from today’s financial services companies?

“Did the horse carriage producers invent cars? Did the producers of gas lamps invent the lightbulb and electricity? Did the record companies come up with MP3? Those that are doing well in existing market spaces rarely see the need to adjust and innovate soon enough to come up with the next idea. There’s no law behind that. History doesn’t mean that the financial industry can’t do it better. But the odds are against it. They are under acute pressure to understand and react to the big data challenge. To see it as not only a challenge to their existence, but also something that may provide them with the biggest opportunity in their existence. There is also the issue of trust. People are more distrustful today and trust can only be established if there is honesty and transparency, and this is something that banks need to take on board. There’s too much fudging and marketing getting talked. It’s time to get honest with your customers.”

That rather neatly brings us to the final question: What would you do if you could be a banker for a day?

“I would put in place in my organisation a disruptive little unit of data analysts and just smart geeks and let them access the data – not giving them a question to answer, but saying ‘come up with interesting questions’. Then, six months later, I would sit down with them and ask them what they had found. And based on what they found, I would revamp the products and the services that I offer, and think strategically and creatively about what new revenue streams I could create.”