Can Data Replace Intuition?

I’ve recently been listening to an audiobook called “Blitzscaling” by LinkedIn founder and Paypal executive Reid Hoffman. (Stay tuned for my review of this book later on.) One of the chapters of the book covers “key transitions” you need to make as a company as you start to scale. One of the key transitions discussed that really stood out to me was the transition he described as “from pirate to navy”.

To be less poetic about it, this basically refers to the transition you make from being chaotic and “gung ho” about decisions (a necessity when you’re just starting out) to creating a more organized decision making process. As a company grows, so does its organizational chart and the spread of accountability inevitably becomes wider. That means more people need to have a say or at least be aware of decisions made in order to do their jobs well. In my previous job at Kyocera, we employed the ringi system which has become an important part of Japanese management culture. This allowed us to efficiently get accountability and acknowledgment from those who had a stake in decisions.

At this particular stage of growth of a company, what could previously be decided intuitively, either by founders or people who’ve been around from the start, begins to require more reason as people who are new begin to outnumber those who have been there for a while. As decision making spreads out, it now become crucial to have a shared context that everyone can base their decisions on.

Why is data important?

First off, at this point in time, I think there’s so much data available for use that it should almost be criminal to ignore it and not use it to make better decisions. (Conversely, it is criminal to use that same data unscrupulously, so a quick reminder to be extra careful with that.)

That said, I already mentioned that people who are new to a company often have little context on why and how things are the way they are. This makes data extra crucial to onboarding people who might have the right skills and simply need the right context to make magic.

Additionally, as the business grows, it’s just not possible for one person to know and detect every problem and opportunity. Collecting data and reporting key metrics in a dashboard solves this problem. Data can often indicate a potential problem or a potential opportunity looming on the horizon. Note an emphasis on “indicate” and “potential”. Data should not be the end all or be all. Rather, it should be interpreted in the bigger context before making any conclusions. In the end, it might mean nothing at all.

Does it replace intuition?

Absolutely not. Intuition may seem like hogwash to a lot of the “data-driven” crowd, but I strongly believe that intuition in a business setting is a manifestation of a company’s culture and the collective experiences and mission of those who have grown with it. I say that as a good thing. Throwing intuition out the window in favor of purely data-driven decisions would be tantamount to tossing out a company’s very foundation of existence. I suppose that’s why a lot of companies start to suck as they grow.

Intuitive decisions are definitely not a bad thing. They’re straightforward, quick, and give due respect to those who’ve been through the best and worst times with the company. Are they reasonable? That’s something that you’ll need to find out. This is where data comes in. Data helps strengthen the case for an intuitive decision and can be an opportunity to open discussion on why the decision might be wrong.

A timely example

We’re in the midst of the coronavirus pandemic at the moment and now more than ever, we’d love to have the data to be able to make the right day-to-day decisions. Of course, there’s a lot of that available and floating around online and in social media. Disregarding the fake and unverified bits, some of that data does give us hope. Unfortunately, in a lot of cases, it could just be false hope.

A specific example – in my city of Cebu, it was (until a few days ago) consistently announced that we’ve had zero confirmed cases. That seems like really great news on the surface. Intuition might tell you, however, that it seems highly unlikely, given we have one of the busiest airports and seaports in the country and we’ve drawn in a lot of tourists even until the early stages of the outbreak. Likewise, it’s well known that our country is a bit slow on adopting technology, and that includes medical technology to test for new diseases.

Digging deeper into the data validates that intuition. The underwhelming number of people that have been tested and the time it takes for tests to return would tell you that it’s likely a lot of people who might have had the virus have slipped past the tests and, worse, have very likely been around other people in that time. That’s a cause of concern and should tell us that while some numbers show it’s safe to be outside, the reality is that we shouldn’t.

This is an example of how data alone can be dangerous. Intuition is not at all credible on its own, but it can definitely tell you when something doesn’t seem right, even when the numbers say everything’s fine.

Data and intuition

In the end, while there is often tension between them, data and intuition are not necessarily on opposite ends of a spectrum. They both exist for a reason – one to represent culture, tradition, and subjective experience, and the other to represent objective reality and the promises and threats of the future.

They both serve to validate each other, and it’s important for managers to maximize the value of both. The hard facts of data validate what might be a misconception or an invalid bias amidst a changing landscape. Intuition likewise validates what might be skewed, inaccurate, or misrepresented data. In the end, decision making should always be multidimensional and never dictated by a single factor. Like most things in management, balancing both factors can bring about the most success.