Becoming data-driven: Why is it so hard?

Understanding human psyche to answer one of the most basic questions that most organizations are grappling with today. How to adopt a truly data-driven decision-making process in a world that is increasingly anchored in elevating the human experience.

In today's world, companies that can leverage data to make better decisions often come out on top. However, becoming truly data-driven is easier said than done, especially for B2B companies in mature industries that are not natively digital. According to a study by Gartner, "more than 80% of data and analytics projects fail to deliver business value due to issues with data quality, data integration, and data governance". What makes this so hard?

At the core of it, the two notions – one of the organization’s need to be truly data-driven, and the other of how humans are wired (to learn, behave, and react to stimuli) that defines any organization’s culture - are at loggerheads with each other. This makes it hard, very hard, and sometimes an impossible change management nightmare.

There are three basic traits of human psyche that are the core of this clash:

1.    We think in stories and anecdotes … not in rows and columns

We (most of us) are wired to understand and remember stories. We are more likely to remember a story we heard than a list of facts or statistics. This is because stories activate our emotions and engage our attention, making it easier for us to remember and recall information. In contrast, data in any organization is usually stored, managed, extracted, transformed, and presented in rows, columns, tables, and indices. In a form that is almost cold, abstract, and disconnected from the real world, which is not how we understand the world around us. It requires a different kind of thinking and processing to solve challenges for a data-driven organization than the way we naturally process information. This phenomenon is well-documented in academic research.

 2.    We usually have a “PointsSolution” centric view of the world …whereas organizational data challenges are often extensive

Most of us are wired and incentivized to solve point problems. We tend to focus on specific issues or situations and use our experience and knowledge to find a solution. Data, on the other hand, is supposed to be big, encompassing, sometimes abstract, and most times cutting across functions and departments. While each function / department may be doing a phenomenal job in solving their own point problems, it’s inherently challenging for us to take a broader / more abstract view of solving the enterprise-wide data issues. As I watched the recent Academy Award winning movie “Everything Everywhere…” one point that struck me from the movie was the basic lesson about human psychology. If you try to focus on everything, everywhere, all at once what you get is a (spoiler alert) a big, black, unappealing, unappetizing black-holish bagel. As humans, we are built to focus on one or a few things at a time, while data is everything, everywhere, and (sometimes) all at once.

Also, most of the data problems these days stem not from the scarcity of data but from abundance. There are too many systems, processes, people, bots, cookies etc. each creating their own versions of truth of the different data points and solving their specific point problems. Most often, they do this quite effectively and efficiently. But connecting those data points in a comprehensive, meaningful manner is fundamentally important to being data-driven but inherently in contrast with how we operate.

 3.    We crave instant gratification … whereas solving data challenges is rarely ever quick or easy

“Let’s worry about the short-term, in the long run we are all dead anyway.”

Without sparking a debate on the dynamics of macro-economics, I believe the statement here does capture the sentiment of the human psyche today. In a world where our attention span is incrementally deteriorating, where businesses (and thereby mid-to-senior management positions) are rewarded and punished on a quarterly basis by the market, we are focused on quick-wins and fast track solutions. However, when it comes to data, the reality is often quite different. Addressing enterprise-wide data issues requires a sustained effort over time. It is neither quick, nor easy. And moreover, most often it’s not even an end in itself. Rather, it is a means to an end – the end being better decision-making and improved business outcomes. So, when faced with a choice between investing in long-term efforts to improve data quality, governance, and infrastructure, or pursuing short-term gains through quick and dirty analytics, many business leaders end up choosing the latter.

So, what can be done to solve these?

In my experience, solving these challenges requires a three-pronged change management approach.

1.    Establish solving data issues as a key organizational priority:

Change needs to be driven from the top. Have the C-suite fully endorse and drive the program. Establish clear objectives and ring-fence the program to have funding / support over an adequate time horizon.

2.    Identify and empower key change agents:

Identify key mid/senior management stakeholders who understand the business from across multiple functions and empower them to challenge status quo, drive change across departments, and break organizational silos.

3.    Invest in data literacy across the organization:

Provide training and support across the organization to develop data skills. Not just the technical data skills around management, extraction, transformation etc. but also visualization and storytelling based on data. This will help break down the personal barriers to working with data.

In conclusion, working with data is inherently challenging for us (as humans), but it is non-negotiable for organizations today if they want to succeed in the current environment. By developing the necessary skills, and driving the change in a meaningful, sustained manner, organizations can really develop a data-driven culture.

Thoughts and feedback welcome!

____________________________________________________________________________________

1 A report by McKinsey & Company found that companies that make extensive use of data and analytics were more likely to outperform their competitors in terms of profitability and productivity (Bughin et al., 2018)

2  Natively digital B2C companies (even in mature industries) pretty much carried the torch in paving the path for truly data-driven business outcomes, but those are not the focus of the discussion today.

3 Gartner. "Gartner Survey Reveals 84% of Organizations Struggle to Scale AI." Gartner Newsroom, 22 February 2021

4 A study published in the Journal of Experimental Psychology found that people were better at remembering and making decisions based on stories than on lists of facts or statistics (Begg et al., 1992); "Neuroscience research shows that stories stimulate the brain, helping us to better understand and remember complex ideas. When data is presented in the form of stories, it becomes more meaningful and memorable to audiences." (source: Harvard Business Review); "Humans are hard-wired to process information in the form of narratives. We think in stories, we learn from stories, and we remember stories." (source: Forbes)