Over the past two decades, the meaning of data analytics (DA) has undergone significant changes and it continues to expand at present as the field evolves. The changes have been driven by business needs, adoption strategies, required skills and capabilities, and technological advancements. As DA has a broad impact across various sectors, it is essential to comprehend its evolution and anticipate its future developments. The purpose of this article is to assist analytics and business experts in preparing for the future.

Data Analytics 1.0

Skillset and Toolset

The Business Intelligence Era:  The era focused on centralising all enterprise data in a data warehouse such as Teradata, SQL Server, and others. Analysts would leverage ETL and BI tools such as Excel, SQL, DataStage, COGNOS, Microstrategy, and more for business reporting. A few smart individuals would leverage advanced SQL and excel to perform more complex analysis, such as what-if-analysis, correlations, hypothesis tests, basic optimisation operations, and more while automation was restricted to VBA-based macros.

Mindset

Proof of Concept: A handful of analytics professionals from statistics and operations research backgrounds were conducting short-sprint experiments to make a business case for applying advanced math to solve traditional problem statements.

Data Analytics 2.0

Skillset and Toolset

Analytics and Storytelling Era: The effort of statisticians, operations researchers, and savvy business professionals and analysts generated a lot of excitement around advanced mathematics, especially the potential ROI that businesses could generate from the same. Words like Multivariate Regression, Logistic Regression, K-Means Clustering, Linear Programming, Markov Chains, Monte Carlo Simulations, and others became popular in the corporate world. SQL, SPSS, SAS, and R were the dominant programming tools in the hands of these analytics professionals—now being called data scientists. Simultaneously, more modern business intelligence tools like Tableau, Qlik, Spotfire, and others emerged and enabled more effective and efficient data storytelling.

Mindset

Pilot: With the POCs starting to show promise, corporations graduated from an “Analysis” to an “Analytics” mindset and started investing in pilots across high ROI areas—sales, marketing, supply chain, and more. These pilots were limited-scope projects—geography, a product family, a customer segment, a store cluster, and such—to show tangible ROI to the executives.

Data Analytics 3.0

Skillset and Toolset

The Big Data and Machine Learning Era: With the success of these pilots, companies started investing more in analytics capabilities. It quickly became apparent that to find deeper insights, companies will have to leverage both internal and external data. This would mean tapping into structured as well as unstructured data. Hadoop, with its distributed processing of large datasets capabilities, came to the rescue. In addition, by this time, data scientists had started playing with the new shiny toy, machine learning, and leveraging R and Python libraries to generate faster, more accurate, and actionable insights. Data engineers, who would help the data scientists with data-related tasks, thus entered the scene.

Mindset

Project: Business started the full-scope implementation of analytics use cases. These use cases helped organisations improve market share, topline and margins, customer, and employee experience and mitigate risks.

Data Analytics 4.0

Skillset and Toolset

The Cloud, AI, and Automation Era: With projects delivering ROI, demand for scaling up throughanalytics became a necessity. Cloud and AI made this possible. With a keen focus on digital transformation, businesses across industries started migrating to cloud, provisioning data lakes, deploying RPA bots for automating standard manual activities, and leveraging AI-powered algorithms—neural networks and variants—for solving more complex problem statements. A lot of self-serve platforms made life easy for data scientists and business professionals.

Mindset

Program: To institutionalise analytics, organisations started setting up COEs/in-house teams. The goal for companies was now to be more data-driven. They had realised that data and analytics-powered insights were not mere table stakes but in fact key to creating competitive advantage.

Data Analytics 5.0

Skillset and Toolset

The Next Frontier: There is no dearth of innovation in this space. There are new frameworks, tools, and technologies being unveiled nearly every month across the analytics value chain—data engineering, data science, and visualisation. Companies are already experimenting with the next generation technologies. New data management frameworks such as data mesh, data fabrics, knowledge graphs, and others along with intelligent process automation for unstructured data and documents, reinforcement learning for personalisation, and large language models for generative AI are a few examples.

Mindset

Product: Several organisations are beginning to adopt a product orientation for analytics and AI. This is a new course that some organisations have taken primarily because a lot of companies failed in their digital transformation journeys or analytics initiatives primarily due to change management and technological challenges. Since products are designed keeping the end user in mind and developed keeping technology integration and implementation in consideration, the hope is that this approach will increase adoption and, hence, ROI. But only time will tell.

Now that we have understood the evolution of the field, the million-dollar question is: What does the next phase look like? How will it impact businesses and society in general?

In my opinion, there will be considerable disorder and disarray in the near future concerning the emerging fields of data and analytics. The proliferation of platforms such as ChatGPT or Bard has generated a lot of buzz. While some users are enthusiastic about the potential benefits of generative AI and its extensive use in business and daily life, others have raised concerns regarding the accuracy, ethics, and related issues. Nevertheless, in the long run, productivity gains through AI and automation will stimulate innovation, drive business progress, and provide mankind with the work-life balance they have always desired. To those who are doubtful about AI’s impact on employment, I would like to mention the phrase that has been circulating widely on the internet: “AI will not take your job. People who know how to use it will”. 

Instead of fearing AI, it may be time that we embrace its advent wholeheartedly.

This article is written by a member of the AIM Leaders Council. AIM Leaders Council is an invitation-only forum of senior executives in the Data Science and Analytics industry. To check if you are eligible for a membership, please fill out the form here.