At the virtual Data Engineering Summit 2022 (DES22), held on May 30, 2022, global practice head of AI/ML at USEReady Amit Phatak and co-founder and CEO of USEReady Uday Hegde alongside the founder-director of IIIT-Bangalore S Sadagopan talked about the evolution of data and analytics, and what organisations and people need to do to stay relevant and participate in these growing areas, like business intelligence (BI), artificial intelligence (AI), big data, and cloud computing.
In 2013, Harvard Business Review published an article on Analytics 3.0, authored by Thomas Davenport. Here, he talked about how the world of data and analytics has evolved from the era of business intelligence (BI) to the era of big data and then finally to the era of data-enriched offerings. However, to get into Analytics 3.0, there is a need for foundational blocks to make things happen. This includes BI, AI, data platforms and finally, the cloud.
“What are your thoughts on this evolution of analytics from 1.0 to 2.0, and now 3.0? Where do you see data and analytics headed in the next decade,” asked Phatak, opening the panel discussion.
To this, Sadagopan said that previously, human beings used to collect the data for days, weeks and months. Citing the electricity board, he said the humans used to collect data (meter reading). “Today, of course, things are much different; systems capture the data,” he added. However, he said we are at the early stage of machine learning and big data.
“In India, we have QR code readers, surveillance cameras, etc., everywhere,” said Sadagopan, stating that the big data challenges are reasonably met. On the flip side, the analytics experts are not getting enough time to form the models. He suggested that one way of looking at it is through historical and statistical analysis. “We talk of composing a null hypothesis; nobody has time,” he added.
Further, he said today, machines are doing that. “This is what we call pattern recognition, where machines recognise the patterns, but you occasionally find some things that are not going well,” said Sadagopan, saying that a lot of garbage is generated.
Data analytics wave is coming in a big way
Sadagopan believes that change is happening. “I certainly feel that a lot more action will happen at the edge,” he added, saying that in the near future, you will see analytics fast enough to make decisions – i.e., without human intervention. He said that personalised education would be possible for millions of students in every subject.
“What excites me about analytics tomorrow is we will generate insights, which are far more useful than even analytics,” said Sadagopan. He said that the team managing a pandemic or cyclone would get strategies suggested by systems as things unfold, using models, data and edge power.
Further, he said strategies would have to be different as things progress. “Even the strategies will be thrown out by machine, and in some sense, machines will tend to human beings,” said Sadagopan, “Today, unfortunately, the human beings are tending to the machines.”
Quoting Satya Nadella, he said that the system would actually plan your daily schedule for you and give you an option for you to choose. “For us to develop insights, one would need domain expertise,” said Sadagopan, saying that data analytics will dominate a decade across sectors including BFSI, retail, education, healthcare, transportation, etc. He said that analytics would help us build a better planet, addressing larger issues like climate change, better health, and others.
What about the demand?
Continuing the panel discussion, Amit questioned the demand and trends in the space. Answering the same, Hegde said that most of the solutions that we have today are built on four foundational components of business intelligence, big data, machine learning and cloud. He said in the last decade, the role of a CEO has gained tremendous prominence within an enterprise.
Further, he said that the chief data officers (CDOs) are the custodians of enterprise data assets and are responsible for driving the digital transformation within the enterprise. Hegde said that CDOs work with various lines of business, including in the context of data, as they embark on programmes like data monetisation, data literacy, community development, data automation, data engineering initiatives, etc. “We are actually seeing the emergence of a business analytics team,” he added, citing various initiatives within his firm USEReady.
“Many of our customers are interested in migrating from these legacy platforms to more modern platforms like Tableau, PowerBI, and Snowflakes,” he added, saying that they have identified key technology companies that bring vendors or products that are geared towards automation and have partnered with them to driver end-to-end CI/CD for data process automation.
He said they have also implemented data monetisation with embedded analytics within several Fortune 500 enterprise customers. “As you can see, the theme is very consistent, very much aligned to achieve those offerings, initiatives within an enterprise, and continuing along in the context of business analytics,” said Hegde, giving examples of the sales team and HR team on how they are using analytics and machine learning tools.
“Lastly, harmonising data science and data engineering has become a huge paradigm in an enterprise context. Some programmes are geared towards enabling citizen data scientists and data scientists alike,” he added.
The way forward, Creating a data-driven mindset
“Achieving sustainable, competitive advantage from data analytics is a complex endeavour and demands a lot of commitment from the organisation,” said Phatak, citing Gartner and VentureBeat reports, where only 20 per cent of data and analytics solutions deliver business outcomes, and 87 per cent of data and analytics projects never make it to production. “So, how can organisations get value from data and analytics?” he asked, “Also, as practitioners, what areas do we, as a data and analytics community, need to build skills?”
Addressing the pains of professionals trying to build a career as a data engineer, Sadagopan said that individuals need to focus on skills and knowledge. “You need the tools, but tools alone are not sufficient,” Sadagopan added, saying that the candidates should also have basic mathematical and statistical understanding, along with the knowledge of software tools and coding skills.
“Ultimately, the insights are important! How big your model is, how much computing power you are using, and how big a computer you have are also important. But, the model’s usefulness is far more important than how good your tools are,” said Sadagopan.
Further, Sadagopan emphasised the importance of planning. Recalling his advice to students, he said that you should plan before writing a programme, think about all aspects of the model, and not get into tools early on. Lastly, he suggested budding professionals have a long term vision and not just think about today and tomorrow.
“Five-ten years from now, many of them will be done magically by machines; you will not be needed,” said Sadagopan, stating that the data science professionals should not forget that they are citizens and part of this planet and creating a better planet is far more important.
Sadagopan said that organisations should start measuring everything of value for them and use the data rigorously and transparently. “That is an important part of the data-driven mindset,” he added. Further, he suggested that the data-driven mindset must be embedded into your organisational culture.
Adding to this, Hedge said that in the journey of creating a data-driven mindset, every professional must understand, ultimately identifying specific points in the decision making that can be improved by embedding analytics to make decisions better. He further shared various initiatives and boot camps that USEReady has for its employees.
REGISTER HERE TO ACCESS THE CONTENT
