Recently, Analytics India Magazine (AIM) concluded this year’s Data Engineering Submit (DES) powered by Tredence and in association with Infocepts and Tiger Analytics. 

DES 2023, which is India’s one and only summit dedicated to Data Engineering has come to a successful conclusion, bringing together some of the most influential thought leaders, experts, and professionals in the field of data engineering. 

The summit was a unique opportunity for attendees to learn about the latest trends and best practices in data engineering, network with their peers, and gain valuable insights from keynote speakers and panel discussions. 

“We just created the world’s biggest conference focussed purely on data engineering. I was pleasantly surprised how much the field has evolved over years and so much buzz around data engineering in India,” Bhasker Gupta, founder and CEO at AIM, said. 

The success of the Data Engineering Summit 2023 marks a significant milestone in the industry’s continued growth and innovation. Here are some of the interesting topics covered in DES 2023.

Data Engineering in an AI world 

With the increasing volume and complexity of data, organisations are leveraging AI to enhance their data engineering capabilities, automate processes, and drive insights from their data.

Sunil Krishnareddy, VP & Head of Data Engineering services at Genpact, in his talk ‘Data Engineering in the new age – AI is here, where is your data?’ explores how the domain of data engineering is rapidly changing with the advent of popular AI models such as ChatGPT and Midjourney. 

Similarly, Gopan Vijayan Nair, Head of Enterprise Business at NVIDIA, discusses in his talk ‘The Synergy of Data Engineering and Generative AI in Revolutionising Enterprises’ the potential for the integration of data engineering and Generative AI to create powerful solutions that can improve decision-making, automate processes, and enhance customer experiences in the enterprise space.

Continuing on the same line, Dmitry Ustalov, Head of Ecosystem Development at Toloka, in his talk ‘Handling Noise and Subjectivity in Labelled Data’ also discussed how to build, use, and secure representative datasets for AI problems, taking a special attention to crowdsourced data and data obtained from in-house annotation teams

Data Struggle Galores  

Data challenges still haunt enterprises. Some of the challenges highlighted at the event include the changing regulatory landscape, alongside managing data on the cloud.

Kunal Mehta, Senior Manager of Data CoE, and Deepak Kumar, Senior Director of Data Engineering at Publicis Sapient, in their talk ‘Engineering practices for building data resilience in an increasingly cookieless world’ discussed the growing significance of data for businesses, while also acknowledging the challenges of collecting data due to evolving data privacy regulations.

“For data analysts, the challenge is not the unavailability of data, but to trust their data,” Kumar said.

Ops is the New Oil 

At this year’s Data Engineering Summit, the focus was much more than data, and more towards operationalising data, alongside changing dynamics of the field on how it is moving towards DevOps, AIOps, etc. 

“A very few of us know; once the data is migrated, once it comes into the systems, and sits for a long time, you tend to forget about it,” said Mousumi Kar, Senior Manager, Data Engineering at Tredence. She believes that by operationalising the usability of your data, organisations will be able to generate value successfully. 

Development of Modern Day Data Stack

Data modernisation aims to enhance data quality, accessibility, and reliability, and to enable organisations to gain valuable insights and make data-driven decisions in real-time.

Jorawar Singh, Regional Head-Business Development at Artha Solutions and Rajiv Maskara, Director – Data Integration Specialist – India at Qlik, in their talk ‘Unlocking the Power of Data using the modern Data and Analytics Platform’ discussed the importance of having a modern data stack for any organisation that wants to leverage the full potential of their data. 

Similarly, Sathish Kumar Thiyagarajan, Chief Technology Officer at OurKadai Technologies, in his talk ‘Modernising Data Access Layer with Compile Time ORM’ explores the evolution of the data access layer in enterprise applications that rely on relational databases over the past two decades.

Data Mesh, Data Fabric & More

Data Mesh has the potential to revolutionise the way organisations approach data management and enable them to become more data-driven and innovative.

In his talk titled ‘The Rise of the Data Mesh Architecture: A Paradigm Shift in Data Engineering’ Amit Kapur, Vice President, Data Science & Engineering at Lowe’s, discusses how organisations are adopting this approach to overcome the challenges of traditional monolithic data architectures and to improve their data management capabilities.

Similarly, Thirumalvalavan Venkatesan, Head of Data and Analytics Practice – India at EPAM Systems, in his talk ‘Data Fabric or Data Mesh: Which approach best suits building a future-proofed data platform?’, discussed how analytics leaders are looking for ways to improve data management efficiency, unlock better business insights, and differentiate their data assets from competitors.

Furthermore, Muthu Govindarajan, Partner – Data Engineering at Tiger Analytics raised an interesting question. He asks, If machines can become intelligent with the advent of the IoT, why not data platforms?

Another interesting topic covered at DES 2023 was by Yashwanth Kumar, Account Manager Enterprise Sales and Shounak Vijay, Sales Engineer APAC at Fivetran where they showcased how Condé Nast, a global mass media company monetised trillions of data points.