How is GenAI Proving to be a Useful Tool for Data Engineers?

Generative AI has brought about revolutionary changes across numerous domains. How has it influenced data engineering? 

Data Engineering remains pivotal in enabling streamlined data processing, storage, and analysis, which is crucial for informed decision-making.

Prasenjit Ghosh, vice president at Genpact, believes data engineering plays a pivotal role in the transformation of businesses through the effective utilisation of data assets. “Even though they don’t get the recognition, they play a foundational role,” he said while speaking at the Data Engineering Summit (DES) 2024, hosted by AIM Media House in Bengaluru.

In his presentation titled 'GenAI for Data Engineering', Ghosh outlined the three key stages of the data engineering lifecycle- data input, data processing, and data output. 

Various challenges arise across these stages, including data input mismatch, data relationship management, data coding, and data catalogue enrichment. 

Ghosh believes large language models (LLMs) are proving to be important tools for data engineers in overcoming these challenges.

By leveraging these models, Genpact's data engineers are automating and enhancing various tasks, leading to increased efficiency, productivity, and innovation.

Unstructured to Structured Data

Ghosh believes one of the primary use cases of Generative AI for data engineers has been the ability to turn unstructured data into structured data.

LLMs can autonomously analyse unstructured data sources like text documents, images, and audio files. They perform tasks such as text extraction, entity recognition, sentiment analysis, and categorization, effectively organizing the data into structured formats. 

This transformation enables data engineers to convert raw unstructured data into structured datasets, facilitating easier analysis and insights derivation. Ultimately, GenAI streamlines the data engineering process, empowering organisations to extract hidden value within their unstructured data assets.

LLMs could help data engineers in data design, data coding ( copilot, code moderniser), data testing and data analysis.

Increasing Generative AI impact

Ghosh also highlighted that generative AI is going to be transformative as the technology matures. “Now, a very interesting phenomenon is going to happen in the next one to two years. Hardly anybody will go for creating another LLM, right, big players will come, but all the innovations will happen on the edge.

While speaking, Ghosh said in a light-hearted manner that the developments happening in the generative AI space are so rapid that the predictions everyone is making now might not be relevant at all in a year’s time.

Moreover, Ghosh reminisces about a discussion he had with Vishal Sikka, the former CEO of Infosys. At the time, Ghosh, who was working at Infosys, was introduced to OpenAI by Sikka, a close associate of Sam Altman, the CEO of OpenAI.

Interestingly, during Sikka’s tenure at Infosys, the IT giant, along with Elon Musk, AWS, and YC Research, among others, joined hands to make the USD 1 billion donation to OpenAI.

At the end of his end of the session, Ghosh showed a video to the audience, which was developed by prompting an AI text-to-video model.