Let’s put it out straight: The role of data scientists is not fading away anytime soon. Instead, it will continue to evolve, especially with the emergence of new generative AI tools. 

“These tools (LLMs) are beneficial as they increase efficiency and can help get started on a problem when stuck. However, those who claim that these will replace data scientists or data engineering jobs are not fully considering the implications of such a statement,” said Siddhartha Sharan, senior data and applied scientist at Microsoft, in a recent podcast.

Supporting this perspective is AI expert Vin Vashishta said, “Generative AI tools work well enough to augment people, but after a year of working with them, I haven’t seen anything that’s a replacement for people. We’re still in the proof-of-concept phase for most tools, and there are bugs to work out before we talk about AI taking people’s jobs”. 

Boosting Data Scientists with Generative AI

Earlier, data scientists spent hours on tedious tasks like data cleaning and formatting. Generative AI can automate these mundane activities, freeing up data scientists’ time for more complex problems. 

“We spend a lot of time explaining the same things or answering the same questions. As the business scales, that work scales too, and those repetitive tasks add significant overhead. Small Generative AI models make automating those use cases very simple. Offloading simple tasks free people’s time to take on more complex work,” said Vashishta.

With generative AI, data scientists can now use algorithms to generate synthetic data that closely mimic real-world scenarios. This accelerates the data preparation phase, allowing professionals to focus more on the analysis and interpretation of results. Interestingly, Gartner predicts 60% of data for AI will be synthetic to simulate reality, future scenarios and de-risk AI, up from just 1% in 2021.

Moreover, generative AI can empower data scientists to explore data in innovative ways. “Data scientists are evolving into ‘solution scientists’, designing creative solutions using the GenAI toolset, or business automation architects, leveraging AI to build automated solutions for business functions,” said Ruban Phukan, co-founder & CEO at GoodGist.com, a skill development and education co-pilot for corporations. 

However, even with these advancements, generative AI can’t replace the unique skills and problem-solving approach of data scientists. Generative AI falls short in understanding specific business challenges, considering human aspects, or independently acquiring the necessary domain knowledge.

For instance, speaking about sentiment analysis, Sharan said, “It is tricky to say whether it will be completely without humans in the loop right now because our approach is that the first three passes are completed by AI, and then after that, there is a human in the loop to validate the results.”

For Aspiring Data Scientists 

According to Sharan, for the upcoming generation of data scientists, it is important that they stay updated with the use cases of generative AI. “Data scientists should read up on and develop an understanding of various models, knowing their strengths and weaknesses. Your project managers or engineers are not expecting you to quote the solutions. Instead, they seek guidance on which model to consider for a specific problem, which one to deploy, and which one would be more effective in the long term,” Sharan said. 

Further, he opined that it’s necessary for data scientists to know the cost of using various language models. Putting all your data in GPT-4 for summarisation, for instance, may be costly and wouldn’t necessarily make sense, he said. 

“How do you effectively reduce the cost while maintaining a big enough margin for your product? That is a key question and that’s where data scientists can help a lot. That is something data scientists need to learn,” he said.

In fact, if one reviews the criteria for applying for a data scientist role, one would see that most firms have updated the requirements. For example, the job description for a data scientist at HP demands, “As a data scientist with a focus on generative AI, you will work on multiple engagements across HP involving large language models and other new generative AI capabilities.” 

Likewise, AWS expects its senior data scientist to “work across customer engagement to understand what adoption patterns for generative AI are working”.

Similarly, IBM’s job description says, “Stay up to date with the latest trends and advancements in AI, foundation models, and large language models. Evaluate emerging technologies, tools, and frameworks to assess their potential impact on solution design and implementation.” 
Recently IBM in collaboration with Coursera launched a course titled ‘Generative AI for Data Scientists Specialization.’ allowing professionals to upskill themselves.