For the past few years, we have seen a high demand for hiring data scientists among organisations. While this demand goes up, it has been noticed that due to the increasing use of big data and the large chunks of structured and unstructured data, the data engineering roles have seen a rise among organisations. A lot of this could be attributed to the rising significance of data engineers who “primarily transform data into pipelines for the data science team” and “lay the foundation in most data science projects.”
At present, LinkedIn is showing more than 29k job opportunities in the role of data engineering. However, according to the Stitch Benchmark Report, organisations still face a significant shortage of data engineering skills and talents in the market. In other words, it can be said that there has been a massive shortage of data engineers among organisations when compared to data scientists.
To get an industry perspective, Analytics India Magazine spoke to a few experts in this field to understand the reasons behind the void and how the industry can address such issues.
Rajiv Kumar, the Managing Director at Microsoft India, believes that considering the pandemic has accelerated the digital transformation globally, companies are looking to adopt virtual ways to reach out to customers, which has exponentially increased the amount of data generated.
“Data is one of the most valuable business assets as companies can get deep insights from these data and thus be more agile and efficient in their execution and operations. As a result, in digital data-driven enterprises, the role of data engineers has become crucial in generating insights to make informed business decisions,” said Kumar. “The demand for data engineers with deep expertise is increasing multi-fold, and we expect to have continued growth in demand in the coming years.”
Increasing Demand For Data Engineers
“To solve any mission-critical, complex real-world problems, you inevitably have to account for all types of data, including addressing the challenges of structured and unstructured data, both on-prem and in the cloud. Hence, for data scientists to work their magic, you need data engineers to prepare the data architecture for ingestion, streaming, storage and continual analysis,” said Brian Rasmussen, Global Vice President – Analytics, Machine Learning and Autonomous Data Warehouse at Oracle.
He firmly believes that the need to do all the work using raw data is growing, especially in the large scale enterprises dealing with complex problems, in the fast-growing information economy and the emerging AI and ML industries. “These trends are resulting in high demand for data engineers,” he added.
Commenting on the current demand, Saurabh Agrawal, Head – Analytics & ML at Lenskart, said, “The key to scaling up analytics and AI is to focus on organising the data well, supporting creating reusable assets. Even Google mission is to organise information and make it accessible to all. While data scientists focus on deriving value from data, data engineers focus on making the ingredients ready.”
Agreeing to that, the founder & chairman of Zaggle, Raj N, stated that data engineering is the fastest-growing vocation in the technology sector today, with a growth of around 50% year-over-year. The demand is huge, and the number of data engineers has doubled in the past year.
He said, “If you aren’t making your moves based on the data you have, you are missing out on a lot of things. It is necessary to design, manage and optimise the flow of data in an organisation.”
Speaking of the demand, Sarita Digumarti, COO & Co-founder, Jigsaw Academy, also added, “Without ready availability of clean data, there is no point in trying to deploy advanced data science algorithms. And that is why the demand for data engineers has increased exponentially over the last five years. As part of our enterprise training programs, we often see that for every program we run on ML or advanced data science, we run a parallel program focused on data engineering.”
“With more and more organisations moving to data-driven business models, the demand for data engineers is only increasing. As data grows, so does the task of building a reliable, scalable infrastructure to manage it. This is where data engineers play a crucial role. Without data engineers, data scientists would be handicapped,” stated Himanshu Varshney, CEO and Co-Founder at HashedIn.
Dearth Of Data Engineers
While asking if there is more of a market need for data engineers or data scientists, Rasmussen replied that the two roles are very interrelated, and there is high demand for both. He said, “A lot of aspirants are looking to train in data science, while at times underestimating the importance of picking up fundamental data engineering skills, which in my opinion is essential to become a sound data professional.” He considers that this trend over time could lead to a relative shortage of quality data engineers, as in the end, both areas of expertise are interdependent and hugely important.”
However, the founder of Zaggle mentioned that there is more dearth of data engineers than data scientists in the market, but the number of job openings for data engineers is quite less compared to the job openings for data scientists.
On this issue, Digumarti said that there is a larger supply gap in data engineering than in data science. It is also true that with increasing data sizes and the deployment of AI and ML models at scale, the tech expertise required to organise and process data and create robust data pipelines has dramatically increased. “So the talent gap is much larger on the engineering side.”
Commenting on this, Varshney added, “While analytics and insights are critical for delivering business value, many organisations lack the required skill and resources to manage and scale their data in a structured, reliable way.”
The Demand In Future
Talking to industry experts, it was established that the demand for data engineering roles would only grow as organisations look to become more and more data-driven. In fact, a key factor for the growing demand for data engineers is the emergence of the cloud as the dominant data platform.
As a matter of fact, Rasmussen confirmed that on-premise specialised roles like database administrator, data analyst, data architect and BI developer would evolve into this one very important role, that of a “modern cloud data engineer.”
“As data scientists unlock more use cases, data engineers will help scale up the impact. There will be 2-3x more demand for data engineers,” added Agrawal.
Adding to Agarwal, Digumarti stated that the demand is expected to steadily increase in two specific ways — directly, with more and more tech positions specifying core data engineering skills including database and data warehousing skills, ML and MLOps, and distributed computing systems.
As per experts, before the data scientists can use their algorithms and models, the data engineers need to organise and classify the raw data into something suitable. “Data scientists are only the tip of the iceberg; the data engineers form the rest of it. Hence the need for data engineers will only increase in the coming years,” concludes Varshney.
