Data science was termed the sexiest job of the 21st century by Harvard, but in a surprising turn of events, data engineering may overtake this status. “The dearth of data engineers will be felt even more in 2022”, noted a survey by AIM on the top AI and Data Science trends for 2022. This is owing to the increase in digital transformation after the pandemic and the explosion of data following it. 

Register for Data Engineering Summit 2022

Tracking the increasing demand for data engineers

Historically, data engineers have only dealt with distributed systems and Java programming, but now they have to leverage AI, ML and BI to manage data. 

Termed the nerve centre of Digital Strategy by Sriram Narasimhan, Head of Data, Analytics and AI at Cognizant, data engineers’ demand can be seen through recent market statistics. 

In 2020, the Dice Tech Job Report stated data engineering to be the fastest-growing job in technology with a predicted 50% year-over-year growth in the number of open positions. Global employment platform, Monster has published two recent trends and employment reports showing that ‘engineer’ was the number one job search over the month of WHAT. Additionally, engineering was among the top sectors likely to expand in 2022 by 57%. 

Further, the annual salary study conducted by AIM Research in June 2021 showed that data engineers commanded a median salary greater than big-data scientists or AI engineers, indicating the growth in importance for the position. The demand for this can be seen with big tech companies like Google, IBM, Cloudera and SAS that have initiated data engineering certification programs to skill and upskill employees. However, they don’t have certifications in data science or AI engineering. “With the market for artificial intelligence and machine learning-powered solutions projected to grow to $1.2B by 2023, it’s important to consider business needs now and in the future. To address the new skills data engineers now need, we updated our Data Engineering on Google Cloud learning path,” Google stated.

Why is data engineering becoming sexier? 

Data is everywhere and in varied forms, and it needs to be perfected to derive actionable insights. As analytics professionals, data engineers are responsible for generating, cleaning, processing, and storing data in a way that makes it ready for analysis. Additionally, they formulate data management architectures for companies to democratise data access and establish efficient pipelines. Essentially, data engineers lay the foundation for data scientists or AI/ML professionals to use the data to derive business solutions. 

According to Mathangi Sri, VP Data Science & Head of Data at Gojek, the data engineering demand can be attributed to the emergence of large scale data. “Every big company today has close to ten million customers and millions of transactions a day. So, engineering is required first to make it accessible for real-time models to perform,” she explained. “Factors like penetration of data, sources of data, absence of a state-of-the-art system to present these and data governance is becoming important. Earlier, systems crashing used to be a good sign, but it is not the case today as it can destroy companies overnight. Other companies are there waiting for the failure to happen.”

The eminent shortage

“There is a huge demand for competent data engineers. This is the right time to immerse in it and grab the opportunity,” said Prasad Srinivasa, Assistant Vice President at Genpact. Clearly, many believe so, given the demand for data engineers has outstripped its supply since 2016, according to Quanthub. However, the shortage of engineers is more severe than that for data scientists, and it only seems to be on the rise. “As of 2021, LinkedIn is showing more than 29K job opportunities in data engineering as organisations still face a significant shortage with not enough data engineering talent in the market,” said Sriram Narasimhan.

Data engineering is a relatively specialised field, as opposed to data science, that is prone to continuous upskilling and generalised positions. With the increasing demand for technical expertise, the talent gap in data engineering only keeps growing. Building multi-disciplinary teams and encouraging data engineering education and upskilling seems to be the key in the future.