Listen to this story

This report aims to define the scope of data engineering in India. Data engineers work across various sectors, experience levels, and cities in India. This report delves deeper into different aspects of data engineering. It has defined what is data engineering, who are data engineers, and what benefits enterprises derive from establishing these functions within their organisations.

The study analyses several trends across multiple dimensions to provide a holistic view of the data engineering market. It presents an overview of open jobs, attrition rate, median salary across sectors, cities, and experience levels. Furthermore, market sizing and projection highlight the industry’s scope and opportunities. The key skill section presents an overview of skills required to be a data engineer.

The detailed analysis helps the reader develop a clear overview of the key trends and scope of the data engineering market. This report can be studied by recruiters, industry policymakers, companies, and analytics/data science experts/aspirants.

---

Key Highlights

Market

• The size of the Data Engineering market in India is USD 18.2 billion in 2022. This is predicted to grow at a CAGR of 36.7% in the next five years to reach USD 86.9 billion by 2027.

• Banking/Insurance employed the highest share of data engineers at 37.7% among all non-IT sectors.

• The highest share (30.8%) of data engineers have a work experience of 3-6 years.

Salary

• The median salary offered to data engineers is INR 17.0 lakhs per annum.

• Data engineers in the Internet/eCommerce industry command the highest median salary of INR 28.5 lakhs per annum.

• Data Engineers working in the Delhi NCR region get paid the highest median salary of INR 19.3 lakhs per annum, followed by Bengaluru at INR 19.0 lakhs per annum.

Jobs

• The total number of open jobs for data engineers as of the last week of May 2022 is 36,457.

• There are more than 10,000 jobs in the market for data engineers who can command a salary of INR 10-15 lakhs per annum.

• More than two in five (44.9%) jobs are listed for data engineers with 5 to 10 years of experience.

Attrition

• The overall attrition rate for data engineers is 34.1%.

• The highest attrition rate is recorded for data engineers with 0-2 years of experience, at 41.7%.

• The attrition rate for boutique analytics firms is almost two times (1.8) the attrition rate of IT services, which is the lowest across company types.

Key skills

• More than 9 in 10 data engineers (92.0%) can work in Python.

• In terms of usage of the cloud, more share (39.0%) of professionals can work in Azure than in other technologies

• SQL is the most popular programming language among data engineers with 0-3 years of experience with 100.0% of them being able to work on it.

---

---

Market

Data Engineering Market

The future of data engineering is exciting. Traditionally, companies have focused on analysing and visualising data. Now, most organisations have started to think about better ways to transform, manage, and track their data for better analysis. To refine these processes, data engineering plays a pivotal role. Hence the demand for data engineers is skyrocketing.

AIMResearch estimates that the data engineering market will grow at a CAGR of 36.7% in the next five years, growing from USD 18.2 billion in 2022 to USD 86.9 billion in 2027. The share of data engineering as a percentage of the analytics market will grow from 29.8% in 2022 to 43.2% in 2027.

Currently, the key sectors to contribute to this growth are IT, Internet/eCommerce, and Banking & Insurance. The median salary paid to data engineers is INR 17.0 lakhs per annum. However, in terms of cities, Delhi/NCR tops the list with a median salary at INR 19.3 lakhs per annum, followed by Bangalore at INR 19.0 lakhs per annum. Furthermore, the number of open jobs for data engineers is 1.8 times higher than for data scientists as per Jobs Study 2022 published by AIM Research.

---

Distribution of data engineers by sector

More than half of data engineers in the non-IT sectors are employed in the Banking/Insurance sector (37.7%), followed by Energy/Industrials at 21.8%. The share of employees in Internet/eCommerce, Pharma/Healthcare, and Telecom have similar shares, ranging from 9.2% to 9.5% of the total market. Data engineering employees in the Media & Entertainment sector account for 3.5% of the total. The lowest share of employees is noticed in Travel/Hospitality (0.5%).

Sectors with higher digitisation like Banking/Insurance or Retail & CPG are now hiring a significant number data engineers to streamline data pipelines for better analysis and insights.

---

Share of professionals by years of experience

The highest share (30.8%) of data engineers recruited by organisations fall in the 3-6 years work experience bracket. More than one in two (53.7%) analytics professionals have a work experience of 3 to 10 years as companies show preference in hiring data engineers with a certain level of work experience.

Organisations prefer hiring data engineering experts with a certain level of work experience since the cost of upskilling freshers is higher.

There is also a higher focus on hiring data engineers in strategic or leadership positions as well. Data engineers with more than 12 years of experience make up 14.9% of the total professionals.

---

Share of data engineers as a percentage of total analytics function by company type

In this section, we have highlighted the share of data engineers in comparison to the total analytics professionals across different company types. The highest number of recruitment of data engineers (24.1%) is in boutique firms, whereas it is lowest in domestic firms (1.9%). Boutique analytics firms are more mature in terms of their analytics capabilities and offer end-to-end solutions. Domestic firms are fairly new in adopting data-driven technologies.

The boutique firms provide end-to-end expertise across domains, software, and techniques needed for any analytics project. Hence, the demand for data engineers and data scientists is on the higher side.

The second-highest adoption (14.1%) of data engineering is by IT companies. IT ecosystems enable quick adoption of data engineering due to the availability of infrastructure and talent. At the same time, there is a thinning line between the role of many computer engineers engaged in data management processes and data engineers.

IT firms are followed by Consulting, Captives and Startups with data engineers making up 10.7%, 9.6%, and 3.5% of their analytics functions.

---

---

Salary

Median salary of data engineers by years of experience

The higher demand for the role among the companies and the low supply of data engineers is one of the primary reasons for the high salaries of data engineers in the industry. As per AIMResearch Salary Study 2022, the median salary for data engineers is INR 17.0 lakhs per annum compared to the median salary of INR 16.8 lakhs per annum of all analytics professionals in India.

The median salary of data engineers with less than three years of experience stands at INR 6.7 lakhs per annum. This increases by 5.2 times for data engineers with more than 12 years of experience who command a median salary of INR 35.2 lakhs per annum.

The median salary of data engineers with 3-6 years of experience is 76.7% higher compared to those with 0-3 years of experience. Similarly, data engineers with 6-10 years of experience command a salary of 73.8% higher than professionals in the 3-6 years bracket. This jump lowers to 29.0% when comparing the median salary of 10-12 years work experience bracket with 6-10 years bracket and to 32.5% when comparing 12+ years bracket to 10-12 years.

The median salary of data engineers is higher than that of analytics professionals across all work experience brackets. This gap is highest for professionals with 6-10 years of experience, as data engineers in the bracket command a salary 33.9% higher than overall analytics professionals. This gap is 18.3% for professionals with less than 3 years of experience and 14.2% for professionals with more than 12 years of experience.

---

Median salary of data engineers by sectors

The median salary of data engineers in Internet/eCommerce (INR 28.5 lakhs per annum) is the highest across all sectors. This median salary is nearly similar to Banking/Insurance and CPG/Retail sectors standing at INR 26.6 and 26.9 lakhs per annum, respectively. This median is more than two times higher than the salary received by data engineers in the IT sector (12.6 lakhs per annum). This is mainly because of the high share of data engineers in the sector, many of which fall in the lower salary brackets.

The second-lowest median salary after IT is paid to data engineers in traditional industries like Energy/Industrials, at INR 17.3 lakhs per annum and Telecom at INR 20.7 lakhs per annum. Data engineers in Pharma & Healthcare and Media & Entertainment command a median salary of INR 22.8 lakhs per annum to INR 23.8 lakhs per annum.

---

Median salary of data engineers by company type

In terms of median salary by company type, captives are the highest payers, offering INR 28.7 lakhs per annum. On the other hand, IT firms pay their data engineers the least (INR 12.6 lakhs per annum).

The second-highest median salary is offered by Startups at INR 23.0 lakhs per annum. This is followed by Consultancy Firms (INR 18.3 lakhs per annum), Domestic Firms (INR 17.8 lakhs per annum), and Boutique companies (INR 17.2 lakhs per annum).

Captives offer their data engineers a 67.4% salary higher than Boutique analytics firms and 61.2% higher than Domestic Firms.

---

Median salary of data engineers by cities

The Delhi/NCR is a top destination for data engineers that command a median salary of INR 19.3 lakhs per annum. This is followed by Bangalore and Mumbai at INR 19.0 and 17.5 lakhs per annum respectively.

Chennai offers the lowest median salary at INR 14.8 lakhs per annum, 23.5% lower than Delhi. Despite the high cost of living, data engineers in Mumbai command INR 17.5 lakhs per annum, only INR 50,000 per annum higher than that of Pune (INR 17.0 lakhs per annum).

The increasing startup culture, a business-friendly infrastructure, and proximity to government institutions have helped the Delhi NCR region become a popular destination for businesses to set up operations. As a result, the city commands the highest median salary for data engineers.

---

---

Jobs

Open jobs for Data Engineers

Since the pandemic, nearly every enterprise, regardless of its domain, is adopting technology to drive innovation, optimise processes and stay competitive in the market.

Data has evolved into the new oil that enables businesses to thrive, develop, and grow. Any organisation’s core data science strategy will cover data infrastructure, data storage, mining, modelling, crunching, and metadata management. So to implement these strategies, data scientists and data engineers work closely together. The data scientist focus on analysing, testing, aggregating, refining and conveying data to the enterprise, whereas data engineers lay the foundation for these projects. The data engineers primarily focus on designing, building and arranging pipelines for the stakeholders.

Organisations now have understood the importance and need for data engineers in implementing successful data strategies. Thus, there is a spike in the open jobs for data engineers. As per the AIMresearch, the total number of listed jobs for data engineers is 36,457.

This section discusses the share of open jobs for data engineers across sectors, years of experience and salary brackets.

---

Open jobs across key sectors

terms of sector, the highest number of open jobs are listed across the IT industry, covering nearly 53.8% of the total. However, the IT industry is excluded from the study for better comparison among key sectors.

Excluding IT, eCommerce/Internet has emerged as the largest job provider among key sectors, with a total of 3,796 open jobs, followed by BFSI at 3,728 jobs. Individually, both make up around a third of the total non-IT data engineering jobs in India.

The Energy & Utilities sector also has a noticeable number of open jobs (1,562) for data engineers, making up 13.7% of the total non-IT jobs. Many enterprises in the industrial sector are now in the process of digitisation by collecting data through technologies like IoT to automate processes.

The early adopters of emerging technologies are at the forefront of offering more data engineering jobs than their counterparts.

---

Open jobs by years of experience

Open jobs for professionals with 5-10 years are the highest. The total number of jobs in the category is 16,363, which is 44.9% of the total open job. It is followed by 7,980 open jobs in the 2-5 years of experience category.

The share of open jobs for the freshers or data engineers with 0-2 years of experience is 15.1%. whereas the share of open jobs accounts to 18.1% for data engineers with more than ten years of experience.

---

Open jobs by salary brackets

The distribution of open data engineering jobs with respect to salary brackets is similar to the distribution of jobs across the work experience brackets.

The highest number of open jobs are posted in the salary bracket of INR 10-15 lakhs per annum, which is 10,809 or 29.6% of the total jobs. This is followed by the open jobs in 6-10 lakhs (9,362) and 15-25 lakhs (8,790) brackets.

Open jobs for the entry-level roles commanding salaries between INR 0-6 lakhs per annum stand at 4,722. Around 2,773 or 7.6% of the open jobs are listed for salaries more than INR 25 lakhs per annum.

It has been observed that businesses hire the most data engineers in the range of INR 10-15 lakhs per year. They can multitask, deliver more in less time, and bring stability to the team with their expertise.

---

---

Attrition

Attrition rate of data engineers

As per AIM research, the overall attrition rate for data engineers is 34.1%. The higher demand for data engineers in the market has created many opportunities for engineers’ to hop jobs and command more salaries. In addition, there are several other factors which have contributed to such a high attrition rate, including:

• Mismatch in employer expectations – Organisations are rushing to adopt data science, AI, and Machine learning into their organisations, but do not have the clear goals and processes in place to derive results out of it. Hence, data engineers’ are hired to build sophisticated data infrastructure and data pipelines, while end up being companies’ data janitors.

• Lack of domain knowledge – To build the data pipelines per the business requirement, the data engineers must have a fair understanding of the business model. The lack of domain knowledge can affect the working of business and lead to frustration among data engineers about theirs.

• Rigid organisational structure – Many organisations have a rigid structure of doing business and they are very prone to adapt to changes. However, data engineering is a highly evolving field and data engineers want to work on new and emerging challenges.

---

Attrition of data engineers by sector

At 44.9%, the Travel/Hospitality industry witnessed the highest attrition rate in the last 12 months. This is followed by CPG/Retail (37.0%) and Banking/Insurance (36.0%). The largest recruiter of data engineers, the IT sector, witnessed an attrition rate of 35.6%.

In contrast to their peers, new-age technology-driven sectors such as Internet/eCommerce and Pharma/Healthcare experienced a relatively lower attrition rate. The rate of attrition for data engineers in these sectors stood at 24.2% and 30.6% respectively.

Telecom had the lowest attrition rate with 23.5% of the data engineers in the sector switching jobs in the last 12 months.

---

Attrition of data engineers by company types

At 53.3%, boutique analytics firms witnessed the highest attrition of data engineers among all types of companies. Startups also witnessed high attrition as one in two (49.3%) data engineers working in these firms switched jobs over the last year. This is followed by Captive firms at 46.0%, and consulting and domestic firms at 42.9% and 41.2% respectively. On the other hand, IT companies witnessed the lowest attrition rate at 28.8%.

The worldwide stock market sell-off, the slowdown in VC funding for startups, and cost-cutting in their overall operating expenses are all contributing factors to the greater attrition rate across companies.

---

Attrition of data engineers by experience levels

As the years of experience increase, the rate of attrition for data engineers comes down. The attrition rate is highest (41.7%) for data engineers with an experience of 0-2 years. It is almost the same for data engineers with 3-5 years of experience at 40.6%. The attrition rate comes down to 15.8% for engineers with more than ten years of experience. The attrition rate drops considerably (by 12.88pp) for the 6-10 years experience bracket compared to data engineers in the 3-5 years experience bracket.

Data engineers with 3-5 years of experience are high in demand and usually have multiple job offers in hand when hunting new job opportunities.

---

---

Skills

Key skills for a Data Engineer

As per AIMResearch analysis, the top three skills or technologies that data engineers have or work in are Python (92.0%), SQL (82.0%), and Big Data (63.0%). When it comes to cloud technologies, data engineers can work in Azure (39.0%) or AWS (36.0%) more than GCP (16.0%).

The demand for Python for data engineers remains high across years of experience. A 100.0% of data engineers with more than 12 years of experience can work in Python. On the other hand, the demand for SQL reduces with more years of experience. While 100.0% of data engineers with less than three years of experience can work in SQL, this number reduces to 62.5% for professionals with more than 12 years of experience. This trend is exactly the reverse for Big Data or Data Warehousing. 55.0% of data engineers with less than 3 years of experience can work on Big Data. This number increases to 87.5% for professionals with more than 12 years of experience.

---

---

Market Potential

Data Engineering Technology Capability

Data Engineering Technology Capability

We had come up with this chart in the Analytics India Industry Study 2022. Here we try to see only the key technologies and how data engineering is stacked up against other areas. With high demand and large market potential, data engineering is positioned at the top of the high-impact technologies quadrant.

This highlights the significance of Data Engineering which forms the precursor to developing high-end technologies that it shares the quadrant.

Technologies like Deep Learning and IoT are state-of-the-art technologies that see a huge market potential but will not excel without streamlining data collection processes. Here deep learning will play a crucial role in enabling the technologies.

---

Please read the full report here: