At the SkillUp 2021, Sourav Saha, academic dean at Praxis Business School, and Prasad Srinivasa, assistant vice president at Genpact, spoke about the exciting career opportunities in data engineering.
Saha is an educator and an advanced analytics domain expert with around two decades of experience in business consulting, research and development. On the other hand, Srinivasa has over 20 years of experience in developing AI/ML solution frameworks and data analytics products for banking and financial institutions.
What’s the buzz all about
In the age of content overload, organisations have to create personalised solutions to stay relevant and competitive. Companies are collecting a large amount of data — structured, semi-structured and unstructured — from multiple channels in different formats.
Srinivasa believes data engineering, which involves collecting, provisioning and maintaining excellent quality data for insights, is essential in driving business outcomes. For this, a data engineer needs to design and develop a scalable data architecture, set up processes that pool data from multiple sources, check the data quality, and eliminate corrupt data, etc.
Pandemic notwithstanding, Genpact, with more than 1,800+ data engineers, has helped Fortune 500 companies across different industry verticals to navigate the new normal, Srinivasa said.
In January 2021, Genpact acquired data engineering and analytics firm Enquero for an undisclosed amount to bolster enterprise digital transformation services through advanced analytics. In May last year, Accenture acquired Ahmedabad-based Big Data firm Byte Prophecy to enhance AI, analytics capabilities in emerging markets.
“There is a huge demand for competent data engineers. This is the right time to immerse in it and grab the opportunity,” said Srinivasa.
Data engineering vs data science
While data engineers build the underlying infrastructure and architecture for data generation, data scientists use data to derive actionable insights, said Saha.
“There is a serious demand for data engineering professionals,” he added.
Srinivasa talked about different data engineering roles, such as:
- Data orchestration services: This is for entry-level roles, typically involves data integration and data enrichment.
- Data architecture and governance: Involves moving data to the cloud, developing data warehousing, data consolidation, etc., apart from leveraging big data and data lake services for data enrichment. The role also deals with aspects like data quality, metadata etc.
- Data strategy/ advisory: This is for experienced individuals working on developing/defining/designing data strategy and cloud roadmap for enterprise, data-based initiatives, alongside providing AI/ML and advanced analytics services.
Key skills
Srinivasa said a data engineer calls for three skills — data literacy, technical prowess and hands-on experience in developing various use cases.
- Data literacy: Data literacy involves reading and interpreting the data.
- Technical prowess: Knowledge of coding languages like Python and R and ability to design flowchart and develop algorithms.
- A basic understanding of the features of data engineering tools.
- Should be able to understand how to design and develop a data model, entity-relationship diagram, etc.
- Once you are comfortable with at least one database (be it RDBMS (Relational Database Management System) such as MySQL, Oracle and Teradata), you can explore other tools like NoSQL. Most importantly, learn to prepare data for reporting, dashboarding and machine learning models.
- An understanding of Big Data.
- Hands-on experience of use cases/applications of data engineering: Besides technical prowess, hands-on experience is key to become a competent data engineer. You can either work on publicly available datasets or go on websites like Kaggle to participate in data engineering challenges and exposure.
