Seven Steps To Speed-Up Data Science Project Lifecycle

More than 90 per cent of the respondents nodded in agreement that data collection and data preparation eat up a large chunk of their time.

“A data strategy with a scalable, real-time cloud data platform as a central pillar is key”

Data scientists need clean, current, and consolidated data sets to unlock the potential of machine learning. However, they often face challenges due to infrastructure constraints and data that is siloed and outdated. A data strategy with a scalable, real-time cloud data platform as a central pillar is key to address these challenges.

To provide a roadmap, two industry veterans, Shaji Thomas, VP of Cloud & Data Engineering at Ugam; and Swagata Maiti, Technical Architect of IP & Data Products at Ugam, joined us at Deep Learning DevCon 2021 (DLDC) to engage in a talk on the topic titled ‘To data prep or to data science. That’s the question’. The duo deep-dived into the topic and suggested seven techniques to help data scientists build a scalable data platform. 

The presentation started with a basic question from the attendees: How do data scientists spend their most time on data preparation or building scalable ML models?

More than 90 per cent of the respondents nodded in agreement with Shaji that data collection and data preparation eats up a large chunk of their time. Several bottlenecks exist when it comes to data prep or collection; this includes:

  • Data silos and infrastructure constraints
  • Inability to find the right data
  • The repeated effort of feature engineering
  • Data is not clean
  • Inability to handle streaming data
  • Lack of protection of personally identifiable information (PII) data
  • Testing and deployment is error-prone

Scale-up in seven steps

Despite visible challenges, Shaji says that it is possible to have solutions to these problems, that too in a short span of time. Shaji said, “It’s possible, provided you or your organisation have a strong data strategy in place, adopted a set of techniques that created a scalable data platform that could accelerate the whole data science lifecycle.” He further suggested to have:

  • A scalable cloud data warehouse that ensures multiple benefits such as a central data repository can scale storage and compute separately, support zero-copy clones, deliver full support for DevOps and third-party access data.
  • A data catalogue ensures a structured way to discover data. Having a data catalogue can enhance productivity as it helps discover data quickly, resulting in a continuous update of metadata, and finally helps in getting more context into the data. 
  • A feature store to be able to define, search, and reuse features. Additionally, it helps in tracking model performance and feature drift.
  • Automated data curation and validation process can help define business rules that can normalise the data and curate it into the pipeline.

Talking about streaming data ingestion, Swagata Maiti said, “As per the research, only 40 per cent of manufacturers are using inventory management software, and the rest 60 per cent are still relying on either excel or offline methods. As a result, on average, lots of manpower getting lost with high inaccuracy.” Moreover, large datasets are becoming an uphill task for most organisations, hence adapting streaming data ingestion, one can achieve a massively scalable, resilient to failure and highly available platform for real-time data streaming and complex problem processing in the cloud.

Last but not least, Swagata says that adopting hashing technology to protect PII data is necessary. It helps in the automatic removal of PII data from in-flight streaming systems and helps in the anonymisation of customer data. The methodology used here is shown below.

It is to understand that a data science life cycle is a series of data science steps that you go through to complete a project or analysis. Because each data science project and team are unique, each data science life cycle is also unique. From understanding business problems to data collection, data preparation, data modelling and data deployment – all these steps are equally important and need to be taken care of.

Download our Mobile App

kumar Gandharv
Kumar Gandharv, PGD in English Journalism (IIMC, Delhi), is setting out on a journey as a tech Journalist at AIM. A keen observer of National and IR-related news.

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Upcoming Events

15th June | Online

Building LLM powered applications using LangChain

17th June | Online

Mastering LangChain: A Hands-on Workshop for Building Generative AI Applications

Jun 23, 2023 | Bangalore

MachineCon 2023 India

26th June | Online

Accelerating inference for every workload with TensorRT

MachineCon 2023 USA

Jul 21, 2023 | New York

Cypher 2023

Oct 11-13, 2023 | Bangalore

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox