To a business, machine learning can deliver much-needed insights in a faster and more accurate way.
The main objective of having a proper pipeline for any ML model is to exercise control over it. A well-organised pipeline makes the implementation more flexible. It is like having an exploded view of a car engine where you can pick the faulty pieces and replace it- in our case, replacing a chunk of code.
The term ML model refers to the model artefact that is created by the training process.
The learning algorithm finds patterns in the training data that map the input data attributes to the target (the answer to be predicted), and it outputs an ML model that captures these patterns.
A model can have many dependencies and to store all the components to make sure all features available both offline and online for deployment, all the information is stored in a central repository.
A pipeline consists of a sequence of components; components which are a compilation of computations. Data is sent through these components and is manipulated with the help of computation.
Pipelines, unlike the name, would suggest, are not one-way flows. They are cyclic in nature and enable iteration to improve the scores of the machine learning algorithms. And, make the model scalable.
A typical machine learning pipeline would consist of the following processes:
- Data collection
- Data cleaning
- Feature extraction (labelling and dimensionality reduction)
- Model validation
- Visualisation
Data collection and cleaning are the primary tasks of any machine learning engineer who wants to make meaning out of data. But getting data and especially getting the right data is an uphill task in itself.
Data quality and its accessibility are two main challenges one will come across in the initial stages of building a pipeline.
The captured data should be pulled and put together and the benefits of collection should outweigh the costs of collection and analysis.
For this purpose, a data lake is recommended for every organisation. A data lake is a centralised repository that allows the user to store both structured and unstructured data at any scale. It also enables ad-hoc analysis by applying schemas to read, not write. In this way, the user can apply multiple analytics and processing frameworks to the same data.
Since every case has its own bargain for the amount of data, usually in an unsupervised setting, things can go out of hand if the quantity of data available for training is less.
Use Cases
A machine learning model’s life cycle needs to be more adaptable to model tuning and monitoring. With new data coming in frequently, there can be significant changes in the outcomes.
Currently, improvements are being made to the existing neural networks to make them run even when the data is vague and when there is a lack of labelled training data.
The concept of meta-learner, where the model infers the values predicted from other lower level AI models. Lower level models like those that are built on image classification, reinforcement learning tasks, etc.
Whenever the meta-learner gets a prediction right, it rewards itself and gets penalised by the actual value when wrong. This helps in optimizing the low-level AI model’s architecture, hyperparameters, and dataset tuning.
Google, on the other hand, has released Snorkel framework to use diverse organizational knowledge resources like internal models, ontologies, knowledge graphs to generate training data for machine learning models at web scale.
For example at LinkedIn, the blueprint of a machine learning model would more or less consist of the same procedures; data collection, processing, training and testing the models and so on.
The major part of data or to be more precise, the most crucial data with respect to LinkedIn is based on the kind of jobs liked, jobs saved and connections made. So, recommending jobs to an individual and calculating the probability of a job posting being checked are one of the few important features of the dataset.
At LinkedIn, the ML team proceeds by building a domain-specific language (DSL) and then a Jupyter notebook to integrate the select features and for parameter tuning.
Most of the model training occurs offline where the ML teams train and retrain the models every few hours. For this, they avail the services of Hadoop. LinkedIn’s own Pro-ML training service is updated with newer model types for hyperparameter tuning. This training service leverages Azkaban and Spark to ensure that there is no missing input data.
AI teams are closely connected to the product team. This bridges the gap for researchers to collaborate and share their findings with fellow experts who might be working on similar problems. Hence reducing the redundancies and increasing output.
Currently, enterprises are struggling to deploy machine learning pipelines at full scale for their products. Common problems include- talent searching, team building, data collection and model selection to say few. To tap the most out of AI, it is necessary to build service-specific tools and frameworks in addition to the existing models.
