Guide To ClearML: Zero Integration MLOps Solution

ClearML is an open-source, zero-integration machine learning/deep learning experiment manager and ML-Ops solution.
ClearML: Zero Integration MLOps Solution

ClearML is an open-source, zero-integration machine learning/deep learning experiment manager and ML-Ops solution. It integrates using just two lines of code. Additionally, like GitHub, ClearnML has version control,  but without worrying about conflicts, commit issues. The scalable design of ClearML aims to help developers and researchers manage complex projects with minimal integration effort. It comes with a plethora of helpful functionalities such as:

  • Saving ML models and datasets that can be shared on cloud storage systems.
  • Displaying a summary of python packages used in experiments with their specific version.
  • Web UI for remotely stopping, monitoring, and running experiments.
ClearML architecture and components,

The Python package integrates ClearML into your existing code using two lines of code. The ClearML server stores experiments, models, and workflow data, and the Web UI experiment manager. It is available as a hosted service and open source for you to deploy your own server. ML-Ops orchestration is done by the ClearML agent that can be on a remote system.


Sign up for your weekly dose of what's up in emerging technology.

Installing and setting up ClearML client and agent 

To run a client you need to register an account on a ClearML-server here. You’ll be prompted to create credentials in the getting started screen, generate them, and copy them to your clipboard for later.

ClearML getting started screen

Install clearml client

pip install clearml

To configure the client run


Paste the credentials you generated earlier and confirm the default configuration for the Host, Web App, API, and Filestore.

Integrating with two lines of code
 from clearml import Task
 task = Task.init(project_name='hello', task_name='hello world')
 print("Analytics India") 

When this is run, the output is not only printed on your machine but also logged in the server.

ClearML hello world task out

An agent is needed to run experiments, pipelines, and automation. Remember, we ran the hello world task locally on our system. The agent should be installed and configured on the machine you want it to run. 

pip install clearml-agent 

Configure the agent just like the client.

clearml-agent init

Start the agent in foreground mode. This means the agent will print the output to the screen. In actual service mode, the output will be stored automatically into a temporary file. 

clearml-agent daemon --queue default --foreground

Creating a simple ML pipeline using ClearML

We’re going to create a very basic pipeline with three stages- The first stage will upload the dataset artifact, the second stage will process the data, and the final stage will create and train the model.

Create experiment tasks corresponding to the three stages

  1. Downloading the data and storing it as an artifact named dataset(
 from clearml import Task, StorageManager
 task = Task.init(project_name="pipeline", task_name="pipeline step 1 dataset artifact")

 # only create the task, it will be executed remotely later

 # simulate local dataset, download one, so we have something local
 local_iris_pkl = StorageManager.get_local_copy(

 # add and upload local file containing the dataset
 task.upload_artifact('dataset', artifact_object=local_iris_pkl)

 print('uploading artifacts in the background')
  1. Getting the data from the previous task, processing it, and uploading the processed data(
 import pickle
 from clearml import Task, StorageManager
 from sklearn.model_selection import train_test_split

 Task.init(project_name="pipeline", task_name="pipeline step 2 process dataset")
 # Use either dataset_task_id to point to a tasks artifact or
 # use a direct url with dataset_url
 args = {
     'dataset_task_id': '',
     'dataset_url': '',
     'random_state': 42,
     'test_size': 0.2,
 # store arguments, these will be changed in the cloned task when the pipeline is run
 print('Arguments: {}'.format(args))

 # only create the task, we will actually execute it later

 # get dataset from task's artifact
 if args['dataset_task_id']:
     dataset_upload_task = Task.get_task(task_id=args['dataset_task_id'])
     # download the artifact
     iris_pickle = dataset_upload_task.artifacts['dataset'].get_local_copy()
 # get the dataset from a direct url
 elif args['dataset_url']:
    # simulate local dataset
     iris_pickle = StorageManager.get_local_copy(remote_url=args['dataset_url'])
     raise ValueError("Missing dataset link")

 # open the local copy of the dataset
 iris = pickle.load(open(iris_pickle, 'rb'))
 # "process" data
 X =
 y =
 X_train, X_test, y_train, y_test = train_test_split(
     X, y, test_size=args['test_size'], random_state=args['random_state'])

 # upload the processed data
 print('Uploading process dataset')
 task.upload_artifact('X_train', X_train)
 task.upload_artifact('X_test', X_test)
 task.upload_artifact('y_train', y_train)
 task.upload_artifact('y_test', y_test)

 print('Notice, artifacts are uploaded in the background')
  1. Creating, training, and storing the model(
 import joblib
 import numpy as np
 from sklearn.linear_model import LogisticRegression
 from clearml import Task

 task = Task.init(project_name="pipeline", task_name="pipeline step 3 train model")

 # This will be changed in the cloned task when the pipeline is run
 args = {
     'dataset_task_id': '',


 print('Retrieving Iris dataset')
 dataset_task = Task.get_task(task_id=args['dataset_task_id'])
 X_train = dataset_task.artifacts['X_train'].get()
 X_test = dataset_task.artifacts['X_test'].get()
 y_train = dataset_task.artifacts['y_train'].get()
 y_test = dataset_task.artifacts['y_test'].get()
 print('Iris dataset loaded')

 model = LogisticRegression(solver='liblinear', multi_class='auto'), y_train)

 joblib.dump(model, 'model.pkl', compress=True)
 loaded_model = joblib.load('model.pkl')
 result = loaded_model.score(X_test, y_test)

 print('model trained & stored')

Run the three stages in order to create the base task in the server.

Base tasks on CLearML server
Base tasks created in the server.

Create a pipeline, add the three stages and run it( 

 from clearml import Task
 from clearml.automation.controller import PipelineController

 # Connecting ClearML with the current process
 task = Task.init(project_name='pipeline', task_name='pipeline demo',
                  task_type=Task.TaskTypes.controller, reuse_last_task_id=False)

 # Creating the pipeline
 pipe = PipelineController(default_execution_queue='default', add_pipeline_tags=False)

 # Adding the first stage to the pipeline, a clone of the base tasks will be created and used
 pipe.add_step(name='stage_data', base_task_project='pipeline', base_task_name='pipeline step 1 dataset artifact')

 # overriding  the arguments of the second stage(adding the first stage’s task_id)  before and adding it to the pipeline
 pipe.add_step(name='stage_process', parents=['stage_data', ],
               base_task_project='pipeline', base_task_name='pipeline step 2 process dataset',
               parameter_override={'General/dataset_url': '${stage_data.artifacts.dataset.url}',
                                   'General/test_size': 0.25})

 # overriding the arguments of the the third stage(adding the second stage’s task_id) and adding it to the pipeline
 pipe.add_step(name='stage_train', parents=['stage_process', ],
               base_task_project='pipeline', base_task_name='pipeline step 3 train model',
               parameter_override={'General/dataset_task_id': '${}'})

 # Starting the pipeline 
 # Wait until pipeline terminates
 # cleanup everything

Note that the base tasks are not executed, they are first cloned, and then the clone tasks are executed.

All tasks and pipeline controller on CLearML server
Artifacts created by the three stages

The output of the pipeline controller


More Great AIM Stories

Aditya Singh
A machine learning enthusiast with a knack for finding patterns. In my free time, I like to delve into the world of non-fiction books and video essays.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM