Argo’s Rise To Leading Kubernetes Workflow Engine In Just 4 Years

Argo is an open source container-native workflow engine for Kubernetes.
Listen to this story

Argo is an open-source container-native workflow engine for Kubernetes. It was introduced by Applatex (owned by Intuit), which offers Kubernetes services and open source products. Argo is a workflow orchestration layer designed to be applied to step-by-step procedures with dependencies. It can handle tens of 1000s of workflows at once, each 1,000 steps each. These step-by-step procedures with dependencies are referred to as a Directed Acyclic Graph (DAG) in most current computing solutions.

Some relevant DAG tools include Apache Airflow, Prefect, and Luigi. Just like all other Kubernetes-related capabilities, Argo is declarative, configurable, and meant to operate across environments. If you have already installed the extension on your Kubernetes cluster, users may deploy the Argo manifest, which abstracts the idea of a DAG. 

Argo allows one to automate, schedule, and manage complicated workflows and applications on Kubernetes with relative ease. It is implemented as a Kubernetes Custom Resource Definition (CRD). Argo workflows can be handled via kubectl and natively integrates Kubernetes with services like secrets, volumes and role-based access control (RBAC). Argo’s new software supports workflow capabilities, including parameter replacement, artifact, fixtures, loops, and recursive workflows.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

 Argo workflows can be used in a variety of ways, including:




  • Machine Learning
  • Time- and event-based workflow execution policies
  • Organizing and executing widespread deployments of complex, multifaceted applications
  • CI/CD pipelines
  • ETL, Data Analytics & Data Science
  • Data processing pipelines

Argo is currently being utilised for new discoveries in physics CERN, for 3D rendering at CoreWeave. The latest use is in Intuit’s machine learning and data processing platform. Argo Workflows is used by over 100 companies, including Adobe, BlackRock, Capital One, Alibaba Cloud, Data Dog, Google, GitHub, Datastax, IBM, Intuit, NVIDIA, New Relic, SAP and Red Hat.

(Source : GitHub – Argo Workflow v3.0)

Argo Workflows v3.0

Since its inception three years ago, Argo Workflows has come a long way. It is not only cloud-native but also simple, fast and cost-effective. The latest launch of Argo Workflows v3.0 comes with an array of new features and robust upgrades such as the introduction of brand new APIs for Argo Events, controller high-availability, moving the repository including Go modules support. Furthermore, its key-only artefacts feature makes it easier to perform map-reduce operations. 

New UI 

Argo’s new UI now also supports Argo Events. A new event-flow page enables users to view how event sources and sensors are linked and display animations whenever a message is received. Along with this, one can create and update event sources and sensors directly in the user interface using the same visual language used for workflows. It also has a new workflow log viewer that easily allows one to view the init and tail the whole workflow. 

Argo vs others

As mentioned earlier, there are many new tools for orchestrating tasks and data workflows in the market, each with a unique quality that can make it hard to choose which ones to use. 

While Apache Airflow is the most popular and most feature-rich programme, most teams turn to Argo when they are already using Kubernetes. On the other hand, Luigi is a simpler option for beginners. MLFlow and Kubeflow are better tailored to serving a narrow set of requirements to deploy machine learning models and implement experiments.

Since Argo is built on top of Kubernetes, each task is performed as a separate Kubernetes pod. It is designed for teams who already use Kubernetes for most of their infrastructure but prefer YAML to describe them rather than Python.

Argo’s goal is to enable developers and operators to adopt containers and Kubernetes for developing and deploying distributed applications. Argo is being used in a number of open source projects and enterprises including RocksDB Cloud, Heron, Apache BookKeeper, Apache Pulsar, and AppDynamics. It makes Kubernetes accessible to a broader community of developers.

Last year, the Argo Project joined the Cloud Native Computing Foundation (CNCF) as an incubation-level hosted project. The Argo project hopes to closely work with a number of projects that are already members of the foundation, and to “empower organizations to declaratively build and run cloud-native applications and workflows on Kubernetes using GitOps,” Intuit VP of Product Development Pratik Wadher said in a statement.

Ritika Sagar
Ritika Sagar is currently pursuing PDG in Journalism from St. Xavier's, Mumbai. She is a journalist in the making who spends her time playing video games and analyzing the developments in the tech world.

Download our Mobile App

MachineHack

AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIM Research

Pioneering advanced AI market research

Request Customised Insights & Surveys for the AI Industry

CORPORATE TRAINING PROGRAMS ON GENERATIVE AI

Strengthen Critical AI Skills with Trusted Corporate AI Training

Our customized corporate training program on Generative AI provides a unique opportunity to empower, retain, and advance your talent.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox
MOST POPULAR