Argo is an open-source container-native workflow engine for Kubernetes. It was introduced by Applatex (owned by Intuit), which offers Kubernetes services and open source products. Argo is a workflow orchestration layer designed to be applied to step-by-step procedures with dependencies. It can handle tens of 1000s of workflows at once, each 1,000 steps each. These step-by-step procedures with dependencies are referred to as a Directed Acyclic Graph (DAG) in most current computing solutions.
Some relevant DAG tools include Apache Airflow, Prefect, and Luigi. Just like all other Kubernetes-related capabilities, Argo is declarative, configurable, and meant to operate across environments. If you have already installed the extension on your Kubernetes cluster, users may deploy the Argo manifest, which abstracts the idea of a DAG.
Argo allows one to automate, schedule, and manage complicated workflows and applications on Kubernetes with relative ease. It is implemented as a Kubernetes Custom Resource Definition (CRD). Argo workflows can be handled via kubectl and natively integrates Kubernetes with services like secrets, volumes and role-based access control (RBAC). Argo’s new software supports workflow capabilities, including parameter replacement, artifact, fixtures, loops, and recursive workflows.
Argo workflows can be used in a variety of ways, including:
- Machine Learning
- Time- and event-based workflow execution policies
- Organizing and executing widespread deployments of complex, multifaceted applications
- CI/CD pipelines
- ETL, Data Analytics & Data Science
- Data processing pipelines
Argo is currently being utilised for new discoveries in physics CERN, for 3D rendering at CoreWeave. The latest use is in Intuit’s machine learning and data processing platform. Argo Workflows is used by over 100 companies, including Adobe, BlackRock, Capital One, Alibaba Cloud, Data Dog, Google, GitHub, Datastax, IBM, Intuit, NVIDIA, New Relic, SAP and Red Hat.
(Source : GitHub – Argo Workflow v3.0)
Argo Workflows v3.0
Since its inception three years ago, Argo Workflows has come a long way. It is not only cloud-native but also simple, fast and cost-effective. The latest launch of Argo Workflows v3.0 comes with an array of new features and robust upgrades such as the introduction of brand new APIs for Argo Events, controller high-availability, moving the repository including Go modules support. Furthermore, its key-only artefacts feature makes it easier to perform map-reduce operations.
Argo’s new UI now also supports Argo Events. A new event-flow page enables users to view how event sources and sensors are linked and display animations whenever a message is received. Along with this, one can create and update event sources and sensors directly in the user interface using the same visual language used for workflows. It also has a new workflow log viewer that easily allows one to view the init and tail the whole workflow.
Argo vs others
As mentioned earlier, there are many new tools for orchestrating tasks and data workflows in the market, each with a unique quality that can make it hard to choose which ones to use.
While Apache Airflow is the most popular and most feature-rich programme, most teams turn to Argo when they are already using Kubernetes. On the other hand, Luigi is a simpler option for beginners. MLFlow and Kubeflow are better tailored to serving a narrow set of requirements to deploy machine learning models and implement experiments.
Since Argo is built on top of Kubernetes, each task is performed as a separate Kubernetes pod. It is designed for teams who already use Kubernetes for most of their infrastructure but prefer YAML to describe them rather than Python.
Argo’s goal is to enable developers and operators to adopt containers and Kubernetes for developing and deploying distributed applications. Argo is being used in a number of open source projects and enterprises including RocksDB Cloud, Heron, Apache BookKeeper, Apache Pulsar, and AppDynamics. It makes Kubernetes accessible to a broader community of developers.
Last year, the Argo Project joined the Cloud Native Computing Foundation (CNCF) as an incubation-level hosted project. The Argo project hopes to closely work with a number of projects that are already members of the foundation, and to “empower organizations to declaratively build and run cloud-native applications and workflows on Kubernetes using GitOps,” Intuit VP of Product Development Pratik Wadher said in a statement.