Every year, millions of Indian consumers flock to digital storefronts during marquee retail events. During The Big Billion Days 2025, Flipkart expected 250–300 million unique visitors during the entire event, but recorded 606 million visits in the first 48 hours alone, earning a record single-day GMV of about $200 million.
For engineering teams, those numbers create a unique challenge. Traffic that would normally be spread across months comes flooding in a matter of days, leaving little room for infrastructure failures, misconfigured alerts, or slow incident response.
According to Flipkart, chaos engineering has become key to preparing for those moments. Ahead of recent Big Billion Days events, the company used chaos drills to validate alerting systems, test failure-detection mechanisms, and identify weaknesses in deployment configurations before they could affect customers.
The approach produced tangible results. Flipkart said proactive testing uncovered and remediated six critical misconfigurations in alerting and deployment specifications. Compared with previous sales events, the mean time to detect (MTTD) infrastructure failures improved by approximately 20%, while the mean time to resolve (MTTR) decreased by roughly 25%.
At KubeCon + CloudNativeCon India 2026 in Mumbai, Uma Mukkara, co-creator of the LitmusChaos project, and Aditya Sridasyam, a software development engineer at Flipkart, talked about the platform and engineering practices behind those improvements.
Building Reliability as a Shared Platform
Flipkart, India’s largest digital e-commerce platform, operates across a highly distributed architecture comprising hundreds of tightly coupled microservices running across Kubernetes clusters and virtual machines.
“The traffic that usually runs through the system across an entire year gets concentrated into just a few days during these mega sales,” Mukkara noted. “Given those numbers, reliability cannot be a single team’s concern; it has to be a horizontal platform that everyone can leverage.”
To support that goal, Flipkart built a centralised chaos engineering platform using LitmusChaos, a CNCF-hosted open-source project that deliberately injects controlled faults to identify system weakenesses. The company adopted a strictly open-source foundation to ensure resilience testing remained highly available while accommodating the diverse requirements of internal platform teams.
Before settling on LitmusChaos, the team ran a proof-of-concept evaluating it against Chaos Monkey and Chaos Mesh, two other widely used chaos engineering tools. LitmusChaos won out for its Kubernetes-native architecture, extensibility model, and built-in resilience probes
“Our initial thought was to build a standardised chaos engineering platform where hybrid multi-tenancy was a core requirement,” Sridasyam explained. “We needed to ensure high availability for our resilience testing, especially because different teams came to us with very specific operational requirements.”
Rather than requiring individual engineering teams to deploy and manage separate testing stacks, Flipkart adopted a hybrid multi-tenant architecture. A centralised control plane houses the Litmus Operator (a Kubernetes-native framework), workflow controllers, and a Chaos Center portal backed by MongoDB, allowing multiple teams to share a common testing platform while reducing operational overhead.
The platform centrally manages resource quotas and permissions while providing dedicated service accounts, roles, and deployments for individual platform cohorts such as Database-as-a-Service, Service Mesh, and Kubernetes-as-a-Service. This separation allows teams to independently execute resilience tests without affecting other business-critical platforms.
According to Flipkart, the architecture currently supports 15 central platform teams, enabling them to run failure simulations and generate resilience scores without risking cross-tenant disruption. The result is a model that scales reliability testing across the organisation without introducing new operational risks.
Customising Open Source for Large-Scale Testing
Building a shared platform was only part of the challenge. Running automated resilience experiments reliably at scale required significant customisation of the underlying open-source tooling.
“Litmus is extensible by design and workflow-driven, utilising Argo Workflows underneath. This allows teams to build a sequence of operations and get complicated chaos workflows to run with a single click.”
The workflow-driven architecture enables teams to automate complex failure scenarios and execute them repeatedly as part of their operational readiness process.
However, running those workflows consistently in busy staging environments proved difficult at first. The default on-demand helper pod model introduced scheduling bottlenecks that prevented experiments from executing reliably in densely packed clusters.
Sridasyam said scheduling delays and image pull latency caused more than 10% of staging experiments to fail. To address the issue, the team moved to a DaemonSet-based architecture, deploying long-lived injector pods on every node. According to the company, the change reduced experiment failures to nearly zero and increased confidence that test outcomes reflected genuine system weaknesses rather than issues with the testing infrastructure itself.
The company also encountered limitations when simulating real-world incidents. Certain scenarios required dynamically identifying targets before injecting failures, such as locating the active leader pod in a distributed database cluster.
To support these use cases, Flipkart developed “Script Runner,” a custom fault type built on top of LitmusChaos. The component launches a container that executes a user-defined script and writes the resulting target information into a shared ConfigMap. Subsequent fault-injection stages can then consume those outputs and execute failures against dynamically selected targets.
The capability allows engineers to recreate production-like incidents more accurately than static fault injection alone, improving the realism and effectiveness of resilience testing.
Extending Chaos Engineering Beyond Kubernetes
Flipkart’s resilience platform extends beyond Kubernetes. By integrating with internal cloud APIs, the company brought its virtual machine infrastructure under a single testing framework, enabling both cloud-native workloads and VM-based systems to be subjected to resilience testing through a common platform.
The extended VM chaos runners execute stress chaos and network chaos directly on virtual machine instances, while experiment metrics are consolidated into a unified observability layer powered by Prometheus and Grafana.
According to the company, this provides engineering teams with a single view of system behaviour during simulated failures, helping them validate recovery mechanisms and identify bottlenecks across a much larger portion of the technology stack.
To accelerate adoption across teams, Flipkart also built an internal Chaos Hub, where platform and application teams can publish and reuse experiment templates instead of building chaos workflows from scratch. Rollout began with the central platform teams responsible for Kubernetes, networking, service mesh, and database infrastructure, before gradually extending to application teams across the organisation
From Experiments to Operational Readiness
Flipkart’s chaos engineering platform currently supports more than 2,000 experiment executions across roughly 500 unique testing scenarios and over 100 namespaces from a single interface.
Ahead of recent Big Billion Days events, teams used these chaos drills to validate alerting behaviour, verify failure-detection mechanisms, and identify weaknesses in deployment configurations before they could result in incidents during peak traffic periods.
The team identified six critical misconfigurations and remediated across alerting and deployment configurations.
Reflecting on the initiative, the company said implementing LitmusChaos at its operational scale and empowering platform teams that support hundreds of downstream developers had been an especially rewarding journey.
Flipkart has also given back, contributing more than five fixes and enhancements back to the upstream project, including improvements to multi-tenant probe management, custom image registry support, experiment visibility, and UI consistency—changes that benefit the broader CNCF community using the platform, not just Flipkart's own deployment
The operational discipline built through repeated chaos drills has also delivered measurable improvements in incident response.
The scenarios tested through the platform now serve as the foundation for active corporate runbooks, ensuring remediation procedures are rehearsed well before production incidents occur. As a result, incident response becomes less dependent on ad hoc decision-making and more predictable during high-pressure outages.
Looking ahead, Flipkart plans to make automated chaos testing a mandatory step in its CI/CD pipelines and intends to open-source its DaemonSet-based high-availability injection model for the wider cloud native community to use.
Your reaction
Discussion
What's your take on this story?
No comments yet — be the first to share your view.