It was an ordinary Wednesday for Andres Freund, a Microsoft software engineer, as he ran routine tests on his Linux machine. But as he investigated a curious slowdown in the “xz” compression library, a little-known piece of Linux software, he stumbled upon something far more nefarious. This sophisticated backdoor could have given attackers unfettered access to millions of computers worldwide.

“The upstream xz repository and the xz tarballs have been back doored,” Freund posted on the OSS-Security mailing list on March 29th, 2024. The news ricocheted through the Linux world as the implications came into focus.

It soon became clear that the attack was a deliberate and sophisticated subversion of a critical piece of the open-source ecosystem. The malicious code, cunningly disguised, granted attackers a foothold on countless Linux systems worldwide.

“This could have been the most widespread and effective backdoor ever planted in any software product,” warned Alex Stamos, the chief trust officer at SentinelOne, on the severity of the breach.

As the Linux community scrambled to contain the damage, a clearer picture of the attack emerged. Under the alias “Jia Tan,” the person had spent months patiently worming their way into the xz project, first with innocuous contributions, then with subtle malicious changes.

Lasse Collin, the project’s original owner for over a decade, was bullied into giving it up to ‘Jia Tan.’ When he could not keep up with the work, he was pressured by a couple of phantom online accounts to relinquish control of the project. Today, Collin, the original XZ maintainer, has taken back control of the project and is cleaning the code.

The Power of Open Source Community 

As Freund alerted the community, security experts and system administrators worked to understand the scope of the problem and develop a fix. One of the first steps was to identify which versions of xz were compromised. 

The community quickly determined that versions 5.6.0 and 5.6.1 contained the malicious code. Linux distributions that had included these versions in their testing or unstable branches swiftly replaced them with safe versions.

When the backdoor was discovered, Linux distributions immediately acted to protect their users. Debian, for example, replaced the compromised version of xz with an earlier, safe version. They kept the new version number to avoid breaking any dependencies but added a note to clarify that it was actually the older, secure version.

Meanwhile, security experts dug into the malicious code to determine exactly how it worked. They found that the attackers had used a clever trick to hijack certain functions in the xz library, allowing them to run their own code and gain control over affected systems. 

Andres Freund, the Microsoft engineer who first discovered the backdoor, described it as “a very mysterious attack.” He noted that “the attackers clearly spent a lot of effort trying to hide what they were doing.”

Despite the attackers’ attempts to cover their tracks, the open-source community was able to dissect the malware and share their findings. By working together and leveraging their collective expertise, the community was able to develop and distribute fixes for the vulnerability quickly.

Within hours, a group of volunteer developers, security experts, and system administrators had mobilised to analyse the malware, patch the vulnerability, and share vital information. One person wrote on HackerNews, “We had to race last night to fix the problem after an inadvertent break of the embargo.”

The Godot Foundation of the open-source Godot Engine posted on X, “As an open-source project ourselves, we try our best to guard the product and our contributors against malicious actors. We consider ourselves really fortunate to have co-maintainers in many areas, even whole teams, to be able to scrutinise PRs closely.”

“Open source (community) caught it and reacted quickly. Like, good job random people on the Internet, open source worked,” Darren Shepherd, Chief Architect & Co-Founder of Acorn Labs, posted on X. In the face of the unprecedented attack on one of the widely used compression libraries, the open source community’s swift response and collaborative problem solving is what open source is all about: working together to fix problems.