SSH was running just 0.5 seconds slower, so I looked into it, and...
Was there a backdoor in Linux?
— XZ Utils Backdoor, CVE-2024-3094
March 2024.
Andres Freund, who worked as a PostgreSQL developer at Microsoft, discovered something unusual while using a development version of Debian.
SSH logins were slower than usual.
It wasn’t even that slow, though.
The difference was roughly this:
Before: ~0.30s
After: ~0.80s
About 0.5 seconds.
Most people would probably have thought something like this:
“The server’s a little slow today.”
And they would most likely have just moved on.
But Freund didn’t let it slide.
CPU usage was abnormal, and Valgrind was also reporting strange errors.
So he began investigating the cause.
A few days later, I posted this information on a security forum.
There was a backdoor in the XZ Utils used on Linux.
And it wasn’t just a simple vulnerability.
It was malware that someone had intentionally planted.
But what is XZ, and why did it cause such a stir?
If you’ve used Linux, you’ve probably come across a file like this at least once.
linux.tar.xz
backup.tar.xz
source.tar.xz
XZ is a compression format, and xz-utilsis the set of tools and libraries that handle it.
The problem is that this is a very low-level library.
Application
|
v
System Libraries
|
v
liblzma
|
v
XZ Utils
Even if you don’t use the `xz` command directly, other programs on your system may depend on `liblzma`.
And in certain Linux environments, there was one unexpected connection.
sshd
|
v
libsystemd
|
v
liblzma
OpenSSH itself did not use XZ directly.
However, in some systemd-based distributions, such as Debian, a structure was in place where `sshd` indirectly loaded `liblzma` while using systemd-related features.
This provided an excellent entry point for attackers.
The backdoor wasn’t even blatantly present.
When people think of searching for malware, they usually imagine opening the source code to look for things like this.
connect_to_attacker();
execute_payload();
Naturally, the attacker did not do that.
One of the reasons the XZ incident became so famous was that the concealment method was quite sophisticated.
Parts of the malicious logic were hidden inside files that looked like legitimate test data.
tests/files/bad-3-corrupt_lzma2.xz
tests/files/good-large_compressed.lzma
Judging by the name alone, it looks like a fixture for testing a compression library.
In fact, it was stored at `tests/files`.
However, during the build process, an obfuscated shell script extracted the data from within this file and reassembled it.
The general workflow was as follows.
Source Tarball
|
v
Modified Build Script
|
v
Fake Test Data
|
v
Hidden Payload Extraction
|
v
Object Code Injection
|
v
Modified liblzma
There was an even more sophisticated aspect to it.
The critical M4 macro that triggers the attack was not present in the standard source tree of the Git repository but was included in the distributed 5.6.0 and 5.6.1 source tarballs.
Therefore,
Git source
the results visible to someone merely reviewing the code
Release tarball
and those who actually build the code might have seen different results.
This is a very dangerous pattern in supply chain attacks.
The target was SSH
Ultimately, the tampered `liblzma` file checked whether specific conditions were met during execution.
It wasn’t malware that ran indiscriminately on any system.
Analysis revealed that the attack code was configured to operate primarily in specific environments such as the following.
Linux
x86_64
glibc
GCC
GNU ld
Debian/RPM build environments
systemd
sshd
In other words,
“Let’s infect every possible PC.”
but rather
“Let’s run only in environments that are highly likely to host actual Linux servers.”
.
This type of condition checking also makes analysis more difficult.
If a security researcher runs the binary in any environment
Nothing happens.
it becomes much harder to determine whether it is malware.
They even checked to see if it was being debugged
It appears the attacker had even considered the possibility of the code being analyzed.
Among the conditions Freund identified was logic that checks for specific environment variables and debugging environments.
For example, the backdoor operates normally in a legitimate environment but changes its behavior in an analysis environment.
Normal Environment
|
v
Backdoor Active
Debug Environment
|
v
Backdoor Disabled
Behavior was observed that checks for LD_DEBUGs, LD_PROFILE, and certain debugger environments.
This is a concept similar to “anti-analysis” and “anti-debugging,” which are commonly discussed in malware analysis.
At this point, it’s hard to believe that a developer simply inserted strange code by mistake.
However, the truly frightening part isn’t the code itself.
If you look at the XZ incident purely from a technical perspective, topics like obfuscation, dynamic linkers, IFUNC, and SSH hooks stand out the most.
But personally, I find another aspect even more intriguing.
The attacker didn’t hack the project to steal the maintainer’s account.
They became the maintainer themselves.
The account at the center of the attack was Jia Tan, with the GitHub username JiaT75.
The first XZ contribution was in October 2021.
The content wasn’t anything special.
.editorconfig
Update:
After that, they continued to make legitimate bug fixes and code improvements.
2022.
2023.
I contribute consistently.
And I gradually began to earn the project’s trust.
For about two years, I acted just like a regular developer.
To summarize the timeline, it went something like this.
2021
|
+-- First harmless contribution
|
2022
|
+-- Regular patches
+-- Community trust increases
+-- More project responsibility
|
2023
|
+-- Maintainer-level access
+-- Release management
|
2024
|
+-- XZ 5.6.0
+-- Backdoor introduced
Jia Tan contributed to the XZ project for over two years, starting in late 2021, and eventually gained commit and maintainer privileges.
Because of this incident, some in the security community at the time even remarked:
“This isn’t a code supply chain attack—it’s a human supply chain attack.”
Even stranger people are emerging
This is where the story gets even more interesting.
To Lasse Collin, the existing XZ maintainer,
development was moving too slowly.
We need a new maintainer.
People began appearing on the mailing list, pressuring him with comments like this.
Notable names include Jigar Kumar and Dennis Ens.
They pressured the existing maintainer to grant Jia Tan more authority.
As a result, Jia Tan’s role in the project gradually expanded.
It was never definitively confirmed whether these accounts were sockpuppets operated by the attacker or by the same organization.
However, they were strongly suspected based on the timing of their appearance and their behavior.
In terms of structure alone, it reads like a social engineering textbook.
Contributor
|
v
Build Reputation
|
v
External Pressure
|
v
Gain Maintainer Trust
|
v
Obtain Commit Access
|
v
Control Releases
|
v
Insert Backdoor
What’s interesting here is that the attacker didn’t launch a malicious PR campaign from the very beginning.
Instead, they first gained the victims’ trust.
A method even more frightening than GitHub account compromise
Typical supply chain attacks follow this pattern.
Maintainer
|
v
Account Compromise
|
v
Malicious Release
XZ was different.
Attacker
|
v
Become Contributor
|
v
Become Maintainer
|
v
Legitimate Access
|
v
Malicious Release
The access privileges themselves weren’t stolen.
The access was granted through legitimate procedures.
Therefore, this isn’t a problem that can be solved by MFA alone.
As you can see on GitHub, they simply
Authorized Maintainer
made a commit.
Even while talking about Zero Trust, when it comes to open-source projects,
"They’ve been contributing for a long time, so they must be trustworthy."
"
The XZ incident exploited that vulnerability precisely.
And all of this nearly succeeded—
The version containing the malware
XZ Utils 5.6.0
XZ Utils 5.6.1
.
Fortunately, however, at the time, most of these versions were still in the process of being incorporated into the latest development and pre-release Linux distributions.
Red Hat checked the relevant packages in Fedora 40 Beta and Fedora Rawhide and immediately recommended rolling back to version 5.4.x. RHEL was not affected.
In other words, if the issue had been discovered just a little later,
XZ 5.6.x
|
v
Linux Distribution
|
v
Stable Release
|
v
Production Servers
|
v
Mass Deployment
it could have turned out that way.
That is why this incident is often referred to as
“the incident that nearly caused a major disaster for the entire Internet”
.
While this is a slightly exaggerated way of putting it, it is true that the potential scope of the impact was enormous.
But the reason it was discovered is truly absurd.
Let’s go back to the beginning.
The attacker
had infiltrated a legitimate project for an extended period
gained maintainer privileges, and
hid the payload in a test file,
created different versions of the release tarball and the Git source,
executed the payload only on specific Linux environments,
bypassed the debugger environment, and
and even tampered with the SSH authentication path.
It was quite sophisticated.
But the reason they were eventually caught was
SSH became slightly slower.
In the measurements published by Freund, there was a case where a failing SSH connection slowed down from approximately 0.299sto 0.807s.
The backdoor’s process of searching for symbol tables and other items in memory consumed CPU resources and created latency.
From the attacker’s perspective, it was a truly minor performance bug.
However, the developer did not simply overlook that minor anomaly.
Unexpected CPU Usage
|
v
SSH Latency
|
v
Performance Investigation
|
v
liblzma
|
v
Obfuscated Payload
|
v
Supply Chain Backdoor
I suspect this might be one of the most expensive 0.5 seconds in the history of security.
What white-hat hackers can learn from this incident
Looking at the XZ incident, we can see that security analysis doesn’t always
Find CVE
Run Exploit
Get Shell
.
The starting point was simple.
Why is this process slower?
From there,
perf
Valgrind
GDB
Dynamic linking analysis
Binary analysis
Build script analysis
Git history analysis
they expanded their scope.
In fact, in his initial public post, Freund did not refer to himself as a “security researcher” or a “reverse engineer.”
He was simply a developer who followed strange system behavior to its logical conclusion.
This is one of the key skills in security:
Not brushing it off by thinking, “I guess that’s just how it is.”
Why is the CPU usage so high?
Why is SSH running slowly?
Why did this dependency suddenly appear?
Why is the release tarball different from the Git source?
These small discrepancies can sometimes be the starting point for uncovering actual security breaches.
Another lesson: Open source is not free infrastructure
There’s been a lot of discussion since the XZ incident.
When we think of the core infrastructure of the internet,
Google
Microsoft
Amazon
Cloudflare
we tend to think of large corporations.
However, if you look beneath the surface, you’ll find that, surprisingly,
One Maintainer
One Repository
One Library
there are cases where countless systems around the world are supported by them.
XZ was one of them.
The attacker didn’t just focus on technical vulnerabilities.
They viewed the maintenance structure of the open-source project itself as an attack surface.
Consequently, in supply-chain security, the focus has expanded beyond code reviews to include
maintainer access control
reproducible builds
release artifact verification
multi-party review
dependency provenance
Maintainer Burnout
We began to view even these kinds of issues as falling within the realm of security.
How did I check whether my server was affected at the time?
One of the most basic ways to check at the time was to look at the XZ version.
xz --version
The two main versions involved in the issue were as follows.
5.6.0
5.6.1
It was also helpful to investigate which libraries were linked to “sshd.”
ldd "$(which sshd)"
However, this was not an incident where the mere presence of a single version string was enough to definitively determine whether exploitation was actually possible.
This is because various factors—such as the distribution, architecture, build method, and whether systemd was involved—all played a role.
Summary
The attack sequence in the XZ Utils incident can be summarized as follows.
Build Reputation
|
v
Gain Maintainer Access
|
v
Control XZ Releases
|
v
Hide Payload in Test Files
|
v
Manipulate Build Process
|
v
Inject Malicious liblzma
|
v
Reach sshd Indirectly
|
v
Potential Pre-auth RCE
The preparation period for the attack lasted over two years.
What ultimately brought this long-running operation to an end was neither massive security hardware nor
an AI-based malware detector,
nor a powerful EDR.
It was a small question raised by a single developer.
“But why is SSH using so much CPU?”
One of the most dangerous phrases in security is
“I guess that’s just how it is,”
If that’s the case,
then one of the most powerful questions might, surprisingly, be quite simple.
“Why?”
The XZ backdoor incident demonstrated just how much that single question can prevent a major disaster.
끝까지 읽었네… 메밀 승인 🐾