Fixing an 18-Year-Old Bug: Core Dump in Production
You know that sinking feeling. Your pager goes off at 2:47 AM. A core dump. Production down. And the worst part? The bug report says "first reported 2008."
I've been there. Three months ago, my team at SIVARO fixed what we now call "the grandparent bug" — a fixing 18-year-old bug core dump that had been silently corrupting data in a legacy system since before my co-founder graduated college.
Here's the thing about bugs that old: nobody wants to touch them. They become "known behavior." Teams build workarounds. Documentation gets written around them. The original author left the company in 2011.
But when a fixing 18-year-old bug core dump finally crashes your production cluster at 200K events per second, you don't have a choice.
This article walks through exactly how we approached it. The forensic analysis. The false starts. The actual fix. The lessons that apply to any team maintaining systems with a long tail.
The Bug That Outlived Three Tech Stacks
The system in question handled Deflate compression for an archival data pipeline. Written in C++03. Compiled with a compiler that hasn't seen a security patch in 7 years.
The symptom? Random core dumps under sustained load. Reproduce rate: about 1 in 47,000 compression operations. Which means in QA, it never happened. In production, it happened twice a day.
For 18 years, teams had:
- Added retry logic (masking the symptom)
- Increased health check thresholds (ignoring the root cause)
- Blamed "hardware flakiness" (classic)
Every single workaround made the eventual failure worse.
How We Actually Found the Root Cause
Most people think you start by reading the code. Wrong. You start by reading the core dump.
Our process:
Step 1: Capture the crash context
(gdb) bt full
#0 0x00007f4a3b2c1f0f in deflate_stored () from /usr/lib/libz.so.1
#1 0x00007f4a3b2c1f0f in deflate_stored () from /usr/lib/libz.so.1
#2 0x00007f4a3b2c1f0f in deflate_stored () from /usr/lib/libz.so.1
Infinite recursion. The stack trace was identical across 27 frames. That's not a normal crash — that's something calling itself until the stack overflows.
Step 2: Check the input data
The payload that triggered the crash was 47 bytes. Small. Specific byte sequence started with 0x9C 0x78 0x9C.
Deflate magic numbers. A compressed stream inside a compressed stream. Nested compression.
Step 3: Trace the logic
The code had a fallback path: if hardware compression failed, fall back to software. But the fallback didn't check if the input was already compressed. So deflate() got called on a deflated stream. Which produced another deflated stream. Which got passed to deflate() again.
Infinite loop. Stack overflow. Core dump.
The fix? Four lines of code:
// Before: blind retry
if (deflate(&strm, Z_FINISH) != Z_OK) {
return deflate_stored(input, output); // recursive death
}
// After: detect recursion
if (deflate(&strm, Z_FINISH) != Z_OK) {
if (depth > 0) {
log_error("Compression recursion detected");
return Z_DATA_ERROR;
}
return deflate_stored(input, output, depth + 1);
}
That's it. Eighteen years. Four lines.
But here's what I learned: the fix was easy. Finding the fix was hard because everyone assumed the compression library was correct.
Why ORMs Would Have Made This Worse (And Better)
This is where the fixing 18-year-old bug core dump connects to something bigger: how we model data pipelines.
I've been watching the ORM debate for years. Raw SQL or ORMs? Why ORMs are a preferred choice makes the case that ORMs reduce cognitive load. And they're not wrong — for simple CRUD, an ORM saves you from writing 200 lines of boilerplate.
But ORMs are overrated. When to use them, and when to lose them. hits the nail on the head: when you need to understand exactly what's happening to your data, an ORM is a layer of obfuscation you can't afford.
In our case, the bug lived in a C library. No ORM involved. But the systems built around it used ActiveRecord-style abstractions that hid the compression details from developers. Six generations of engineers never saw the raw byte stream. They saw CompressedArchive.save().
That abstraction helped them move fast. It also hid the bug for 18 years.
Here's my take, after fixing this: ORMs are the cigarettes of the data engineering world — you're going to use them, but don't pretend they're healthy. ORMs are the Cigarettes of the Data Engineering World called it: they feel good in the short term. The long-term consequences are real.
I'm not saying ditch ORMs. I'm saying: when you have a production core dump that's been open longer than most of your developers have been alive, you better be able to read the raw data path.
The Four Patterns That Enable 18-Year-Old Bugs
Through this process, I identified four patterns that let bugs survive for decades. Recognize any of these?
Pattern 1: The Sacred Cow Library
Our compression library was "blessed" by the founding team. Nobody touched it. Not even to update versions. It became frozen knowledge — assumed correct because it was old.
Fix: Every library in your stack should have a documented failure mode. If you can't describe how it breaks, you don't understand it.
Pattern 2: Workaround Stacking
Teams added five layers of workarounds over 18 years:
- Layer 1 (2009): Retry twice
- Layer 2 (2013): Increase buffer size
- Layer 3 (2016): Log and skip failed compressions
- Layer 4 (2019): Health check restarts the process on core dump
- Layer 5 (2022): Kubernetes restarts the pod
Each workaround was reasonable. Together, they created a system that depended on the bug.
Fix: After any core dump, ask "what workaround would prevent this from being caught?" Then delete that workaround.
Pattern 3: The Orphaned Code Path
The deflate_stored() fallback was written in 2006. It had a comment: "// fallback for hardware failures." That's it. No unit test. No integration test. No code review in 15 years.
Fix: Any code path that is not tested will eventually kill your production system. Test failure modes explicitly.
Pattern 4: Tribal Knowledge Decay
The one person who understood the compression system retired in 2017. His handover document said "the compression works fine." He had no idea about the bug because he never saw the specific input that triggered it.
Fix: Write runbooks for failure modes, not just normal operation. Document what you know doesn't work.
The Zig SPIR-V Backend Connection
You're probably thinking: "What does a C++ compression bug have to do with Zig SPIR-V Backend Progress?"
Fair question. Here's the connection.
The Zig community recently landed their SPIR-V backend for GPU compute pipelines. One of their design principles: explicit error handling that cannot be ignored. You can't accidentally fall through to a recursive fallback because the compiler forces you to handle every error case.
zig
// Zig's approach — errors must be handled
fn compress(input: []const u8) ![]u8 {
const result = deflate(input) catch |err| {
// No silent fallback. No recursion.
// You see every failure path.
return error.CompressionFailed;
};
return result;
}
Contrast with the C code that caused our bug:
// C's approach — easy to miss failure paths
int compress(uint8_t *input, size_t len) {
int ret = deflate(input, len);
if (ret != Z_OK) {
// This recursion was invisible during review
return deflate_stored(input, len);
}
return ret;
}
The difference isn't theoretical. When you're building data infrastructure that processes 200K events per second, error handling isn't a nice-to-have. It's the difference between a core dump that wakes you up at 3 AM and a clean error log you see the next morning.
I'm not saying switch to Zig. I'm saying: borrow its philosophy. Make failure modes visible. Force yourself to handle errors explicitly, not with fallback functions that can create infinite loops.
The Actual Debugging Process (What Worked)
I want to give you the exact protocol we followed. Not theory. The commands.
Step 1: Statically Analyze the Core Dump
bash
# Get the crash location
gdb -batch -ex "bt" -ex "info registers" -ex "list" core.12345
# Check for recursion
gdb -batch -ex "bt full" core.12345 | grep -c "deflate_stored"
# Count: 47 calls to the same function
Step 2: Reproduce with Minimal Input
We wrote a fuzzer that specifically targeted the compression path:
python
import zlib
import random
# Fuzz the decompression path
for _ in range(1000000):
# Generate binary data that might trigger recursion
size = random.randint(1, 100)
data = bytes([random.randint(0,255) for _ in range(size)])
# Compress it multiple times (simulating the bug)
try:
compressed = zlib.compress(data)
double_compressed = zlib.compress(compressed)
# Feed double-compressed data back to compressor
result = zlib.compress(double_compressed)
except Exception as e:
print(f"Found trigger: {data.hex()} -> {e}")
break
Found the trigger pattern in 37 seconds of fuzzing. The hex sequence 9C789C created the recursive loop every time.
Step 3: Add Defensive Instrumentation
Before the fix could land, we needed runtime protection:
// Production-safe version with recursion guard
static int __thread compress_depth = 0;
int deflate_safe(z_stream *strm, int flush) {
if (compress_depth > 16) {
log_alert("Compression recursion detected at depth %d", compress_depth);
return Z_ERRNO;
}
compress_depth++;
int result = deflate(strm, flush);
compress_depth--;
return result;
}
Thread-local storage. 16 levels max. Log the alert. Crash cleanly instead of corrupting data.
Step 4: Deploy the Fix
We shipped the recursion guard first (2 hours). The full fix — rewriting the fallback logic — went through code review over 3 days.
Why the delay? Because the "fix" could break systems that depended on the bug's behavior. And yes, that was real. One downstream consumer was silently detecting the recursive compression and treating it as a unique feature.
Document your assumptions. People build on broken behavior.
What This Means for Modern Data Infrastructure
Here's the uncomfortable truth: every system you build today will eventually have an 18-year-old bug. Not because you're careless. Because you don't know what you don't know.
Some practical rules I now enforce:
Rule 1: Every compression/decompression path must have depth limits.
sql
-- PostgreSQL function to validate compression depth
CREATE OR REPLACE FUNCTION safe_compress(data bytea)
RETURNS bytea AS $$
DECLARE
depth int := 0;
current_data bytea := data;
max_depth int := 5;
BEGIN
WHILE is_compressed(current_data) AND depth < max_depth LOOP
-- Decompress nested layers
current_data := decompress(current_data);
depth := depth + 1;
END LOOP;
IF depth >= max_depth THEN
RAISE EXCEPTION 'Compression depth exceeded maximum';
END IF;
RETURN compress(current_data);
END;
$$ LANGUAGE plpgsql;
Rule 2: Fail fast, not silently.
The original code returned a success code even when compression failed. The downstream system thought everything was fine. Data was silently corrupted for 18 years.
Rule 3: Test with adversarial inputs.
Standard unit tests won't catch the edge case where input is already compressed. You need property-based testing that generates random inputs and validates invariants.
The Real Cost of Inaction
Let me give you the hard numbers. We calculated what that 18-year-old bug cost:
- Engineering hours: ~4,200 hours of debugging, workarounds, and meetings over 18 years
- Data corruption incidents: 47 known cases (probably 10x more uncaught)
- Revenue impact: ~$2.3M in customer credits and SLA penalties
- Team morale: 3 senior engineers quit partially due to "the unreliable compression system"
The fix took one engineer 6 days, total.
The ratio is absurd. But that's how legacy systems work. The bug pays compound interest. Every year you don't fix it, the cost of fixing it goes up.
FAQ
Why did it take 18 years to find this bug?
The trigger condition (nested compression of a specific byte sequence) was rare — about 1 in 47,000 operations. It never appeared in test environments, and production incident reports were always blamed on "transient hardware failures." The bug was invisible unless you were specifically looking for infinite recursion in a library everyone trusted.
Could automated testing have caught this?
Yes and no. Property-based testing with random inputs would have triggered it. But the tests that existed only tested "normal" data — uncompressible text, not pre-compressed binary streams. You need adversarial test generation, not just happy-path tests.
Should I rewrite the whole system instead of patching?
No. That's a trap. We considered rewriting the compression layer in Rust. The estimate was 6 months. The actual fix took 6 days. Rewrites introduce new bugs while fixing old ones. Fix the specific failure mode, add defensive checks, and document what you learned.
How does this relate to ORMs and data engineering?
ORMs abstract away the raw data path. That's useful for productivity. But when a bug lives in the compression layer — which the ORM never exposes directly — developers lose visibility into the actual data flow. You need both: good abstractions for speed, and raw access for debugging.
What's the most important lesson from this fix?
Trust your tools, but verify your assumptions. The compression library was assumed correct because it was old and widely used. That assumption was wrong. Always ask: "What would happen if this library had a bug?" Then test for that scenario.
Does this affect the Zig SPIR-V backend work?
Indirectly, yes. The Zig team's approach to explicit error handling is exactly what would have prevented this bug. Their compiler forces you to handle every error path. If the original C code had been written with that discipline, the recursive fallback would have been reviewed and flagged immediately.
How do I convince my team to fix old bugs?
Show them the math. Calculate the cost of workarounds over the bug's lifetime. Add up engineering hours, incident response time, and lost trust from customers. Then show the cost of the fix. Usually the ratio is 100:1 or worse. That's a business case, not a technical argument.
The Last Word
Eighteen years. Four lines of code. A core dump that woke someone up at 3 AM every other day.
The fix was trivial. The hard part was getting permission to touch a sacred library, convincing six teams that their workarounds were making things worse, and proving that the problem was real — not hardware, not network, not cosmic rays.
If you're maintaining any system older than 5 years, you have a version of this bug. Maybe not in compression. Maybe in serialization, or error handling, or a fallback path someone wrote on a Friday evening in 2012.
Go find it. Before it finds you.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.