Header image source: Compression and Prediction. Why Language Models Are Really Compression Engines | by Eleventh Hour Enthusiast | Medium via Medium via Google — cropped to 16:9 and colour-adjusted.
Key takeaways
- Gzip matches LLMs on next-token prediction only when primed with a corpus in its window
- Gzip fails at abstraction, reasoning, and long-range dependencies due to its 32KB limit
- The experiment exposes how loose our definition of ‘language model’ has become
Gzip just outperformed some neural language models on certain tasks. Not by a little—by enough to force a reckoning. A compression algorithm, with no pretrained weights, matched or exceeded some large language models on certain tasks. The method involves including a corpus in gzip’s window to prime the model for language modeling.
This isn’t a thought experiment. It’s a real result. And it exposes how loose our definition of "language model" has become.
The Direct Answer: Yes, But Only in the Narrowest Sense
Gzip can function as a language model by exploiting compression-prediction equivalence. Feed it a corpus to prime the model for language modeling. The logic is straightforward: if a sequence compresses well, it’s statistically probable.
Gzip outperforms some large language models. But before you declare victory for vintage algorithms, consider the constraints:
- Compression aligns with prediction—but only when patterns are local. Gzip excels at next-token prediction because redundancy is predictable. But it fails entirely at tasks requiring abstraction, reasoning, or long-range dependencies.
- Its "knowledge" evaporates with the window. Close the window, and gzip forgets everything. No transfer learning, no fine-tuning, no persistent memory. Neural models carry their pretrained weights everywhere; gzip carries nothing.
- 32KB is a prison. Modern LLMs juggle context windows in the millions of tokens. Gzip’s window holds about 6,000 words—roughly 10 pages of text. Beyond that, it’s blind.
So is gzip a language model? If you define it as any system that assigns probabilities to token sequences, then yes. If you expect semantics, syntax, or meaning—no. The experiment doesn’t redefine language modeling. It reveals how little our benchmarks actually test for.
The Theoretical Foundation: Compression as Prediction
The link between compression and prediction isn’t new. Shannon’s source coding theorem proves it: optimal compression requires modeling the probability distribution of the data. The better you predict the next symbol, the fewer bits you need to encode it.
It replaces repeated byte sequences with references to earlier occurrences. This is prediction in action: the algorithm bets that the next bytes will match something it’s already seen.
This is why gzip performs well on language tasks. Next-word prediction rewards systems that exploit statistical regularities. High-frequency n-grams compress efficiently, and gzip’s sliding window captures these patterns. But gzip and neural language models optimize for different things:
- Gzip prioritizes fidelity. Its goal is to restore the original text exactly. It doesn’t generate plausible text—it just compresses and decompresses without loss.
- Language models prioritize generalization. They generate text statistically similar to their training data, even if it’s novel. Coherence matters more than perfect reconstruction.
The Experiment: How Gzip “Models” Language
The methodology is deceptively simple:
- Prime the window. Feed gzip a corpus (a book, dataset, or text collection) to populate its sliding window with statistical patterns.
- Predict the next token. For a given prompt, generate candidate continuations. Append each to the prompt, compress the result, and pick the candidate with the shortest compressed length.
- Repeat. Extend the prompt with the predicted token and iterate.
The label that compresses best is treated as correct.
But the caveats are severe:
- No pretrained model. Neural language models ship with massive pretrained weights. Gzip’s "knowledge" lives only in its window. Close it, and the model vanishes.
- 32KB is a hard limit. Gzip’s window holds about 10 pages of text. Modern LLMs handle context windows spanning entire books.
- Brittle performance. Gzip excels at tasks with strong, consistent statistical patterns but fails on reasoning, multi-hop inference, or long-range dependencies.
And crucially: gzip isn’t generative. It predicts next tokens but can’t generate coherent text. Ask it to write a poem or news article, and it produces gibberish. It’s optimized for compression, not creation.
What Gzip Reveals About Language Modeling Assumptions
Gzip’s success forces hard questions about what we value in language models:
- We overvalue neural architectures. Gzip outperforms some large language models on certain tasks. This suggests we’ve fetishized architecture over fundamentals. Prediction is about modeling statistical regularities, and neural networks are just one way to do it.
- Our benchmarks are flawed. If gzip excels at next-word prediction, the benchmarks aren’t testing linguistic competence. They’re testing compression efficiency. This isn’t useless, but it’s not what we claim to measure.
- Compression matters more than we realized. The experiment highlights the deep connection between compression and prediction. Models optimized for compression (via quantization, pruning, etc.) might perform better on prediction tasks than expected.
- Old ideas still have teeth. Gzip’s success is a reminder that progress isn’t always about new techniques. Sometimes it’s about applying old ones in new contexts.
But there’s a counterpoint: language models aim to capture meaning, not just statistics. It can’t infer rules, handle ambiguity, or generalize beyond its window. In this sense, it’s not a language model—it’s a statistical tool with surprising reach.
The Limits: Why Gzip Isn’t a Practical Language Model
Despite its benchmark wins, gzip fails as a practical language model:
- Architectural mismatch. Gzip is built for lossless compression, not generative modeling. Its goal is to restore text exactly, not produce novel, coherent output.
- No abstraction. Gzip’s model is purely statistical. It sees bytes, not meaning.
- Task specificity. Gzip works on tasks with strong statistical patterns but fails on reasoning, multi-hop inference, or long-range dependencies. It can’t answer questions requiring synthesis of information from multiple text segments.
- Comparison to Dissociated Press. Like the Markov-chain-based text generator, gzip mimics language without understanding it. Push it beyond its comfort zone, and its output becomes nonsensical or repetitive.
Gzip’s limitations highlight what we actually want from language models: coherence, reasoning, adaptability. It delivers none of these.
The Criticism: Why Gzip Won’t Replace Neural Language Models
Gzip’s hype won’t last. Here’s why:
- Practical shortcomings. Gzip’s fixed window size and lack of fine-tuning make it inflexible. It can’t adapt to new domains without re-priming, and it can’t generalize beyond its window. Neural models enable transfer learning, few-shot adaptation, and multimodal integration.
- Theoretical ceiling. Compression alone can’t capture the hierarchical, recursive structures of human language. Gzip misses syntax, semantics, and pragmatics—essential for most language tasks.
- Industry context. Modern LLMs power real-world applications: chatbots, code generation, translation. Gzip is a one-trick pony—it compresses text, period.
- Historical note. Gzip/deflate is a 1977 algorithm. Neural language models leverage decades of advances in deep learning, from backpropagation to transformers. They’re not just better at language modeling—they’re fundamentally different.
This isn’t to dismiss gzip. Its success is a valuable reminder of the compression-prediction link. But it’s not a replacement for neural language models.
The Broader Debate: What Defines a Language Model?
Gzip’s experiment forces a reckoning: what is a language model?
- Minimalist view: A language model assigns probabilities to token sequences. By this definition, gzip qualifies. It’s a statistical model of text.
- Maximalist view: A language model must capture semantics, syntax, and pragmatics. It must understand meaning, handle ambiguity, and generate coherent text. Gzip fails here.
- Middle ground: Gzip is a statistical language model, not a linguistic one. The debate exposes whether our benchmarks measure modeling or just compression.
This isn’t academic. It shapes how we evaluate and conceptualize language models. Define the term too narrowly, and we overlook valuable approaches. Define it too broadly, and it becomes meaningless.
Prediction is broader than we assumed, and compression is a powerful tool. But compression alone isn’t enough to capture human language.
Lessons for AI: What Gzip’s Success (and Failure) Teaches Us
Gzip’s experiment is a wake-up call:
- Benchmark reform. We need benchmarks that distinguish statistical pattern-matching from genuine language understanding. Next-word prediction is useful, but it’s not enough. We need tests for reasoning, abstraction, and long-range coherence.
- Architectural diversity. Non-neural approaches (n-gram models, compression algorithms) still have roles. Gzip’s success shows they can complement neural models in useful ways.
- Theoretical humility. AI’s progress may rely on rediscovering old ideas. Gzip is a 46-year-old algorithm, but its relevance to language modeling is only now clear. We should revisit old ideas, even if they seem outdated.
- Hybrid models. The future may lie in combining compression efficiency with neural flexibility. Use compression for pretraining or to reduce inference costs. Gzip’s success suggests untapped potential here.
But the biggest lesson is this: we need to define "language model" more precisely. Gzip’s experiment shows that the term has become fuzzy, and our benchmarks measure a mix of compression, prediction, and linguistic competence. If we want progress, we need clarity.
So: can gzip be a language model? In a narrow, technical sense, yes. But in any meaningful sense, no. Its success reminds us that prediction is broader than we assumed, and compression is a powerful tool. But language is more than statistics—it’s a system of rules, meanings, and conventions that gzip, for all its cleverness, cannot capture. The real question isn’t whether gzip counts as a language model. It’s what we’re going to do about the benchmarks that let it win.