pied piper

A Lossless Compression Solution
Making Data Storage Problems Smaller

BIG IDEAS. SMALL FILES.

PIED PIPER / COMPRESSION RESEARCH

WELCOME TO SOMETHING SMALLER.

What is Pied Piper?

Big ideas shouldn't be limited by file size. We're building a more compact way to store data, with attention to every line and every byte. We started with system logs. This is just the beginning.

33.8%

smaller archive

On the HealthApp sample, compared with
the best tested baseline setting.

5 / 6

samples with smaller archives

Compared with the tested settings
of gzip, XZ and Zstandard.

6 / 6

byte-exact restorations

Every restored file matched
its original, byte for byte.

First measurements · 20 October 2026 · small samples of 2,000 lines each

LESS REPETITION. MORE ORDER.

Every line tells a story.
We find a shorter way to keep it.

Text repeats. Numbers change. Store them separately, and a conventional compressor may find more patterns.

Find the common ground

Separate repeated text from numerical fields. Keep a single template instead of storing the same fragments again.

Encode the changes

Group numbers by field and choose a compact representation: the values themselves or the differences between them.

Keep every detail

Apply XZ and store everything needed for exact restoration, right down to leading zeros and line endings.

A SIMPLE EXAMPLE

One message.
Plenty of repetition.

Switch views to see the idea behind separating text and values.

A simplified illustration of the principle, not the contents of an actual archive.

10:00:01 INFO request=1001 status=200
10:00:02 INFO request=1002 status=200
10:00:03 INFO request=1003 status=200
10:00:04 INFO request=1004 status=200

The text repeats; only the time and request number change.

FEWER PROMISES. MORE MEASUREMENTS.

Let the bytes do the talking.

Six real samples from Loghub.
Archive sizes include all metadata.

HealthApp

2,000 lines · 187,456 bytes before compression

Archive size, bytes · smaller is better

Baselines: gzip 9, XZ 9 and Zstandard 3 / 19 / 22. Our prototype uses XZ 9 as its final stage. Timings come from a single run.

All experiment results

Complete archive sizes. The best baseline is the smallest archive across the five tested settings of other compressors.
Sample	Best baseline	Baseline, bytes	Pied Piper, bytes	Difference
Apache	Zstandard 19	7,161	5,208	−27.3%
HDFS	XZ 9	42,768	32,760	−23.4%
Linux	Zstandard 19 / 22	11,326	11,576	+2.2%
OpenSSH	Zstandard 19 / 22	10,854	9,204	−15.2%
HealthApp	XZ 9	13,304	8,812	−33.8%
Zookeeper	Zstandard 22	16,554	15,280	−7.7%

EXPERIMENT NO. 001

A good idea
starts with
a fair test.

Download results as JSON

We're just getting started. Our first prototype shows that understanding log structure can help compression. Next, we need to find out how well that result holds up.

Data: Six Loghub samples of 2,000 lines each. Three initial samples and three additional datasets.
Validation: Every restored file compared byte for byte with its original, plus 14 edge-case checks.
Limitations: Our archive was larger on Linux. The Python prototype is slower at decompression. Memory usage has not yet been measured.
Next: Full datasets, independent sources, and comparisons with specialized log compressors.

We combine established techniques and explore how they work together. Research novelty has not been established. References: Loghub ↗, LogReducer ↗, LogShrink ↗.

BIG IDEAS START WITH A SMALL TEAM.

Who We Are

Anton Sidorov, founder of Pied Piper — Anton Sidorov
Founder & CEO