We backed up 510,000 files and the database did not grow by a single row

Anyone backing up object storage full of small files runs into two costs, and it is the second one that ruins the month: the money burned on requests, and the database that grows alongside the file count until it becomes the bottleneck.

This is the write-up of a test we ran to solve the second problem. It has the numbers, the method and — the part that matters most — what would not hold up in a different context.

The index problem

To recover an individual file from inside an archived bundle, you need to know which bundle it is in and where. That index has one entry per file. With 510,000 files, that is 510,000 entries. With a billion, a billion.

The obvious answer is a table in the database. It works, and we used it for a good while: with a properly sized Postgres, we measured 16,000 to 17,000 inserts per second, stable up to 30 million rows. The detail that nearly slipped past us is that the stability depended on RAM — at 15 GB the rate degraded over the course of the load; at 32 GB it stayed flat. In other words: it works, but it forces you to grow the database alongside the customer's archive, forever.

The alternative we tested was taking the index out of the database. Each bundle's manifest becomes an object in the storage itself, next to the data, and the database keeps only what is small and rarely changes: customers, jobs, bundles. Search reads the manifest on demand and keeps a local cache.

What was measured

A 4 vCPU, 32 GB RAM VM on Oracle Cloud, reading from OCI buckets and writing to AWS. No simulation on either end — real source, real destination.

Synthetic data set of 510,000 files of ~150 KB, spread across three customers and three buckets, with one customer deliberately split across two different buckets to test isolation.

Packing

The database throughout

That is the number in the headline. Half a million files went into the vault and the index table did not take a single row. The database sat essentially idle while the machine packed, compressed and uploaded.

Recovery

Keeping the index out of the database is only worth it if recovery stays fast. We measured both ends:

• One specific file, inside a customer with 300,000 files and zero rows in the database: answer in seconds.
• A whole customer: 30,000 of 30,000 files in 2m20s, around 260 files per second, all verified at the destination.

Disaster drill

The test that matters is not the backup, it is the day the source no longer exists. We restored from only the archive on AWS, without touching the source cloud, and compared the sha256 of every restored file against the checksum recorded in the manifest. Identical, bit for bit.

When a server dies mid-run

Packing runs on a fleet of preemptible machines — the kind the cloud can shut down at any moment, and that cost a fraction of the price. We killed two on purpose during the load. The watchdog detected it and launched replacements in 35 seconds, and the work carried on from where it was.

How far this scales

At ~120 files per second per worker, a billion files would take roughly 96 days with one worker, or 10 days with ten. In a separate campaign, with 1 TB and ~7 million files, we measured ~110 files/s per process and ~330/s with four processes on the same 4-core machine — the bottleneck there is CPU in the concatenation, not network or disk.

What this does not prove

This is the part usually missing from published benchmarks, and without it the numbers above are worth little.

The files were synthetic and of similar size (~150 KB). A real archive has an uneven distribution, and very large files change how packing behaves. Expect different numbers.

Taking the index out of the database has a price, and it shows up on the first search. If the manifest is not in the local cache, it has to be fetched from storage before answering. On a cold cache the first query is slower than a SELECT. In a code review of this very work we found a defect exactly there — cold-cache search was not rehydrating the manifest — which was fixed and revalidated before we considered the test valid. If your usage pattern is constant, scattered searching, the database table may be the right choice.

The per-worker rate is what it is. ~120 files/s is not fast in absolute terms; the gain comes from running many cheap workers in parallel, not from any one of them being quick. If you need a short window with few nodes, this architecture is not for you.

We did not measure cost in money in this test. The savings on requests come from aggregation — 510,000 objects became 16 — and that is straightforward arithmetic, but it is not what this test measured. Do not conflate the two.

How to reproduce it

The path is the same for any tool that aggregates files into bundles:

1. Generate a synthetic data set with known size and count, across more than one bucket, with at least one customer split between two — that is where isolation failures show up.
2. Measure the database before and after: row count, size on disk and CPU during the load. Without the "before", the "after" means nothing.
3. Test recovery at both extremes: a single file and the whole archive.
4. Run the disaster drill by cutting off the source, and compare checksums. A backup that has never been restored is not a backup.
5. Kill a node mid-load, on purpose.

Step 5 is the one almost nobody does, and it is what separates a lab number from an architecture that survives production.

This test was run with NubliVault, by Nublify. The numbers come from our lab, not from a customer — precisely so they can be questioned and redone. If you run something similar and land on different numbers, we want to hear about it.