Why backing up millions of small files costs so much

There is a moment when the cloud bill stops making sense. You look at the volume stored, look at the price list, multiply — and the invoice is far larger than the multiplication. The explanation is almost always the same: you are not paying for space, you are paying for operations.

The calculation nobody does upfront

Object storage providers charge for two very different things:

• Storage, per gigabyte per month. That is the number everyone looks at.
• Requests, per operation. That is the number almost nobody adds up.

While files are large, the second line is irrelevant. A 10 TB archive made of 500 MB files is about 20,000 objects — request cost disappears next to storage.

Now change the average size. The same 10 TB in 300 KB files becomes roughly 35 million objects. Copying that to a backup bucket means 35 million write operations. And since almost every backup runs again periodically, that number repeats on each run.

The maths is direct:

objects  = total_volume ÷ average_size
write_cost = (objects ÷ 1,000) × price_per_thousand_PUT

Run that with your real volume and your region's price before deciding anything. It usually explains the invoice on its own.

The minimum billable size trap

There is a second effect that makes everything worse, and this one is specific to archive tiers.

In AWS deep archive classes, every object carries separately billed metadata: 32 KB charged at the archive tier's rate, plus 8 KB charged at the standard class rate. That is 40 KB of overhead per object.

For a 300 MB file, 40 KB is noise. For an 8 KB file, you are paying for 48 KB — six times the actual data. Multiply by millions of objects and "the cheapest AWS tier" ends up costing more than the standard one.

There is also a minimum billing period: 180 days in deep archive. Deleting an object before that saves nothing — you pay the 180 days regardless. That makes small files with short lifecycles a terrible candidate for individual archiving.

Why copying object by object is the wrong strategy

Most sync tools do the obvious thing: for each object at the source, one write operation at the destination. It is correct, it is simple, and it is exactly what gets expensive when files are small. You pay for:

1. One write operation per file, on every run that includes it.
2. The 40 KB of metadata per file, if the destination is an archive tier.
3. The 180-day minimum per file, even for ephemeral ones.

None of those improve with volume discounts. They scale linearly with the number of files — and the number is precisely what grows in your scenario.

The way out: change the unit

If the cost is per object, the answer is to reduce the number of objects without losing a single file. In other words: group before you send.

Packing thousands of files into one compressed bundle turns thousands of write operations into one. Those same 35 million objects, grouped into bundles of a few gigabytes, become a few thousand operations. All three lines fall together: fewer requests, metadata charged once per bundle instead of per file, and the 180-day minimum applied to a handful of large objects.

Compression is a bonus, not the point. The point is the count.

The problem grouping creates — and how to solve it

Grouping has an obvious cost: you lose the ability to recover a single file. If restoring a 300 KB document requires downloading a 5 GB bundle, you have traded a cost problem for an operations problem.

That is why grouping only works alongside an index. Every file needs to be recorded with the bundle it lives in and its position inside it. With that index, recovering an individual file is still possible: find the bundle, read only the necessary range, extract that one file.

Without an index, grouping is savings today and pain on the first recovery. With an index, it is savings with no operational trade-off.

What to do before deciding

1. Measure the real average size of your objects. Do not estimate — list and divide. The difference between 300 KB and 3 MB changes the conclusion entirely.
2. Add up the requests, not just storage. Use the formula above with your region's current price list.
3. Check the minimums of the destination tier, both size and retention.
4. Simulate before moving. Any serious tool should tell you how many objects and how much cost the operation will produce before writing the first byte.

Prices change and vary by region — always confirm against the provider's official table. What does not change is the shape of the bill: in archives of small files, the count matters more than the volume.

NubliVault was built around that idea: it packs each client's files into .tar.zst bundles, ships them to AWS, and keeps a searchable manifest that lets you restore an individual file without downloading the whole bundle. You can simulate the cost or try it for 14 days.