Maximum usable size of the repository? Petabyte scale possible?

General Topics
7

I pulled the trigger on it, got ~257TB in the repo for now (after compression) and pumping more:

Count: 126900820
Total Bytes: 283.2 TB
Total Packed: 257.6 TB (compression 9.0%)
By Method:
  (uncompressed)         count: 86253380 size: 193.6 TB
  zstd-fastest           count: 40647440 size: 89.6 TB packed: 64.1 TB compression: 28.5%
Average: 2.2 MB
Histogram:

        0 between 0 B and 10 B (total 0 B)
   927062 between 10 B and 100 B (total 73.4 MB)
 18537306 between 100 B and 1 KB (total 6.9 GB)
  8198209 between 1 KB and 10 KB (total 29 GB)
  9680197 between 10 KB and 100 KB (total 437.5 GB)
 22635108 between 100 KB and 1 MB (total 9.6 TB)
 66922938 between 1 MB and 10 MB (total 247.6 TB)
        0 between 10 MB and 100 MB (total 0 B)

Things seem to be OK for now, everything works (well, besides this weird corruption thing, unclear what caused it still - luckily it was easily fixable, once you know what is going on).

We will see how it goes, but so far I am very impressed by the design and engineering behind Kopia - it very elegant, feature-rich, efficient (multithreading everywhere, no hard locks and bottlenecks on repos/resources, with ‘eventually consistent’ model) and seem to scale well.

1 Like