Kopia repository in Backblaze B2 takes up 38.2GB instead of 24.6GB

Support
1

I’ve deleted all previous snapshots.

I ran sudo kopia maintenance run --full --safety=none multiple times.

But, the size of the repository on Backblaze B2 is larger than it should be. The bucket is 38.2GB.

Running sudo kopia content stats yields:

Count: 365524
Total Bytes: 24.6 GB
Total Packed: 24.5 GB (compression 0.3%)
By Method:
(uncompressed) count: 305917 size: 24.5 GB
zstd-fastest count: 59607 size: 107.3 MB packed: 33.3 MB compression: 68.9%
Average: 67.2 KB
Histogram:
0 between 0 B and 10 B (total 0 B)
29100 between 10 B and 100 B (total 2.1 MB)
131659 between 100 B and 1 KB (total 56.9 MB)
127299 between 1 KB and 10 KB (total 423.6 MB)
58186 between 10 KB and 100 KB (total 1.8 GB)
12192 between 100 KB and 1 MB (total 3.4 GB)
7088 between 1 MB and 10 MB (total 18.8 GB)
0 between 10 MB and 100 MB (total 0 B)

2

Make sure you change the bucket lifecycle settings to “Keep only the last version of the file”:

image
image1136×880 48.5 KB

3

Thanks for the assist.

But, even this isn’t helping.

The size of the repository just keeps growing.

I turned on Keep only the last version of the file on Backblaze, and despite waiting, it still hasn’t deleted what I presume are the older versions of these files?

4

A few other things:

  • check you don’t have any old snapshots (I initially backed up a couple of large folders that I then excluded from backup - I had to go back and delete those old snapshots)
  • did you wait enough time for B2 to delete the old versions? For me it seemed to take a day or two
  • and finally, Kopia is extremely conservative about deleting old objects/blobs (I forget what they’re called) so it can take a few days for the size of the B2 bucket to drop as Kopia performs housekeeping on the repository. Look in the Kopia logs and you might see messages about it not deleting stuff because insufficient time has elapsed.
5

Wow. I thought it would be done in 10 minutes.

I’ll wait till tomorrow, I guess.

6

As I stated in the original post:

7

Waiting fixed the issue.

8

Also note that Backblaze lifecycle rules take 24 to 48 hours to kick in. Acording to this blog https://blog.tymscar.com/posts/backblazeb2cleanup/

In my case, after I changed Lifecycle setting from: “Keep all versions” to “Keep only the last version” take 24 hrs to remove old versions, and the reduce the space

Commands to check space used:

# Get file count and space used by bucket
b2 bucket get mybucket --show-size | grep -iE “bucketname|totalsize|filecount”
    "bucketName": "mybucket",
    "fileCount": 42710,
    "totalSize": 30,284,830,374

# List current files
b2 ls -r -l  “b2://mybucket/” | awk ‘{sum += $5} END {printf “%.0f\n”, sum}’
4,181,535,577

# List all versions and deleted/hidden files
b2 ls -r -l --versions  “b2://mybucket/” | awk ‘{sum += $5} END {printf “%\047.0f\n”, sum}’
30,284,830,374

After change lifecycle policy AND WAIT 24 hours, the space was reduced to the actual files in kopia repository

b2 bucket get mybucket --show-size | grep -iE "bucketname|totalsize|filecount"
    "bucketName": "mybucket",
    "fileCount": 1461,
    "totalSize": 4907139347

b2 ls -l --versions  "b2://mybucket/" | wc -l
1461

b2 ls -l --versions "b2://mybucket/" | awk '{sum += $5} END {printf "%\047.0f\n", sum}'
4,907,139,347

Very close to kopia stats

kopia blob stats
Count: 669
Total: 4.2 GB
Average: 6.3 MB
Histogram:

        0 between 0 B and 10 B (total 0 B)
        4 between 10 B and 100 B (total 135 B)
        9 between 100 B and 1 KB (total 3.5 KB)
       87 between 1 KB and 10 KB (total 466.4 KB)
       39 between 10 KB and 100 KB (total 1.4 MB)
      331 between 100 KB and 1 MB (total 194 MB)
        7 between 1 MB and 10 MB (total 26.7 MB)
      192 between 10 MB and 100 MB (total 4 GB)

So my space went from 30G to 4.9 G because I have snapshots since 2023, but no useful anymore, I just keep 21 daily and 3 monthly snapshots

kopia policy show --global
Policy for (global):

Retention:
  Annual snapshots:                        0   (defined for this target)
  Monthly snapshots:                       3   (defined for this target)
  Weekly snapshots:                        0   (defined for this target)
  Daily snapshots:                        21   (defined for this target)
  Hourly snapshots:                        0   (defined for this target)
  Latest snapshots:                       21   (defined for this target)
  Ignore identical snapshots:          false   (defined for this target)
...