2x the 'Total Bytes" reported by 'kopia content stats' vs B2 destination

[bbccdd]

Hi all. Backblaze reports 2.6 TB in use for the bucket that contains a repo which reports 1.3 TB.

I had expected the numbers to be closer to each other.

I am using ransomware protection implemented as documented and a 90 day object lock backblaze backend using the S3 api with a kopia repository server in front of it. However not much changes in the underlaying data that could account for the difference.

Where should I troubleshoot?

Details:

$ kopia content stats | head -n3
Count: 955900
Total Bytes: 1.3 TB
Total Packed: 1.2 TB (compression 4.6%)

$ kopia repository status
Config file: /[…]/repository.config

Description: Repository in S3 bucket1
Hostname: […]
Username: […]
Read-only: false
Format blob cache: 15m0s

Storage type: s3
Storage capacity: unbounded
Storage config: {
“bucket”: “bucket1”,
“endpoint”: “s3.eu-central-003.backblazeb2.com”,
“accessKeyID”: “[…]”,
“secretAccessKey”: “*******************************”,
“sessionToken”: “”
}

Unique ID: […]
Hash: BLAKE3-256
Encryption: AES256-GCM-HMAC-SHA256
Splitter: DYNAMIC-4M-BUZHASH
Format version: 3
Content compression: true
Password changes: true
Max pack length: 21 MB
Index Format: v2

Epoch Manager: enabled
Current Epoch: 60

Epoch refresh frequency: 20m0s
Epoch advance on: 20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin: 4h0m0s
Epoch checkpoint every: 7 epochs

Blob retention mode: COMPLIANCE
Blob retention period: 2160h0m0s

Related but unanswered post: 'Kopia content stats' total bytes doesn't match what backblaze reports

Backblaze config:

$ kopia content verify
Listing blobs…
10000 blobs…
20000 blobs…
[…]
Listed 96407 blobs.
Verifying all contents…
Verified 1 contents, 0 errors, estimating…
Verified 555179 contents, 0 errors, estimating…
Finished verifying 955902 contents, found 0 errors.

$ kopia snapshot verify --file-parallelism=16 --parallel=16 --verify-files-percent=1.0
Listing blobs…
10000 blobs…
[…]
Listed 96407 blobs.
Processed 0 objects.
Processed 3544 objects.
[…]
Finished processing 659894 objects.

$ kopia maintenance info
Owner: […]
Quick Cycle:
scheduled: true
interval: 1h0m0s
next run: 2024-11-21 13:46:33 CET (in 24m6s)
Full Cycle:
scheduled: true
interval: 24h0m0s
next run: 2024-11-21 14:46:25 CET (in 1h23m59s)
Log Retention:
max count: 10000
max age of logs: 720h0m0s
max total size: 1.1 GB
Object Lock Extension: enabled
Recent Maintenance Runs:
full-rewrite-contents:
2024-11-20 14:48:46 CET (7m51s) SUCCESS
2024-11-18 14:50:00 CET (7m58s) SUCCESS
2024-11-17 01:04:12 CET (12m7s) SUCCESS
2024-11-15 01:04:21 CET (13m13s) SUCCESS
2024-11-13 01:03:13 CET (9m16s) SUCCESS
generate-epoch-range-index:
2024-11-20 14:57:48 CET (0s) SUCCESS
2024-11-19 14:50:31 CET (0s) SUCCESS
2024-11-18 15:00:00 CET (0s) SUCCESS
2024-08-17 01:36:13 CEST (0s) SUCCESS
2024-08-16 01:52:40 CEST (0s) SUCCESS
cleanup-logs:
2024-11-20 14:57:56 CET (1s) SUCCESS
2024-11-19 14:50:38 CET (1s) SUCCESS
2024-11-18 15:00:08 CET (59s) SUCCESS
2024-08-17 01:36:17 CEST (1s) SUCCESS
2024-08-16 01:52:47 CEST (1s) SUCCESS
extend-blob-retention-time:
2024-11-20 14:56:42 CET (1m5s) SUCCESS
2024-11-19 14:49:25 CET (1m5s) SUCCESS
2024-11-18 14:58:04 CET (1m54s) SUCCESS
2024-11-18 01:03:52 CET (2m5s) SUCCESS
2024-11-17 01:16:40 CET (2m22s) SUCCESS
compact-single-epoch:
2024-11-21 12:46:41 CET (0s) SUCCESS
2024-11-21 11:46:40 CET (0s) SUCCESS
2024-11-21 10:46:39 CET (0s) SUCCESS
2024-11-21 09:46:39 CET (0s) SUCCESS
2024-11-21 08:46:38 CET (0s) SUCCESS
delete-superseded-epoch-indexes:
2024-11-20 14:57:48 CET (7s) SUCCESS
2024-11-19 14:50:32 CET (6s) SUCCESS
2024-11-18 15:00:01 CET (7s) SUCCESS
2024-08-17 01:36:13 CEST (3s) SUCCESS
2024-08-16 01:52:43 CEST (3s) SUCCESS
full-delete-blobs:
2024-11-19 14:49:03 CET (21s) SUCCESS
2024-11-18 01:03:29 CET (22s) SUCCESS
2024-11-16 01:04:02 CET (24s) SUCCESS
2024-11-14 01:03:59 CET (25s) SUCCESS
2024-11-11 09:15:04 CET (18s) SUCCESS
full-drop-deleted-content:
2024-11-20 14:56:38 CET (4s) SUCCESS
2024-11-19 14:48:58 CET (3s) SUCCESS
2024-11-18 14:57:59 CET (5s) SUCCESS
2024-11-18 01:03:24 CET (4s) SUCCESS
2024-11-17 01:16:20 CET (20s) SUCCESS
snapshot-gc:
2024-11-20 14:46:33 CET (2m13s) SUCCESS
2024-11-19 14:46:31 CET (2m27s) SUCCESS
2024-11-18 14:46:30 CET (3m29s) SUCCESS
2024-11-18 01:00:25 CET (2m58s) SUCCESS
2024-11-17 01:00:34 CET (3m35s) SUCCESS
advance-epoch:
2024-11-21 12:46:41 CET (0s) SUCCESS
2024-11-21 11:46:40 CET (0s) SUCCESS
2024-11-21 10:46:39 CET (0s) SUCCESS
2024-11-21 09:46:39 CET (0s) SUCCESS
2024-11-21 08:46:39 CET (0s) SUCCESS
cleanup-epoch-markers:
2024-11-20 14:57:48 CET (0s) SUCCESS
2024-11-19 14:50:31 CET (0s) SUCCESS
2024-11-18 15:00:00 CET (0s) SUCCESS
2024-08-17 01:36:13 CEST (0s) SUCCESS
2024-08-16 01:52:43 CEST (0s) SUCCESS

$ kopia maintenance run --full
Running full maintenance…
Looking for active contents…
Looking for unreferenced contents…
GC found 2889 unused contents (1.8 GB)
GC found 30055 unused contents that are too recent to delete (4.9 GB)
GC found 949839 in-use contents (1.2 TB)
GC found 834 in-use system-contents (30.5 MB)
Previous content rewrite has not been finalized yet, waiting until the next blob
Found safe time to drop indexes: 2024-11-20 13:46:33.385408686 +0100 CET
Dropping contents deleted before 2024-11-20 13:46:33.385408686 +0100 CET
Looking for unreferenced blobs…
deleted 100 unreferenced blobs (1.8 GB)
deleted 200 unreferenced blobs (3.3 GB)
[…]
Deleted total 1763 unreferenced blobs (13.3 GB)
Extending retention time for blobs…
extended 100 blobs
extended 200 blobs
[…]
Found 93954 blobs to extend
Extended total 93954 blobs
Compacting an eligible uncompacted epoch…
Cleaning up no-longer-needed epoch markers…
Attempting to compact a range of epoch indexes …
Cleaning up unneeded epoch markers…
Cleaning up old index blobs which have already been compacted…
Cleaned up 72 logs.
Finished full maintenance.

$ kopia --version
0.18.2 build: c70f1a1c1164ee8676f85f9a1cea6de0a782a3ae from: kopia/kopia