Hello Kopia experts
I wish you all a happy new year.
Unfortunately I’ve had a small problem since yesterday and I’m hoping for help from the forum, because unfortunately I can’t get any further on my own.
First of all, I would like to point out that I am not dependent on the backup here and now. I am primarily interested in understanding Kopia better and building up the necessary trust to entrust Kopia with my backup.
About 6 weeks ago I also had a very positive experience when a hard disk failed and I was able to restore a very up-to-date version with Kopia.
The short version:
Recently (I noticed it yesterday ) creating a snapshot takes an unexpectedly long time. Data is being written / uploaded although (almost) nothing has changed in the source.
The command “kopia.exe maintenance run --full” returns the following result
Running full maintenance...
Looking for active contents...
ERROR error processing : error verifying k1bc588ce243fa76c8e0d33d1439848a8: error getting content info for k1bc588ce243fa76c8e0d33d1439848a8: content not found
Finished full maintenance.
ERROR snapshot GC failure: error running snapshot gc: unable to find in-use content ID: error processing snapshot root: error verifying k1bc588ce243fa76c8e0d33d1439848a8: error getting content info for k1bc588ce243fa76c8e0d33d1439848a8: content not found
My repository is now 7TB in size and the upload took several weeks. I would like to know whether I may have made a mistake here or whether Kopia caused the error in this scenario (RClone support is not yet fully stable).
I am using Kopia 0.18.2 and the repository was created with 0.17.
Perhaps you have some tips for me on what may have gone wrong here and how I can fix it.
For example, I have no idea which file in the repository stores the blob? k1bc588ce243fa76c8e0d33d1439848a8 and whether the file may no longer exist at all.
I also can’t rule out that the cloud provider has lost data (it doesn’t look like it, but since I don’t understand the extensive repository file structure with over 200,000 files, I don’t have an overview of it either.
I look forward to feedback to understand more about how it works.
Thanks in advance.
Update 3.1.2025
additional repository information
kopia content verify
Listing blobs...
10000 blobs...
20000 blobs...
30000 blobs...
40000 blobs...
50000 blobs...
60000 blobs...
70000 blobs...
80000 blobs...
90000 blobs...
100000 blobs...
110000 blobs...
120000 blobs...
130000 blobs...
140000 blobs...
150000 blobs...
160000 blobs...
170000 blobs...
180000 blobs...
190000 blobs...
200000 blobs...
210000 blobs...
220000 blobs...
Listed 227783 blobs.
Verifying all contents...
Verified 1 contents, 0 errors, estimating...
Finished verifying 43134 contents, found 0 errors.
kopia content stats
Count: 43134
Total Bytes: 38 GB
Total Packed: 31.1 GB (compression 18.3%)
By Method:
(uncompressed) count: 5160 size: 9.2 GB
zstd-better-compression count: 33696 size: 28.8 GB packed: 21.8 GB compression: 24.1%
zstd-fastest count: 4278 size: 17.6 MB packed: 5.1 MB compression: 71.2%
Average: 880.8 KB
Histogram:
0 between 0 B and 10 B (total 0 B)
228 between 10 B and 100 B (total 17.4 KB)
14536 between 100 B and 1 KB (total 6 MB)
5186 between 1 KB and 10 KB (total 19.7 MB)
5712 between 10 KB and 100 KB (total 235.5 MB)
5938 between 100 KB and 1 MB (total 2.2 GB)
11534 between 1 MB and 10 MB (total 28.6 GB)
0 between 10 MB and 100 MB (total 0 B)
kopia blob stats
Got 10000 blobs...
Got 20000 blobs...
Got 30000 blobs...
Got 40000 blobs...
Got 50000 blobs...
Got 60000 blobs...
Got 70000 blobs...
Got 80000 blobs...
Got 90000 blobs...
Got 100000 blobs...
Got 110000 blobs...
Got 120000 blobs...
Got 130000 blobs...
Got 140000 blobs...
Got 150000 blobs...
Got 160000 blobs...
Got 170000 blobs...
Got 180000 blobs...
Got 190000 blobs...
Got 200000 blobs...
Got 210000 blobs...
Got 220000 blobs...
Count: 227783
Total: 5.3 TB
Average: 23.1 MB
Histogram:
0 between 0 B and 10 B (total 0 B)
7 between 10 B and 100 B (total 114 B)
136 between 100 B and 1 KB (total 26.2 KB)
173 between 1 KB and 10 KB (total 679.8 KB)
245 between 10 KB and 100 KB (total 11.4 MB)
60 between 100 KB and 1 MB (total 21.4 MB)
67 between 1 MB and 10 MB (total 238.4 MB)
227095 between 10 MB and 100 MB (total 5.3 TB)
and the output from an larger maintenance job that cleaned up ~115GB a few days ago when everything was fine
kopia.exe maintenance run --full
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
GC found 95199 unused contents (4.5 GB)
GC found 4 unused contents that are too recent to delete (46.1 KB)
GC found 2454104 in-use contents (5 TB)
GC found 82 in-use system-contents (311.5 KB)
Previous content rewrite has not been finalized yet, waiting until the next blob deletion.
Found safe time to drop indexes: 2024-12-23 17:13:47.287199 +0100 CET
Dropping contents deleted before 2024-12-23 17:13:47.287199 +0100 CET
Looking for unreferenced blobs...
deleted 100 unreferenced blobs (2.4 GB)
deleted 200 unreferenced blobs (4.8 GB)
deleted 300 unreferenced blobs (7.2 GB)
deleted 400 unreferenced blobs (9.6 GB)
deleted 500 unreferenced blobs (12 GB)
deleted 600 unreferenced blobs (14.4 GB)
deleted 700 unreferenced blobs (16.8 GB)
deleted 800 unreferenced blobs (19.2 GB)
deleted 900 unreferenced blobs (21.6 GB)
deleted 1000 unreferenced blobs (24.1 GB)
deleted 1100 unreferenced blobs (26.5 GB)
deleted 1200 unreferenced blobs (28.9 GB)
deleted 1300 unreferenced blobs (31.3 GB)
deleted 1400 unreferenced blobs (33.7 GB)
deleted 1500 unreferenced blobs (36.1 GB)
deleted 1600 unreferenced blobs (38.5 GB)
deleted 1700 unreferenced blobs (40.9 GB)
deleted 1800 unreferenced blobs (43.3 GB)
deleted 1900 unreferenced blobs (45.7 GB)
deleted 2000 unreferenced blobs (48.1 GB)
deleted 2100 unreferenced blobs (50.6 GB)
deleted 2200 unreferenced blobs (53 GB)
deleted 2300 unreferenced blobs (55.4 GB)
deleted 2400 unreferenced blobs (57.8 GB)
deleted 2500 unreferenced blobs (60.2 GB)
deleted 2600 unreferenced blobs (62.6 GB)
deleted 2700 unreferenced blobs (65 GB)
deleted 2800 unreferenced blobs (67.4 GB)
deleted 2900 unreferenced blobs (69.8 GB)
deleted 3000 unreferenced blobs (72.3 GB)
deleted 3100 unreferenced blobs (74.6 GB)
deleted 3200 unreferenced blobs (77 GB)
deleted 3300 unreferenced blobs (79.5 GB)
deleted 3400 unreferenced blobs (81.9 GB)
deleted 3500 unreferenced blobs (84.3 GB)
deleted 3600 unreferenced blobs (86.7 GB)
deleted 3700 unreferenced blobs (89.1 GB)
deleted 3800 unreferenced blobs (91.5 GB)
deleted 3900 unreferenced blobs (93.9 GB)
deleted 4000 unreferenced blobs (96.3 GB)
deleted 4100 unreferenced blobs (98.8 GB)
deleted 4200 unreferenced blobs (101.2 GB)
deleted 4300 unreferenced blobs (103.6 GB)
deleted 4400 unreferenced blobs (106 GB)
deleted 4500 unreferenced blobs (108.5 GB)
deleted 4600 unreferenced blobs (110.9 GB)
deleted 4700 unreferenced blobs (113.3 GB)
deleted 4800 unreferenced blobs (115.7 GB)
Deleted total 4845 unreferenced blobs (116.4 GB)
Compacting an eligible uncompacted epoch...
Cleaning up no-longer-needed epoch markers...
Attempting to compact a range of epoch indexes ...
Cleaning up unneeded epoch markers...
Cleaning up old index blobs which have already been compacted...
Cleaned up 0 logs.
Finished full maintenance.
kopia maintenance info
Owner: Tester@desktop-9do5qn0
Quick Cycle:
scheduled: false
Full Cycle:
scheduled: true
interval: 168h0m0s
next run: 2025-01-08 19:28:43 CET (in 121h28m33s)
Log Retention:
max count: 10000
max age of logs: 720h0m0s
max total size: 1.1 GB
Object Lock Extension: disabled
Recent Maintenance Runs:
full-drop-deleted-content:
2025-01-01 03:03:40 CET (36s) SUCCESS
2024-12-24 22:20:08 CET (3s) SUCCESS
2024-12-23 18:14:13 CET (6s) SUCCESS
2024-12-23 18:02:46 CET (5s) SUCCESS
2024-12-22 16:30:01 CET (2s) SUCCESS
full-rewrite-contents:
2025-01-01 03:02:35 CET (1m4s) SUCCESS
2024-12-23 18:02:08 CET (35s) SUCCESS
2024-12-22 12:35:18 CET (32s) SUCCESS
2024-12-18 21:06:25 CET (42s) SUCCESS
2024-12-18 06:28:01 CET (2m55s) SUCCESS
snapshot-gc:
2025-01-01 19:28:58 CET (0s) ERROR: unable to find in-use content ID: error processing snapshot root: error verifying k1bc588ce243fa76c8e0d33d1439848a8: error getting content info for k1bc588ce243fa76c8e0d33d1439848a8: content not found
2025-01-01 18:08:58 CET (0s) ERROR: unable to find in-use content ID: error processing snapshot root: error verifying k1bc588ce243fa76c8e0d33d1439848a8: error getting content info for k1bc588ce243fa76c8e0d33d1439848a8: content not found
2025-01-01 18:02:01 CET (0s) ERROR: unable to find in-use content ID: error processing snapshot root: error verifying k1bc588ce243fa76c8e0d33d1439848a8: error getting content info for k1bc588ce243fa76c8e0d33d1439848a8: content not found
2025-01-01 03:00:11 CET (2m22s) SUCCESS
2024-12-24 22:19:35 CET (31s) SUCCESS
cleanup-logs:
2025-01-01 03:06:34 CET (0s) SUCCESS
2024-12-24 23:04:17 CET (0s) SUCCESS
2024-12-23 18:15:19 CET (0s) SUCCESS
2024-12-23 18:03:42 CET (0s) SUCCESS
2024-12-22 17:13:20 CET (0s) SUCCESS
compact-single-epoch:
2025-01-01 03:04:17 CET (28s) SUCCESS
2024-12-24 23:03:29 CET (3s) SUCCESS
2024-12-23 18:14:20 CET (13s) SUCCESS
2024-12-23 18:02:52 CET (0s) SUCCESS
2024-12-22 17:12:32 CET (2s) SUCCESS
full-delete-blobs:
2024-12-24 22:20:12 CET (43m15s) SUCCESS
2024-12-22 16:30:04 CET (42m26s) SUCCESS
2024-12-20 06:33:03 CET (42m59s) SUCCESS
2024-12-18 18:25:23 CET (43m26s) SUCCESS
2024-12-12 23:36:48 CET (42m55s) SUCCESS
generate-epoch-range-index:
2025-01-01 03:04:55 CET (46s) SUCCESS
2024-12-24 23:03:34 CET (0s) SUCCESS
2024-12-23 18:14:35 CET (0s) SUCCESS
2024-12-23 18:02:54 CET (0s) SUCCESS
2024-12-22 17:12:36 CET (0s) SUCCESS
advance-epoch:
2025-01-01 03:04:47 CET (7s) SUCCESS
2024-12-24 23:03:33 CET (0s) SUCCESS
2024-12-23 18:14:35 CET (0s) SUCCESS
2024-12-23 18:02:52 CET (0s) SUCCESS
2024-12-22 17:12:36 CET (0s) SUCCESS
cleanup-epoch-markers:
2025-01-01 03:05:42 CET (0s) SUCCESS
2024-12-24 23:03:34 CET (0s) SUCCESS
2024-12-23 18:14:36 CET (0s) SUCCESS
2024-12-23 18:02:55 CET (0s) SUCCESS
2024-12-22 17:12:37 CET (0s) SUCCESS
delete-superseded-epoch-indexes:
2025-01-01 03:05:42 CET (51s) SUCCESS
2024-12-24 23:03:35 CET (41s) SUCCESS
2024-12-23 18:14:36 CET (41s) SUCCESS
2024-12-23 18:02:55 CET (46s) SUCCESS
2024-12-22 17:12:37 CET (41s) SUCCESS
