I am backing up some files to Onedrive using kopia with rclone repository. And I’ve found that the repo size on Onedrives side balloons really quickly, a folder of 600G locally is about 2T on the repo.
This is the initial snapshot with compression off
Can you do
kopia blob stat
kopia content stats
kopia blob ls
And paste results on some paste service and put the link here.
Since those commands are taking a long time, I might as well add that I tried adding the arguments --vfs-cache-mode=writes as extra args to rclone as recommended for onedrive and this new repo is now undersized compared to source by 10%ish, I think this is due to kopia killing the rclone process while the files in cache are still not completely sync. I think it might be required to use the remote control api to query the status and wait for it to finish before killing rclone.
Also the oversized repo does not list any snapshots despite running the snapshot successfully initially
I have a very similar issue with Azure Storage Explorer reporting 10.4GB and Kopia itself more or less confirming the figure via the commands listed earlier on this post.
However Windows Explorer reports 2.46GB for the target folders (actually a little less due to .kopiaignore).
Typically we only add new content, with the notable exception of SQL Server backups, unsure how the dedupe mechanism works in this regard.
Guidance on how to reduce would be very much appreciated.
Command outputs: Untitled 2021-10-19 16:24:55 - Share Text
I have the same problem with pgzip compression enabled:
local size 40GB
OneDrive size 80GB
This is after the initial backup and later didn’t change.
Here is the desired info:
kopia blob stats
Count: 4199
Total: 79.7 GB
Average: 19 MB
Histogram:
0 between 0 B and 10 B (total 0 B)
4 between 10 B and 100 B (total 72 B)
111 between 100 B and 1 KB (total 18.3 KB)
264 between 1 KB and 10 KB (total 1.4 MB)
250 between 10 KB and 100 KB (total 11.1 MB)
104 between 100 KB and 1 MB (total 29.8 MB)
79 between 1 MB and 10 MB (total 369.6 MB)
3387 between 10 MB and 100 MB (total 79.3 GB)
kopia content stats
Count: 65275
Total Bytes: 40.2 GB
Total Packed: 39 GB (compression 3.2%)
By Method:
(uncompressed) count: 19837 size: 27.5 GB
pgzip count: 32111 size: 12.7 GB packed: 11.5 GB compression: 9.9%
zstd-fastest count: 13327 size: 14.6 MB packed: 5.5 MB compression: 62.5%
Average: 616.3 KB
Histogram:
0 between 0 B and 10 B (total 0 B)
244 between 10 B and 100 B (total 17.1 KB)
21209 between 100 B and 1 KB (total 6.9 MB)
24214 between 1 KB and 10 KB (total 76.4 MB)
6893 between 10 KB and 100 KB (total 236 MB)
3903 between 100 KB and 1 MB (total 1.5 GB)
8812 between 1 MB and 10 MB (total 37.1 GB)
0 between 10 MB and 100 MB (total 0 B)
kopia blob ls
gives a long list like this:
kopia.blobcfg 30 2022-07-03 14:11:19 CEST
kopia.maintenance 1467 2022-07-06 15:12:54 CEST
kopia.repository 1101 2022-07-03 14:11:20 CEST
xe2 12 2022-07-05 15:11:58 CEST
xe3 12 2022-07-06 15:12:37 CEST
xw1656846851 18 2022-07-04 15:12:08 CEST
xs1_62716093e37f316ff7098a0b95d5d2d1-s568c2b9d063af6f2-c1 5171 2022-07-06 15:13:24 CEST
xs0_39707b2cc011886199a8fc048292f0ee-s8d002c3fdca9a6c3-c1 2437059 2022-07-05 15:14:55 CEST
xn3_f2a42c4f9cfd52a06056ca40968e98cd-s36aa10f14d3eb555112-c1 143 2022-07-06 16:20:43 CEST
xn3_cc46d6aeadce3ad73f1f4d85c74de016-s0ad1d02b4f9112a0112-c1 143 2022-07-06 16:01:28 CEST
xn3_93c4e7fc0312673b28990b13b9fa5e99-sbc2b92be58f733a6112-c1 154811 2022-07-06 15:38:48 CEST
xn3_7b27696f3d429e44072bf43e266fda64-s598a5523ffa5416a112-c1 56943 2022-07-06 16:20:32 CEST
...
...
...
_log_20220706121108_e446_1657102268_1657102337_1_23f60117dbc3dfd0112750aeb6866705 6724 2022-07-06 12:12:17 CEST
_log_20220706131351_d961_1657106031_1657106167_1_8541c72a8f1ea659ec110757b6c38400 6977 2022-07-06 13:16:07 CEST
_log_20220706141106_be5f_1657109466_1657109508_1_0343f8b0ff5e984702367a8e84f0f5d5 7077 2022-07-06 14:11:49 CEST
I just wanted to add that kopia snapshot list now show 0B while just a few days ago it was actually showing 39GB or very close to it.
kopia snapshot list /sixer/Documents/
root@nas:/sixer/Documents
2022-07-03 23:11:34 CEST k6fcc261e7792d1baea65ff635f54fa63 0 B drwxrwx--- files:44077 dirs:15940 (daily-4,weekly-2)
2022-07-04 23:11:43 CEST kb783c491cdffe8996934c599aeb7b275 0 B drwxrwx--- files:44079 dirs:15944 (latest-1..3,hourly-1..18,daily-1..3,weekly-1,mont hly-1,annual-1)
+ 19 identical snapshots until 2022-07-06 22:26:36 CEST
This is a known issue with the latest release of Kopia and has already been fixed (see fix(cli): fixed snapshot sizes in the snapshot list by jkowalski · Pull Request #2148 · kopia/kopia · GitHub). However, an updated Kopia has not been released yet with the fix. If this is important for you and you cannot wait, the latest test build has it: Releases · kopia/kopia-test-builds · GitHub
@jkowalski Any further feedback or ideas?
I have similar issue when I use pcloud as backup storage.
Content size are 14.2 GB, but blob size are 842.3 GB…
How can I clean up those blobs?
kopia blob stats
Got 10000 blobs...
Got 20000 blobs...
Got 30000 blobs...
Got 40000 blobs...
Count: 43473
Total: 842.3 GB
Average: 19.4 MB
Histogram:
0 between 0 B and 10 B (total 0 B)
4 between 10 B and 100 B (total 72 B)
3254 between 100 B and 1 KB (total 630.3 KB)
2055 between 1 KB and 10 KB (total 8.1 MB)
761 between 10 KB and 100 KB (total 21.3 MB)
315 between 100 KB and 1 MB (total 109.5 MB)
83 between 1 MB and 10 MB (total 404.2 MB)
37001 between 10 MB and 100 MB (total 841.8 GB)
kopia content stats
Count: 86748
Total Bytes: 14.2 GB
Total Packed: 14.1 GB (compression 0.2%)
By Method:
(uncompressed) count: 80135 size: 14.1 GB
zstd-fastest count: 6613 size: 42.6 MB packed: 13.9 MB compression: 67.4%
Average: 163.2 KB
Histogram:
0 between 0 B and 10 B (total 0 B)
1651 between 10 B and 100 B (total 88.5 KB)
12720 between 100 B and 1 KB (total 4.6 MB)
28176 between 1 KB and 10 KB (total 147.7 MB)
30165 between 10 KB and 100 KB (total 1 GB)
10567 between 100 KB and 1 MB (total 3.3 GB)
3469 between 1 MB and 10 MB (total 9.7 GB)
0 between 10 MB and 100 MB (total 0 B)
Answer myself, when I execute kopia maintenance info, it returns recent maintenance run, but for some reason, my recent run already 4 months ago…
kopia maintenance info
Owner: Allen Lee@desktop
Quick Cycle:
scheduled: true
interval: 1h0m0s
next run: now
Full Cycle:
scheduled: true
interval: 24h0m0s
next run: now
Log Retention:
max count: 10000
max age of logs: 720h0m0s
max total size: 1 GiB
Recent Maintenance Runs:
snapshot-gc:
2022-07-26 21:52:44 CST (7s) SUCCESS
2022-07-25 21:04:55 CST (2s) SUCCESS
2022-07-24 21:04:55 CST (3s) SUCCESS
2022-07-23 21:04:55 CST (3s) SUCCESS
2022-07-22 21:04:54 CST (5s) SUCCESS
cleanup-epoch-manager:
2022-07-26 21:53:16 CST (1s) SUCCESS
2022-07-25 21:05:03 CST (1s) SUCCESS
2022-07-24 21:05:37 CST (3s) SUCCESS
2022-07-23 21:05:14 CST (1s) SUCCESS
2022-07-22 21:05:40 CST (2s) SUCCESS
cleanup-logs:
2022-07-26 21:53:16 CST (0s) SUCCESS
2022-07-25 21:05:03 CST (0s) SUCCESS
2022-07-24 21:05:37 CST (0s) SUCCESS
2022-07-23 21:05:14 CST (0s) SUCCESS
2022-07-22 21:05:40 CST (0s) SUCCESS
full-delete-blobs:
2022-07-26 21:52:52 CST (23s) SUCCESS
2022-07-24 21:04:58 CST (38s) SUCCESS
2022-07-22 21:05:00 CST (40s) SUCCESS
2022-07-20 21:04:57 CST (19s) SUCCESS
2022-07-18 21:04:57 CST (20s) SUCCESS
full-drop-deleted-content:
2022-07-26 21:52:52 CST (0s) SUCCESS
2022-07-25 21:05:03 CST (0s) SUCCESS
2022-07-24 21:04:58 CST (0s) SUCCESS
2022-07-23 21:05:14 CST (0s) SUCCESS
2022-07-22 21:05:00 CST (0s) SUCCESS
full-rewrite-contents:
2022-07-25 21:04:58 CST (4s) SUCCESS
2022-07-23 21:04:58 CST (15s) SUCCESS
2022-07-21 21:04:59 CST (5s) SUCCESS
2022-07-19 21:04:57 CST (2s) SUCCESS
2022-07-17 21:04:59 CST (15s) SUCCESS
When I execute kopia maintenance run --full, it’ll return some error, and looks like it doesn’t have successful run log recorded in kopia maintenance info:
kopia maintenance run --full
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
GC found 11525 unused contents (8.7 GiB)
GC found 4688 unused contents that are too recent to delete (1.5 GiB)
GC found 82070 in-use contents (11.7 GiB)
GC found 2065 in-use system-contents (4.4 MiB)
Rewriting contents from short packs...
unable to rewrite content "04e58e53eb14e265f3ff303a5cbafd46": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
unable to rewrite content "117632e5664b25f0bc073abef19c6d1b": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
unable to rewrite content "2716db45db19a62483a481e8b7b04f16": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
unable to rewrite content "2839d4823c442b1d46a65af984039e9a": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
unable to rewrite content "afe7744ba8bb611f61228751816ba334": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
unable to rewrite content "bf61c61179fab885d22d8f553b84b0ff": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
unable to rewrite content "e184d17776aaf6b2c5cf9f1225901dca": unable to get content data and info: error getting cached content: failed to get blob with ID p6c0e47bdb1a0237be2fa6b133a398955-sdc8eda2055865a26112: BLOB not found
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 7 contents
Finally, When I execute kopia blob gc --advanced-commands=enabled --delete=yes, it removes all unreferenced blobs:
kopia blob gc --advanced-commands=enabled --delete=yes
Looking for unreferenced blobs...
deleted 100 unreferenced blobs (2.3 GB)
deleted 200 unreferenced blobs (4.5 GB)
deleted 300 unreferenced blobs (6.8 GB)
deleted 400 unreferenced blobs (9.1 GB)
.......
deleted 37700 unreferenced blobs (801.3 GB)
deleted 37800 unreferenced blobs (801.3 GB)
deleted 37900 unreferenced blobs (801.3 GB)
deleted 38000 unreferenced blobs (801.3 GB)
Deleted total 38038 unreferenced blobs (801.3 GB)
Now the space it consumed looks normal:
kopia blob stats
Count: 5847
Total: 47.9 GB
Average: 8.2 MB
Histogram:
0 between 0 B and 10 B (total 0 B)
4 between 10 B and 100 B (total 72 B)
1772 between 100 B and 1 KB (total 319.5 KB)
1234 between 1 KB and 10 KB (total 4.5 MB)
496 between 10 KB and 100 KB (total 15.3 MB)
197 between 100 KB and 1 MB (total 63.9 MB)
26 between 1 MB and 10 MB (total 101.3 MB)
2118 between 10 MB and 100 MB (total 47.8 GB)
kopia content stats
Count: 82084
Total Bytes: 12.5 GB
Total Packed: 12.5 GB (compression 0.1%)
By Method:
(uncompressed) count: 78462 size: 12.5 GB
zstd-fastest count: 3622 size: 15.2 MB packed: 5 MB compression: 67.2%
Average: 152.6 KB
Histogram:
0 between 0 B and 10 B (total 0 B)
1650 between 10 B and 100 B (total 88.4 KB)
10068 between 100 B and 1 KB (total 3.7 MB)
27393 between 1 KB and 10 KB (total 142.5 MB)
29434 between 10 KB and 100 KB (total 0.9 GB)
10452 between 100 KB and 1 MB (total 3.2 GB)
3087 between 1 MB and 10 MB (total 8.2 GB)
0 between 10 MB and 100 MB (total 0 B)
I have similar issue with kopia version v0.12.1 when using onedrive as backup storage.