I am backing up some files to Onedrive using kopia with rclone repository. And I’ve found that the repo size on Onedrives side balloons really quickly, a folder of 600G locally is about 2T on the repo.
This is the initial snapshot with compression off
Can you do
kopia blob stat
kopia content stats
kopia blob ls
And paste results on some paste service and put the link here.
Since those commands are taking a long time, I might as well add that I tried adding the arguments --vfs-cache-mode=writes as extra args to rclone as recommended for onedrive and this new repo is now undersized compared to source by 10%ish, I think this is due to kopia killing the rclone process while the files in cache are still not completely sync. I think it might be required to use the remote control api to query the status and wait for it to finish before killing rclone.
Also the oversized repo does not list any snapshots despite running the snapshot successfully initially
I have a very similar issue with Azure Storage Explorer reporting 10.4GB and Kopia itself more or less confirming the figure via the commands listed earlier on this post.
However Windows Explorer reports 2.46GB for the target folders (actually a little less due to .kopiaignore).
Typically we only add new content, with the notable exception of SQL Server backups, unsure how the dedupe mechanism works in this regard.
Guidance on how to reduce would be very much appreciated.
Command outputs: Untitled 2021-10-19 16:24:55 - Share Text
I have the same problem with pgzip compression enabled:
local size 40GB
OneDrive size 80GB
This is after the initial backup and later didn’t change.
Here is the desired info:
kopia blob stats
Count: 4199
Total: 79.7 GB
Average: 19 MB
Histogram:
0 between 0 B and 10 B (total 0 B)
4 between 10 B and 100 B (total 72 B)
111 between 100 B and 1 KB (total 18.3 KB)
264 between 1 KB and 10 KB (total 1.4 MB)
250 between 10 KB and 100 KB (total 11.1 MB)
104 between 100 KB and 1 MB (total 29.8 MB)
79 between 1 MB and 10 MB (total 369.6 MB)
3387 between 10 MB and 100 MB (total 79.3 GB)
kopia content stats
Count: 65275
Total Bytes: 40.2 GB
Total Packed: 39 GB (compression 3.2%)
By Method:
(uncompressed) count: 19837 size: 27.5 GB
pgzip count: 32111 size: 12.7 GB packed: 11.5 GB compression: 9.9%
zstd-fastest count: 13327 size: 14.6 MB packed: 5.5 MB compression: 62.5%
Average: 616.3 KB
Histogram:
0 between 0 B and 10 B (total 0 B)
244 between 10 B and 100 B (total 17.1 KB)
21209 between 100 B and 1 KB (total 6.9 MB)
24214 between 1 KB and 10 KB (total 76.4 MB)
6893 between 10 KB and 100 KB (total 236 MB)
3903 between 100 KB and 1 MB (total 1.5 GB)
8812 between 1 MB and 10 MB (total 37.1 GB)
0 between 10 MB and 100 MB (total 0 B)
kopia blob ls
gives a long list like this:
kopia.blobcfg 30 2022-07-03 14:11:19 CEST
kopia.maintenance 1467 2022-07-06 15:12:54 CEST
kopia.repository 1101 2022-07-03 14:11:20 CEST
xe2 12 2022-07-05 15:11:58 CEST
xe3 12 2022-07-06 15:12:37 CEST
xw1656846851 18 2022-07-04 15:12:08 CEST
xs1_62716093e37f316ff7098a0b95d5d2d1-s568c2b9d063af6f2-c1 5171 2022-07-06 15:13:24 CEST
xs0_39707b2cc011886199a8fc048292f0ee-s8d002c3fdca9a6c3-c1 2437059 2022-07-05 15:14:55 CEST
xn3_f2a42c4f9cfd52a06056ca40968e98cd-s36aa10f14d3eb555112-c1 143 2022-07-06 16:20:43 CEST
xn3_cc46d6aeadce3ad73f1f4d85c74de016-s0ad1d02b4f9112a0112-c1 143 2022-07-06 16:01:28 CEST
xn3_93c4e7fc0312673b28990b13b9fa5e99-sbc2b92be58f733a6112-c1 154811 2022-07-06 15:38:48 CEST
xn3_7b27696f3d429e44072bf43e266fda64-s598a5523ffa5416a112-c1 56943 2022-07-06 16:20:32 CEST
...
...
...
_log_20220706121108_e446_1657102268_1657102337_1_23f60117dbc3dfd0112750aeb6866705 6724 2022-07-06 12:12:17 CEST
_log_20220706131351_d961_1657106031_1657106167_1_8541c72a8f1ea659ec110757b6c38400 6977 2022-07-06 13:16:07 CEST
_log_20220706141106_be5f_1657109466_1657109508_1_0343f8b0ff5e984702367a8e84f0f5d5 7077 2022-07-06 14:11:49 CEST
I just wanted to add that kopia snapshot list now show 0B while just a few days ago it was actually showing 39GB or very close to it.
kopia snapshot list /sixer/Documents/
root@nas:/sixer/Documents
2022-07-03 23:11:34 CEST k6fcc261e7792d1baea65ff635f54fa63 0 B drwxrwx--- files:44077 dirs:15940 (daily-4,weekly-2)
2022-07-04 23:11:43 CEST kb783c491cdffe8996934c599aeb7b275 0 B drwxrwx--- files:44079 dirs:15944 (latest-1..3,hourly-1..18,daily-1..3,weekly-1,mont hly-1,annual-1)
+ 19 identical snapshots until 2022-07-06 22:26:36 CEST
This is a known issue with the latest release of Kopia and has already been fixed (see fix(cli): fixed snapshot sizes in the snapshot list by jkowalski · Pull Request #2148 · kopia/kopia · GitHub). However, an updated Kopia has not been released yet with the fix. If this is important for you and you cannot wait, the latest test build has it: Releases · kopia/kopia-test-builds · GitHub