Backup size much larger than source files

I am backing up some files to Onedrive using kopia with rclone repository. And I’ve found that the repo size on Onedrives side balloons really quickly, a folder of 600G locally is about 2T on the repo.

This is the initial snapshot with compression off

Can you do

kopia blob stat
kopia content stats
kopia blob ls

And paste results on some paste service and put the link here.

Since those commands are taking a long time, I might as well add that I tried adding the arguments --vfs-cache-mode=writes as extra args to rclone as recommended for onedrive and this new repo is now undersized compared to source by 10%ish, I think this is due to kopia killing the rclone process while the files in cache are still not completely sync. I think it might be required to use the remote control api to query the status and wait for it to finish before killing rclone.

Also the oversized repo does not list any snapshots despite running the snapshot successfully initially

I have a very similar issue with Azure Storage Explorer reporting 10.4GB and Kopia itself more or less confirming the figure via the commands listed earlier on this post.

However Windows Explorer reports 2.46GB for the target folders (actually a little less due to .kopiaignore).

Typically we only add new content, with the notable exception of SQL Server backups, unsure how the dedupe mechanism works in this regard.

Guidance on how to reduce would be very much appreciated.

Command outputs: Untitled 2021-10-19 16:24:55 - Share Text

I have the same problem with pgzip compression enabled:

local size 40GB
OneDrive size 80GB

This is after the initial backup and later didn’t change.

Here is the desired info:

kopia blob stats                                                                                                           
Count: 4199                                                                                                                                          
Total: 79.7 GB                                                                                                                                       
Average: 19 MB                                                                                                                                       
Histogram:                                                                                                                                           
                                                                                                                                                     
        0 between 0 B and 10 B (total 0 B)                                                                                                           
        4 between 10 B and 100 B (total 72 B)                                                                                                        
      111 between 100 B and 1 KB (total 18.3 KB)                                                                                                     
      264 between 1 KB and 10 KB (total 1.4 MB)                                                                                                      
      250 between 10 KB and 100 KB (total 11.1 MB)                                                                                                   
      104 between 100 KB and 1 MB (total 29.8 MB)                                                                                                    
       79 between 1 MB and 10 MB (total 369.6 MB)                                                                                                    
     3387 between 10 MB and 100 MB (total 79.3 GB)
kopia content stats                                                                                                        
Count: 65275                                                                                                                                         
Total Bytes: 40.2 GB                                                                                                                                 
Total Packed: 39 GB (compression 3.2%)                                                                                                               
By Method:                                                                                                                                           
  (uncompressed)         count: 19837 size: 27.5 GB                                                                                                  
  pgzip                  count: 32111 size: 12.7 GB packed: 11.5 GB compression: 9.9%                                                                
  zstd-fastest           count: 13327 size: 14.6 MB packed: 5.5 MB compression: 62.5%                                                                
Average: 616.3 KB                                                                                                                                    
Histogram:                                                                                                                                           
                                                                                                                                                     
        0 between 0 B and 10 B (total 0 B)                                                                                                           
      244 between 10 B and 100 B (total 17.1 KB)                                                                                                     
    21209 between 100 B and 1 KB (total 6.9 MB)                                                                                                      
    24214 between 1 KB and 10 KB (total 76.4 MB)                                                                                                     
     6893 between 10 KB and 100 KB (total 236 MB)                                                                                                    
     3903 between 100 KB and 1 MB (total 1.5 GB)                                                                                                     
     8812 between 1 MB and 10 MB (total 37.1 GB)                                                                                                     
        0 between 10 MB and 100 MB (total 0 B)
kopia blob ls 
gives a long list like this: 
kopia.blobcfg                                                                  30 2022-07-03 14:11:19 CEST                                           
kopia.maintenance                                                            1467 2022-07-06 15:12:54 CEST                                           
kopia.repository                                                             1101 2022-07-03 14:11:20 CEST                                           
xe2                                                                            12 2022-07-05 15:11:58 CEST                                           
xe3                                                                            12 2022-07-06 15:12:37 CEST                                           
xw1656846851                                                                   18 2022-07-04 15:12:08 CEST                                           
xs1_62716093e37f316ff7098a0b95d5d2d1-s568c2b9d063af6f2-c1                    5171 2022-07-06 15:13:24 CEST                                           
xs0_39707b2cc011886199a8fc048292f0ee-s8d002c3fdca9a6c3-c1                 2437059 2022-07-05 15:14:55 CEST                                           
xn3_f2a42c4f9cfd52a06056ca40968e98cd-s36aa10f14d3eb555112-c1                  143 2022-07-06 16:20:43 CEST                                           
xn3_cc46d6aeadce3ad73f1f4d85c74de016-s0ad1d02b4f9112a0112-c1                  143 2022-07-06 16:01:28 CEST                                           
xn3_93c4e7fc0312673b28990b13b9fa5e99-sbc2b92be58f733a6112-c1               154811 2022-07-06 15:38:48 CEST                                           
xn3_7b27696f3d429e44072bf43e266fda64-s598a5523ffa5416a112-c1                56943 2022-07-06 16:20:32 CEST
...
...
...
_log_20220706121108_e446_1657102268_1657102337_1_23f60117dbc3dfd0112750aeb6866705       6724 2022-07-06 12:12:17 CEST                                
_log_20220706131351_d961_1657106031_1657106167_1_8541c72a8f1ea659ec110757b6c38400       6977 2022-07-06 13:16:07 CEST                                
_log_20220706141106_be5f_1657109466_1657109508_1_0343f8b0ff5e984702367a8e84f0f5d5       7077 2022-07-06 14:11:49 CEST

I just wanted to add that kopia snapshot list now show 0B while just a few days ago it was actually showing 39GB or very close to it.

kopia snapshot list /sixer/Documents/                                                                                                                                                                                                                                                      
root@nas:/sixer/Documents                                                                                                                                                                                                                                                                                            
  2022-07-03 23:11:34 CEST k6fcc261e7792d1baea65ff635f54fa63 0 B drwxrwx--- files:44077 dirs:15940 (daily-4,weekly-2)                                                                                                                                                                                                
  2022-07-04 23:11:43 CEST kb783c491cdffe8996934c599aeb7b275 0 B drwxrwx--- files:44079 dirs:15944 (latest-1..3,hourly-1..18,daily-1..3,weekly-1,mont                                                                                                                                                                hly-1,annual-1)                                                                                                                                                                                                                                                                                                      
  + 19 identical snapshots until 2022-07-06 22:26:36 CEST

This is a known issue with the latest release of Kopia and has already been fixed (see fix(cli): fixed snapshot sizes in the snapshot list by jkowalski · Pull Request #2148 · kopia/kopia · GitHub). However, an updated Kopia has not been released yet with the fix. If this is important for you and you cannot wait, the latest test build has it: Releases · kopia/kopia-test-builds · GitHub

1 Like

@jkowalski Any further feedback or ideas?