Compression on client side

General Topics
1

Hello,

Can I enable compress on client side using repository server?

kopia snap create /data/srv/
Snapshotting root@kopia:/data/srv ...
 * 0 hashing, 1242038 hashed (146 GB), 0 cached (0 B), uploaded 112.4 GB, estimated 146 GB (100.0%) 0s left
Created snapshot with root k266c8610d86b31326740c5698c29865f and ID 0eba4a640b3de9162f5e921fdf9fda0b in 30m57s

compare with restic

restic --cacert pub.key -r rest:xxx backup /srv/ --read-concurrency=4
enter password for repository:
repository c6c2e21b opened (version 2, compression level auto)
no parent snapshot found, will read all files
[0:00] 100.00%  1 / 1 index files loaded

Files:       1270635 new,     0 changed,     0 unmodified
Dirs:        32353 new,     0 changed,     0 unmodified
Added to the repository: 94.476 GiB (9.742 GiB stored)

processed 1270635 files, 136.206 GiB in 3:11
snapshot 47fd7ca6 saved
2

Yes, compression can be enabled regardless of the backend you are using. Compression settings are saved in your global/host/path policies.

3

compression is enabled on repo server.

And repo size only ~10 GB, (like restic repo)
but restic send to repo only 9.7 compressed data (dedup, compress, send)

kopia dedup, send data, and compress data on repo server side?

kopia content stats
Count: 249011
Total Bytes: 101.1 GB
Total Packed: 10.4 GB (compression 89.7%)
By Method:
  (uncompressed)         count: 83022 size: 3.9 GB
  zstd                   count: 137888 size: 97 GB packed: 6.4 GB compression: 93.4%
  zstd-fastest           count: 28101 size: 183.2 MB packed: 33.3 MB compression: 81.9%
Average: 406.2 KB

and restic

[0:00] 100.00%  2 / 2 index files loaded
scanning...
Stats in raw-data mode:
     Snapshots processed:  1
        Total Blob Count:  262099
 Total Uncompressed Size:  94.486 GiB
              Total Size:  9.693 GiB
    Compression Progress:  100.00%
       Compression Ratio:  9.75x
Compression Space Saving:  89.74%
4

Sorry, but I don’t understand what you are trying to say by this.

The client is doing all the chunking, compression and encryption.

5

I make snap to kopia repo server, client sent to repo server 112 GB, on repo server, disk usage only ~10GB, so kopia work like this: dedup → send to repo → compres on repo side.

6

I have never tested this in detail, but I highly doubt that. Kopia supports “passive” storage backends like SFTP. Compression would not work on such backends.

7

Have you actually monitored the transfer? My guess is, that you are mislead by the snapshot’s output… “sent” vs. “transferred”. A successful snapshot will always “send” the whole size to the repo, while it may only transfer a small amount of that data.

1 Like
8

Yes, of course I monitor traffic with iftop )

iftop ~112 GB sent
and kopia says 112 GB upload

* 0 hashing, 1242038 hashed (146 GB), 0 cached (0 B), uploaded 112.4 GB, estimated 146 GB (100.0%) 0s left