# Deduplication and compression

**URL:** <https://kopia.discourse.group/t/deduplication-and-compression/714>\
**Category:** General\
**Created:** [November 2, 2021, 10:14pm UTC](https://kopia.discourse.group/t/deduplication-and-compression/714 "2021-11-02T22:14:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![koala](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/koala/32/249_2.png) [@koala](https://kopia.discourse.group/u/koala)\
**Post date:** [November 2, 2021, 10:14pm UTC](https://kopia.discourse.group/t/deduplication-and-compression/714/1 "2021-11-02T22:14:36Z")

</div>

Hello everybody,

I was a little confused, that the compression setting is not specified for the whole repository, but can be specified by path.

I’m wondering now if deduplication still works when backing up paths with compression and paths without compression, when both contain the same files.

Kind regards!

---

<div class="post-metadata">

**Author:** ![stpr](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@stpr](https://kopia.discourse.group/u/stpr)\
**Post date:** [November 3, 2021, 12:09am UTC](https://kopia.discourse.group/t/deduplication-and-compression/714/2 "2021-11-03T00:09:28Z")

</div>

Compression _can_ (in a way) be enabled for the whole repository by specifying compression to be enabled globally like so:

```auto
kopia policy set --compression=zstd --global

```

By default all snapshots inhert from global policy, and therefore, compression will be enabled for all backup items. You can of course override it for specific backup items.

As for the deduplication and compression, [a recent change to the repository format](https://github.com/kopia/kopia/pull/1076) introduced as part of V0.9 means that data compression happens after hashing, so it should in theory mean that deduplication shouldn’t be affected by compression at all. So it shouldn’t care whether or not compression was enabled, or what algorithm was used.

---

<div class="post-metadata">

**Author:** ![koala](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/koala/32/249_2.png) [@koala](https://kopia.discourse.group/u/koala)\
**Post date:** [November 3, 2021, 8:29pm UTC](https://kopia.discourse.group/t/deduplication-and-compression/714/3 "2021-11-03T20:29:00Z")

</div>

Thanks for your reply @stpr!

The linked pull request seems to exactly address my questions!

I wonder though what is meant with

> Also since compression will be done after hashing, it has so be done server-side, thus the bandwidth usage and CPU utilization between kopia client to kopia server may change.

I was planning to use an S3 repository and (to my knowledge) no kopia server.

---

<div class="post-metadata">

**Author:** ![stpr](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@stpr](https://kopia.discourse.group/u/stpr)\
**Post date:** [November 3, 2021, 11:23pm UTC](https://kopia.discourse.group/t/deduplication-and-compression/714/4 "2021-11-03T23:23:02Z")

</div>

Kopia has a [‘repository server’ feature](https://kopia.io/docs/repository-server/) targeted at multi-user scenario where multiple users share a single repository, but provides some security features to prevent users from seeing each other’s contents. In such a setup, each user does not directly access the underlying files, but through the server program which takes care of authentication. This is not the typically relevant for single user backups, like perhaps you are using, where you have direct access to the filesystem.