# Question: Can two distinct local repositories sync-to the same remote repository? **URL:** \ **Category:** Support\ **Created:** [May 28, 2022, 7:41pm UTC](https://kopia.discourse.group/t/question-can-two-distinct-local-repositories-sync-to-the-same-remote-repository/1164 "2022-05-28T19:41:40Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![dsimmons](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/dsimmons/32/218_2.png) [@dsimmons](https://kopia.discourse.group/u/dsimmons)\ **Post date:** [May 28, 2022, 7:41pm UTC](https://kopia.discourse.group/t/question-can-two-distinct-local-repositories-sync-to-the-same-remote-repository/1164/1 "2022-05-28T19:41:40Z")
## Context I’m currently in the process of re-designing my “backup architecture” and I was hoping to obtain some clarification around this question before I proceed. I _think_ I know the answer, but a confirmation(s) would be helpful! Topology: - Let “ **DESKTOP** ” be my desktop computer. - Let “ **LAPTOP** ” be my laptop computer. - Let “ **LOCAL COPY** ” be a locally-networked GlusterFS triple-replicated cluster. - Let “ **OFFSITE COPY** ” be Backblaze B2 (or similar, but that’s what I’m currently using). In short, here’s the **desired use case** : - **DESKTOP** runs frequent backups, backing up locally to its own filesystem. - **LAPTOP** runs frequent backups, backing up locally to its own filesystem. - _ **BOTH** _ **DESKTOP** and **LAPTOP** use `sync-to` to redundantly copy to the **LOCAL COPY** “remote” repo. - _ **BOTH** _ **DESKTOP** and **LAPTOP** use `sync-to` to redundantly copy to the **OFFSITE COPY** remote repo. ## Question So, in a nutshell, what I’m asking in plain English is: can _ **distinct** _ repositories (in this case, **DESKTOP** and **LAPTOP** each have their _ **own** _ local repositories) `sync-to` the _ **same** _ “remote” repository/destination? * * * ## Additional Context ### What I’m doing currently: - Both **DESKTOP** and **LAPTOP** run snapshots of `$HOME` using the same repo hosted on **LOCAL COPY** (network FS). - **LOCAL COPY** has `sync-to` invoked on it once-nightly and outputs to B2 ( **OFFSITE COPY** ). ### Why I want to make changes: - Whereas I currently keep most my personal data under `$HOME`, I want to start to partition it out and get more granular for (at least) the following reasons: - Why distinct repos for each of **DESKTOP** and **LAPTOP**? - When it comes to multiple devices, there’s a “core” set of data (~30GB) that I want to sync between them, 80-90%+ which will be redundant (where the primary differences are dotfiles and installation-specific data). I sincerely appreciate those that’ve taken the time to read this far and/or chime in, hopefully this all makes sense! Happy to clarify anything that I may have done a poor job of explaining as far as my use case goes! ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [May 29, 2022, 1:41pm UTC](https://kopia.discourse.group/t/question-can-two-distinct-local-repositories-sync-to-the-same-remote-repository/1164/2 "2022-05-29T13:41:47Z")
Thanks for the write-up. In general `sync-to` can’t really be used to merge repositories because index formats make that impossible. There may be other ways, though: I have few observations: 1. Relying on sync-to as a safety measure has an issue where you may be replicating data corruption (bitrot or bugs) at the source because `sync-to` is just copying files and not performing consistency checks as it does it. 2. I would recommend against using fancy networked filesystems (like GlusterFS) because they sometimes don’t provide expected consistency guarantees (like read-after-write, list-after-write). Not sure if Gluster has those, but I would stay away from that as much as possible. 3. I would similarly recommend using good filesystems with bitrot protection (zfs, btrfs) and performing regular scrubs of data to ensure it’s still ok (this is critical, otherwise bitrot protection does not really do its job). 4. Try to make data recovery as trivial as possible, each layer of complexity can make it harder or slower to recover. I would consider eliminating or simplifying VPNs, GlusterFS if you can do it. Dealing with data loss is one thing when it happens, but not being able to get to the backup is another (when VPN is suddenly misconfigured or when Gluster decided to develop a split brain, etc.) Overall instead of synchronizing I would generally use several independent repositories and backup to them individually, possibly on a different schedule as much as the bandwidth allows it. 1. Each machine backs up all data to its local repository. 2. Each machine backs up all data to the other machine’s repository (desktop-\>local, local-\>desktop), not sure how practical it is, using something like SFTP or webdav or perhaps even 3. Each machine backs up only important data to the cloud while you’re internet bandwidth constrained. 4. And most important, test your backups regularly. ---
**Author:** ![dsimmons](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/dsimmons/32/218_2.png) [@dsimmons](https://kopia.discourse.group/u/dsimmons)\ **Post date:** [May 29, 2022, 6:28pm UTC](https://kopia.discourse.group/t/question-can-two-distinct-local-repositories-sync-to-the-same-remote-repository/1164/3 "2022-05-29T18:28:43Z")
Sincerely thank you (yet again!) @jkowalski for your time and a speedy reply, a detailed response, and of course, your work on this incredible project! I probably said this the last time you gave me a similar response in another thread, but Kopia was exactly what I was after for years! > 1. I would recommend against using fancy networked filesystems (like GlusterFS) because they sometimes don’t provide expected consistency guarantees (like read-after-write, list-after-write). Not sure if Gluster has those, but I would stay away from that as much as possible. > > 2. I would similarly recommend using good filesystems with bitrot protection (zfs, btrfs) and performing regular scrubs of data to ensure it’s still ok (this is critical, otherwise bitrot protection does not really do its job). I did see this in the Kopia documentation recently! Maybe I missed it when I first started using Kopia? In any case, unfortunately, I noticed it well after I had a process/solution in place and had several months worth of Kopia backups using this strategy. For what it’s worth, I do have [bitrot detection and scrubbing](https://docs.gluster.org/en/latest/Administrator-Guide/Managing-Volumes/#bitrot-detection) (see link if interested) enabled on my Gluster cluster, but point taken! I mentioned in my train of thought above that I didn’t like this cluster being a single point of failure, and although I didn’t say it explicitly, another equally significant factor in my decision-making process was having discovered the advice around not using networked filesystems in the documentation recently (that too made me uneasy about having it in my “critical path”). It may also be worth mentioning that, originally, when I began utilizing that networked filesystem as my “source of truth” for backing up my devices, it was largely due to space constraints, as I didn’t have disk capacity to keep both my “working set” _and_ its corresponding backup on the same machine(s). I’ve since increased each device’s capacity, _ **and** _, like you read above, I’m revisiting how I “partition” my overall working set and backups. So for all of those reasons combined, I have more options on the table now, so I’m consequently looking to remove Gluster from being a primary part of the equation (whereas I might still use it for added redundancy). > 1. Relying on sync-to as a safety measure has an issue where you may be replicating data corruption (bitrot or bugs) at the source because `sync-to` is just copying files and not performing consistency checks as it does it. Just to clarify, are you saying that `sync-to` isn’t worth using in any capacity? As an example, on **DESKTOP** and **LAPTOP** , my `crontab` entry reads as follows: ```auto */30 * * * * kopia snapshot create $HOME && kopia snapshot verify --verify-files-percent=1 ``` As an additional related question, does `--verify-files-percent` help alleviate the issues you’ve raised with `sync-to` around verifying consistency? > Overall instead of synchronizing I would generally use several independent repositories and backup to them individually, possibly on a different schedule as much as the bandwidth allows it. re “individually”: as you probably guessed, my hope in “sharing” a destination repository for `N` distinct source repositories was so that I could make use of the content-addressing to save space and eliminate redundancy. But it sounds like, per your comment about the index format, it’s not possible! Just to ensure that I’m following what you’re prescribing in your advice, it sounds like you’re recommending I should have a unique “destination repository” for each local/source repository, even where the majority of the data is shared? So graphically: - **DESKTOP** (A) snapshots to its own local filesystem, then to a remote copy (A). - **LAPTOP** (B) snapshots to its own local filesystem, then to a remote copy (B). - Stated explicitly, I’m assuming `A != B` (each is unique, despite having largely the same data). Thanks again! 🙂 ---
**Author:** ![iBackup](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/ibackup/32/355_2.png) [@iBackup](https://kopia.discourse.group/u/iBackup)\ **Post date:** [June 1, 2022, 10:20am UTC](https://kopia.discourse.group/t/question-can-two-distinct-local-repositories-sync-to-the-same-remote-repository/1164/4 "2022-06-01T10:20:45Z")
If both, DESKTOP and LAPTOP share some files(so you can greatly benefit from deduplication), then I would setup a single, dedicated local repository with `kopia server`, then create on a server 3 distinct users: one is global admin and two clients - desktop & laptop then you can push snapshots to a single repository that has reliable local filesystem like ZFS then use `sync-to` for offsite backups from repository server. This way you can protect your files not only from bitrot but also from more real treat like ransomware that can overwrite your backups, but in case of using dedicate repository server, you can setup as a global admin a policy for DESKTOP & LAPTOP to use APPEND only mode, so in case if original files would be encrypted by ransoware it won’t destroy previous snapshots. If you can’t setup reliable filesystem on repository server (a dedicated computer), then you can use very old utility `mtree` (or Go port: go-mtree on github) that used for checking integrity of directories and files, so if you would run it periodically on repository, you would be sure that repository is Ok. For some very paranoiac cases, one can even use `par2` utility to protect repository from damages on file system level. To check consistency offsite backups that you feed with `sync-to` from repository server, you can periodically connect to and run verification over it to be make sure remote repo isn’t damaged.