Repo is repeatedly "corrupt"

Support
1

Hi, just starting to test Kopia and I’m liking the concept, but I’m a little bit concerned about it because so far it is failing to decrypt my repo and it’s basically unusable. Whenever I start the container, it fails to connect the repo because it can’t decrypt the blobs. I manually have to delete every one of them for it to finally connect, but then the repo is lost.

The repository is stored in backblaze B2. Surely there is something wrong that I’m doing, there is simply no way every blob is corrupt, let alone two times in a row, and especially not on backblaze.

In both cases I installed kopia on TrueNAS Scale with the supplied Truecharts app. I just tested running the standard docker container as well and am getting the same results though.

Setup from a fresh repo and setting up various snapshot policies yields no issues. It’s just when I restart the container it can’t connect because it can’t decrypt the blobs.

Just as an example of the error:

Connect Error: INTERNAL: internal server error: connect error: error opening repository: error connecting: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn0_8a8561edfdbedb5709cfe94c91fbcbe7-sa31078da3c3099a8126-c1: decrypt blob: error decrypting BLOB xn0_8a8561edfdbedb5709cfe94c91fbcbe7-sa31078da3c3099a8126-c1: unable to decrypt content: cipher: message authentication failed```
2

Try some other destination than B2. Or B2 but using S3 API.

If still the same issue than you have to talk to Truecharts app community as it looks like there is some kopia deployment issue on your platform.

Or just forget all Truecharts app and use kopia from command line.

3

I started with Truecharts support and they pointed me here because the issue was outside of them. I also setup a Kopia instance on a linux VM with docker, and attempted to connect to the repo that way. But I get the exact same issue.

I know I can just delete the whole repo and start a new one, but I already did that. I really would like to understand what is going on if I am to continue using it. I just tried creating a new application key with Backblaze that should be S3 compatible, but that yielded the same results.

4

I am using kopia with multiple destinations without slightest issues so I doubt that there is some fundamental problem with kopia itself.

I am not familiar with docker deployment but I can see on the forum that people use it this way as well.

To get more help here you have to provide more details - ideally all. What and how you run it, configuration files, logs etc. Then maybe somebody can spot what you are doing wrong.

Try to access the same “corrupted” repo without using docker - simply by using kopia CLI.

6

Sure. I attempted to connect to the repo using CLI and had the same result. I will go over my full details here.

  1. I setup a Kopia UI instance on TrueNAS Scale using the stable release from TrueCharts. My configurations were confirmed to be correct by Truecharts via their support channel. To configure, it is similar process to docker where I provide a server user / pass, a repo user /pass, and some host paths in read only format.
  2. I created a Kopia repository on Backblaze B2 and created an API key with only access to that bucket. I realize now this meant it was not S3 compatible, but I am not sure that is relevant or not.
  3. I setup my snapshot policies and created 2-3 different snapshot and verified everything was working, including recovery.
  4. I restarted the container hosting my Kopia instance, and noticed it failed to deploy. Reviewing the logs showed the above error.
  5. I deleted my local cache, this did not work
  6. I deleted my entire bucket on Backblaze, this allowed the container to start again but there was no repo to connect to.
  7. I created a new bucket on Backblaze, with new API Keys.
  8. I resetup everything and once again confirmed it was working correctly and recovering correctly.
  9. I once again restarted the app, and the same issue appeared.
  10. This time, I wanted to get to the bottom of it. I downloaded all of my blobs from backblaze so I could start deleting them to see if it was just one or all of them. It was every blob causing this issue.
  11. I setup a linux VM and installed another kopia instance using Docker. I tried to connect the repository and got the same issue until all blobs were deleted, at which point it connected but none of my data was shown (obviously)
  12. I considered if it was a password issue (I am using a long password, about 52 characters), so I tried using a subset of the password to see if there was a character limit that was being imposed somewhere. But if I use any other password the error I get it “INVALID PASSWORD” which suggests to me the password is correct.
  13. I tried manually using Kopia CLI to connect, and again I got the same error.
  14. I created a new S3 compatible API key and attempted to connect to Backblaze B2 that way, same error.
  15. I tried using that new key and connecting through the Amazon S3 or Other Compatible storage option, which again yielded the same error.

And that is I think the full details of what has occurred. Of course, I can delete this bucket again and start again, but I wish there was someway to know exactly what was happening.

Also I agree it is likely not a fundamental Kopia issue. But there is something causing a perplexing failure in my setup and I think I need to know what it is before I choose to use it.

And for testing, here is my docker compose file

version: '3.7'
services:
    kopia:
        image: kopia/kopia:latest
        hostname: kopia-7c4f759cdf-n6hhd
        container_name: Kopia
        restart: unless-stopped
        ports:
            - 51515:51515
        # Setup the server that provides the web gui
        command:
            - server
            - start
            - --disable-csrf-token-checks
            - --insecure
            - --address=0.0.0.0:51515
            - --server-username=bbradley
            - --server-password=<REDACTED>
        environment:
            # Set repository password
            KOPIA_PASSWORD: "<REDACTED>"
            USER: "brian"
        volumes:
            # Mount local folders needed by kopia
            - /home/bbradley/kopia/config:/app/config
            - /home/bbradley/kopia/cache:/app/cache
            - /home/bbradley/kopia/logs:/app/logs
            # Mount path for browsing mounted snaphots
            - /home/bbradley/kopia/browse:/tmp:shared
7

By any chance don’t you have some server side encryption enabled for your B2 buckets? It can be something kopia B2 connector does not understand…

I would try creating repo on some other remote - local or some other cloud. If it works than you would know that issue is with B2 bucket and not with docker kopia setup.

8

I can try starting over and seeing if I can reproduce it on local storage.

9

Your detailed step by step description is perfect - but I have to say this problem leaves me puzzled. Does not help I do not use docker nor B2 but I am fairly familiar with kopia in any other configuration. All your steps are perfectly legit and at least at the moment it does not make sense that it does not work. Obviously there is some reason - hopefully it can be discovered.

10

Me too! I still have a suspicion Truecharts is somehow related, but they shoed me off here. As of now, I just repeated my steps to make sure I had a valid test process, but this time I am not seeing any corruption… The only difference is this time I setup the bucket so that it was S3 compatible. Is it possible that is the difference? Is a standard B2 bucket not supported?

To me, it seems almost as if something was causing some files to be encrypted with the wrong password somehow. But I can’t understand how that would be possible. I don’t know enough about kopia yet to really understand how to debug this one unfortunately.

11

It is always possible that B2 integration in kopia is not working any more as it should be… I have seen it with other software. B2 API being just one of many propriety protocols usually has much less attention than APIs like S3 - which is de facto standard most stick to.

Example here - Preparing a new repository — restic 0.16.4 documentation
Would be interesting to dig into source code of kopia to see what library it is using. Both restic and kopia are golang.

Anybody using B2 with kopia reading this? Please provide some feedback.

I am sorry I can only speculate but I do not see anything obvious yet beyond:

This is why I thought about some B2 server side encryption applied that kopia is not aware of.

12

If I read correctly you always used the TrueCharts instance at some point. Did you try to only use kopia UI (or CLI) locally on your PC, connect to B2 and create a new repo?

I wanted to use kopia on my TrueNAS instance as well but there were some blocking issues for me. I tried to get some help but in the end I decided to host it on a separate machine using portainer.

13

I was previously only using kopia UI via the TrueCharts app. I had everything (apparently) fully working that way. But I was getting this silent corruption when restarting the app that I was never able to understand or get to the bottom of. I don’t know what was responsible for it. But everything appeared to work otherwise.

Now I am instead hosting it in a docker container on a Ubuntu server VM running on TrueNAS Scale. So far I am not having issues doing it this way, but I probably won’t trust it fully with my data until I am running it for a few months without issues at least.

1 Like