I have a backup repository created on a Windows machine on an NFS path hosted on a Linux machine. I then tried to connect to the repository on the Linux machine and received an error:
eric@dev:~$ kopia repository connect filesystem --log-level debug --path /media/eric/kopia-backup-test
Enter password to open repository:
DEBUG Creating cache directory '/home/eric/.cache/kopia/ee807e4fb082da80' with max size 5242880000
DEBUG throttling limits from connection info {"limits":{}}
ERROR unable to add xn3_9ecf335f7d8359c629cc232af415c1e9-s61e168df780fb87c119-c1 to index blob cache: context canceled
ERROR unable to add xs0_c9078e4e551fa206d70319bfd83065a5-s16566f33cadcc745-c1 to index blob cache: context canceled
ERROR failed to open repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: error decrypting BLOB xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: unable to decrypt content: cipher: message authentication failed
ERROR error connecting to repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: error decrypting BLOB xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: unable to decrypt content: cipher: message authentication failed
Running kopia snapshot verify on the Windows machine works fine. Any ideas what the problem may be?
I cleared the cache on the source Windows machine hoping that it would sync any cache files to the NFS share. Unfortunately, it appears to have just blindly deleted the cache and now I get the same error on both the Windows machine and the Linux system.
kopia.exe --config-file=C:\Users\eric\AppData\Roaming\kopia\repository.config cache clear
ERROR failed to open repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: error decrypting BLOB xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: unable to decrypt content: cipher: message authentication failed
ERROR open repository: unable to open repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: error decrypting BLOB xn3_63933a8b6f6b6184e00a1e1c31b9bd52-scd06c22445e2619b119-c1: unable to decrypt content: cipher: message authentication failed
So, looks like I lost my backup. I ran out of disk space on the NFS share, so my guess is that some of the files were not correctly stored on the share and Kopia was pulling them from the cache. Seems like a latent bug since clearing the cache is specifically mentioned as safe:
Cache can be cleared on demand by kopia cache clear or by simply removing appropriate files. It is always safe to remove files from cache.
Can you post some details about that nfs share? I’ve run kopia using a smb share, repository server, or using sftp and it worked as it should be. Are you mounting a nfs share on windows?
I tried removing all zero-sized files matching the pattern .tmp. and still getting the same error loading index blob message shared earlier. I haven’t tried the suggestion of deleting the most recent files, yet.
I was using NFS as a generic term (my bad). The network file system is actually Samba (SMB) running on Ubuntu.
Here is a summary of the system and sequence of events up to the failure
Backup client: Windows Kopia GUI v 0.12.1
Server: Ubuntu 22.04.2 LTS with backup drive using BTRFS on an SSD shared using SMB (Samba) Version 4.15.13
The backup drive SSD ran out of drive space 1 week ago, I removed other files to make space and then it filled up again 3 days ago. After clearing out space both times, all maintenance and backups were successful on the Windows client.
I then tried to do a repository connect filesystem on the server to verify the files and that failed, but I was able to do it successfully on the Windows machine and running kopia snapshot verify on the Windows client also worked.
I then ran cache clear on the Windows client after which the same error loading index blob failure exists on both the Windows client and the Linux server.
At this point, I want to understand what happened and if it is possible to roll the backup back to a known good state (presumably back to before the first drive-full scenario).
If the files are important to you, I suggest to first make a copy of the repository if possible. I am using rsync for this purpose. Then you can try to remove the newest index files one by one as mentioned in the post linked above.
However, as this is also new to me, please proceed with caution. If this is not working, you may have to reach out to the devs.
I had to remove all of the index files per step 4 of the recovery instructions which put me back to an empty backup set. So it seems that the recent disk-full events ended up corrupting 3 months of daily backups.
Not sure if the developer tools will be useful here to recover the data if I truly needed it. I setup a new repository and did a backup and it works from both the Windows client and on the Linux SMB share server.
4. If a repository can’t be opened but was working recently and maintenance has not run yet, it may be helpful to try to remove (or stash away) the most recently-written index files whose names start with x in the reverse timestamp order one by one until the issue is fixed. This will effectively roll back the repository writes to a prior state. Exercise caution when removing the files.
