How to fix missing blobs?

I recently found out about Kopia, and I really like it so far. It combines usability and functionality in a nice package and ticks all the checkboxes.

One thing I particularly like is the ability to skip snapshotting when no data was changed. This feature is missing in many backup tools. When using frequent snapshot schedule, and then needing to restore, say, a deleted file, it would be time-consuming to go through all snapshots (imagine over 100 per day) in order to find the file. Or finding actually changed versions of the file.

However, there is one thing I don’t understand about Kopia. My favorite test with backup tools is deleting a file at the destination and seeing what happens. When I did that with Kopia backup, it didn’t detect anything. Upon reading documentation I ran “snapshot verify”. It detected the missing blob. Good, but what do we do now? Subsequent snapshots happily take place without re-uploading the missing bits. The data is still there at the source. It should be a simple problem to fix.

There are several older discussions about the same issue, but no solution. I tried these (one at a time):
snapshot fix remove-files --object-id= --commit
snapshot fix invalid-files --verify-files-percent=100 --commit

It fixes the issue in terms of “snapshot verify” no longer throwing errors, however, a) missing data is not uploaded, b) the next snapshot runs, after which “snapshot verify” throws errors again, and rightfully so since the data is still missing.

I understand due to the nature of snapshots past snapshots won’t have the missing data. But surely there has to be a way to make Kopia back up missing data on the next run?

Kopia assumes the storage medium is stable and doesn’t try uploading things it thinks it has completed successfully. There is no way to recover a blob from the repository. This missing blob may affect multiple files because Kopia is doing dedups

However, if the source is still available then you can tell kopia to check everything on its next snapshot. So something like “kopia snapshot create --force-hash=100” should make it rescan everything and upload the missing parts.

If it was truly gone then the solution would just be to delete the snapshot or let the snapshot age out.

Thank you for a quick reply. I just tried this command, and it didn’t produce the desired outcomed. Even when running it wasn’t showing enough data uploaded. Running “snapshot verify” right afterwards shows the same “missing blob” error.

Do you have any other suggestions? It seems the only way is to wipe out the repository and start over. Even manually removing all snapshots and doing a new one with force-hash still didn’t help. Kopia is just stuck thinking it has the missing blob no matter what I do.

I had the same problem: kopia snapshot verify reported a missing BLOB. I implemented a (proof-of-concept) recovery tool in Python that is able to recover missing or broken BLOB pack files. With that I was able to recover (i.e. regenerate) the missing BLOB file using the snapshot’s source directory (at least it worked for my case).
The tool is available at GitHub - patrsc/kopia-recover: Python tool to recover broken or missing kopia pack file BLOBs.

Thanks for sharing your tool. I stopped using Kopia since my last post here. I switched to restic + Backrest UI. Restic easily handled my test with deleted blobs, and I also found it better suited for my needs.

After hitting this problem several years ago I did these very same tests to see if things had improved. Not the case I see.

This is an unfortunate shortcoming of otherwise excellent kopia. An HDD developing a bad sector forcing you to throw away a full repo history is hard to accept.

Can you share more details please. How does restic handle missing blobs in your scenario.

As far as I understand it, restic can not recover missing blobs.

From restic documentation:

5. Remove missing data from snapshots

If your repository is still missing data, then you can use the repair snapshots command to remove all inaccessible data from the snapshots. That is, this will result in a limited amount of data loss. Using the --forget option, the command will automatically remove the original, damaged snapshots.

Full doc:

borg has the same limitations by the way.

Of course, it cannot recover data that is not there, no tool can do that. What I was referring to is the ability to back up data that is still present in the source location but was lost in the backup.

These simple steps did exactly what I would expect from a backup tool.

  1. Delete some blobs in the backup destination.
  2. Run this commands:
restic.exe check -r d:\test\restic
restic.exe repair index -r d:\test\restic
  1. The next backup execution showed a warning about missing files but re-uploaded them. Problem solved.

Once again, if the data is lost in both source and target, no recovery is possible. However, if it’s lost in the backup, it should be easy enough to detect and back it up again. Kopia failed this same test when I posted the original message. With restic all I had to do was schedule a regular check with notifications so that I can address the issue if it arises.

Did you try to access or restore those files? I suspect that that silences Kopia about the missing data and broken snapshots, but it remains unavailable in newer snapshots.

Do you mean using restic? Yes, I did do a full end-to-end test with restoring data and doing a binary comparison to the original. If you are asking about Kopia, I tried every suggestion I could find in the manual or online. I don’t remember exactly all the different things I tried back in 2024. Nothing worked.

I did my own tests and I need some help to understand the results. This is for kopia experts in this forum. Please help.

  1. I created a kopia testrepo

  2. I create an empty directory /data/datatmp/aaa for which I do the backup to the new kopia testrepo

  3. I copied three files into the directory aaa . But after each file I created a snapshot. I ended up with 3 snapshots:
    1st snap has only one file debian-13.1.0-amd64-netinst.iso
    2nd snap has also ubuntu_cli_cheat_sheet_2025.pdf
    and 3rd snap has also openBSD_install75.iso

# kopia snap list

root@rakete:/data/datatmp/aaa
2026-02-26 22:12:41 CET k7adbcdc292f46c80512da3b03feb966c 821 MB drwxr-x— files:1 dirs:1 (latest-3)
2026-02-26 22:13:16 CET kb936d0dba18902ca9e8df2170a79fa21 821.2 MB drwxr-x— files:2 dirs:1 (latest-2)
2026-02-26 22:13:51 CET k22751c4171ebb78d359e9f0e4bc75c06 1.5 GB drwxr-x— files:3 dirs:1 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

The last repo mounted shows:

# ls -l

insgesamt 1,5G
-rw-r----- 1 matthias matthias 783M 2026-02-26@22:12 debian-13.1.0-amd64-netinst.iso
-rw-r----- 1 matthias matthias 654M 2026-02-26@22:13 openBSD_install75.iso
-rw-r----- 1 matthias matthias 146K 2026-02-26@22:13 ubuntu_cli_cheat_sheet_2025.pdf

I verified the repo and all is good:

# kopia snapshot verify --verify-files-percent=100 --file-parallelism=10 --parallel=10

Listing blobs…
Listed 80 blobs.
Processed 0 objects (0 B). Read 0 files (0 B).
Processed 5 objects (685 MB). Read 2 files (685 MB).
Finished processing 6 objects (1.5 GB). Read 3 files (1.5 GB).

The repo contains 80 blobs. I deleted 6 of them, leaving only 74 blobs. The verify command complains about it:

# kopia snap verify

Listing blobs…
Listed 74 blobs.
Processed 0 objects (0 B). Read 0 files (0 B).
error processing root@rakete:/data/datatmp/aaa@2026-02-26 22:12:41 CET/debian-13.1.0-amd64-netinst.iso: object Ixa9995cc568f13ca396376d7f4e0d454b is backed by missing blob p6a6cbd821abd8ecddd50885c898024d0-s97f3d99cc518b80213e
error processing root@rakete:/data/datatmp/aaa@2026-02-26 22:13:51 CET/openBSD_install75.iso: object Ix1690a32c30ed2a80d38f15139d056829 is backed by missing blob p84063a74b73e79fdac5c803fe67c2526-sbc83ccaba170d8f113e
Finished processing 6 objects (1.5 GB). Read 0 files (0 B).
encountered 2 errors

I can not repair it:

kopia snapshot verify --verify-files-percent=100 --file-parallelism=10 --parallel=10

Listing blobs…
Listed 74 blobs.
Processed 0 objects (0 B). Read 0 files (0 B).
error processing root@rakete:/data/datatmp/aaa@2026-02-26 22:12:41 CET/debian-13.1.0-amd64-netinst.iso: object Ixa9995cc568f13ca396376d7f4e0d454b is backed by missing blob p6a6cbd821abd8ecddd50885c898024d0-s97f3d99cc518b80213e
error processing root@rakete:/data/datatmp/aaa@2026-02-26 22:13:51 CET/openBSD_install75.iso: object Ix1690a32c30ed2a80d38f15139d056829 is backed by missing blob p84063a74b73e79fdac5c803fe67c2526-sbc83ccaba170d8f113e
Finished processing 6 objects (1.5 GB). Read 1 files (148.6 KB).
encountered 2 errors

Interesting is, that the mounted repo still shows all three files. And I can retreive all 3 files and they match the original files. no diff.

This leaves me with a couple of questions:

  1. Why are the files in the repo still correct? Specifically openBSD_install75.iso and debian-13.1.0-amd64-netinst.iso. They both have missing blobs.

  2. I mounted each of the 3 snapshots and retrieved all files from the snapshots and nothing is missing and all files are accurate. No diff. How can that be?

  3. Why is kopia snapshot verify --verify-files-percent=100 --file-parallelism=10 --parallel=10 not repairing anything?

  4. What can I do to stop the error messages from the kopia verify command?

Sorry, in my haste I thought that was with kopia. I’ve done the same tests and indeed restic does recover easily that way.

Did you unmount/remount after the deletion? It might be file handles are kept open to the deleted files.

Well actually, there are some posts on this forum, which deal with corrupted data and invalid snapshots. The key is, that you run kopia snapshot veriy with the --commit flag, which will remove missing references to missing blobs and repair the snapshot manifests.

Just use the forum’s search and search for “invalid-files”.

I did some more testing and I think this is a flaw in kopia compared to restic. It is very simple to reproduce and I would love to see some folks here trying to reproduce it.

  1. create a test repo
  2. create a source directory with just on bigger file to backup, e.g. in my case /data/datatmp/aaa/debian-13.1.0-amd64-netinst.iso
  3. backup with kopia
  4. remove a blob. If you happen to remove a blob from the repos manifest, the repo is broken and you have to repeat step 1-3 until you remove a blob that belongs to the file:
# kopia snap verify
Listing blobs...
Listed 113 blobs.
Processed 0 objects (0 B). Read 0 files (0 B).
error processing root@rakete:/data/datatmp/aaa@2026-02-27 15:03:34 CET/debian-13.1.0-amd64-netinst.iso: object Ix311a92280fcaa16d6edddd502003342f is backed by missing blob peffa77dd5109f1ae3d2ba4907485053b-s4f4827054c679c4613e
Finished processing 2 objects (821 MB). Read 0 files (0 B).
object Ix311a92280fcaa16d6edddd502003342f is backed by missing blob peffa77dd5109f1ae3d2ba4907485053b-s4f4827054c679c4613e

If I mount the repo I can copy the file to /tmp and do a diff with the original source. There is a diff, which is expected.

kopia can repair the repo with:

kopia snapshot fix invalid-files --verify-files-percent=100 --commit

but this does not repair the file. Up to this point kopia and restic basically behave the same. The file is gone. It can not be repaired.

Here is the big difference between kopia and restic:

kopia: If I do another backup of the folder /data/datatmp/aaa, the correct file which is still in the folder is not put back into the repo. The backup keeps the corrupt file debian-13.1.0-amd64-netinst.iso forever. No further backup will change that.

restic: The next backup puts the correct file debian-13.1.0-amd64-netinst.iso back into the repo. And that also fixes the older snapshot. Old and new snapshot are clean now with the correct file.

For me that looks like a flaw in kopia.

2 Likes

Yes, that’s the conclusion I arrived to when faced with this problem for the first time, and I think that’s the general understanding of the issue once you dig a bit in similar threads. Kopia repairs “fix” indices by removing the files with missing data and replacing them with a special filename, but those files are lost forever in that repo, in past and future snapshots.

I’ve been told in some github issue about it that kopia can’t fix past snapshots because it’s kind of like a tape backup that can go only forward. I’m unconvinced because then it wouldn’t be able either to detect blobs are already there for deduplication of new files but, since I don’t know the internals, I didn’t press the idea.

But it’s true that the inability to recover, by design or by current lack of implementation, is a big shortcoming against restic, because it makes repositories very brittle. It’s indeed the only reason I’ve moved my backups to restic.

Happy to be corrected on any of those points, wouldn’t want to spread any misinformation.

1 Like