Details of maintenance command

Support
1

Hello,

I’ve been playing with Kopia to see how it compares to Restic and one thing that I’m not sure about is exactly what the maintenance command does.

Could someone give some details on this command?

Thanks!

1 Like
2

Does this help?

3

Yes, it does. Thank you!

4

So then if I’m understanding correctly what the maintenance command does, we should only need to run the “maintenance full” periodically to delete content that is no longer needed. Is that correct?

I also see that there is the command “snapshot gc”. If one runs the above maintenance full command, is there a need to run snapshot gc?

I’m trying to get a handle on what’s needed to keep things cleaned up.

Thanks!

5

snapshot gc is an advanced command without some guardrails that are present in maintenance run. It’s recommended to use maintenance going forward which will keep your repo nice and tidy over time.

6

Thank you for the reply!

7

Sorry for resurrecting the topic …

I understand the general design of Kopia, but in this aspect of maintenance I still don’t understand what is actually being done (yes, I read the documentation).

There in the documentation it is said that

to ensure best possible performance and optimal storage usage.
(…)
keeping the number of frequently accessed blobs ( q and n ) low to ensure good performance
(…)
keeping the repository compact and eliminate deleted files that the user no longer wishes to store.

which is vague.

What exactly is being done? Are the d files (related to the snapshots removed by the policies) being deleted?

If so, why does this need to be done on an hourly basis?

(BTW, the link above is broken, the current one is: Maintenance | Kopia)

8

This is still intentionally vague, because details are subject to change, but let me provide a bit more information here (accurate as of v0.8.2 release).

I don’t want to go into too many details to avoid the information becoming stale quickly and to prevent folks overly-optimizing their backup configurations and cargo-culting those. The intention of Kopia and recommendation for most users is to not worry about maintenance at all as it should be automatic and unobtrusive - if that’s not the case, please file bugs.

(There’s always source code if somebody wants to go deeper)

There are two types of maintenance:

  • quick maintenance manages and optimizes indexes and q blobs that store metadata (directory listings, manifests such as snapshots, policies, acls, etc.).
  • full maintenance manages both q and p data blobs (which store contents of all files).

Maintenance is composed of individual tasks grouped into two sets:

Quick Maintenance

This runs frequently (hourly) with the goal of of keeping the number of index blobs (n) small, as high number of indexes negatively affects the performance of all kopia operations. This is because every write session (snapshot command, any policy manipulation, etc.) adds at least one n blob and usually one q blob so it’s very important to aggressively compact them:

  • quick-rewrite-contents - looks for contents in short q packs that utilize less than 80% of the target pack size (currently around 20MB) and rewrites them to a new, larger q pack, effectively orphaning the original packs and making them eligible for deletion after some time.
  • quick-delete-blobs - looks for orphaned q packs (that are not referenced by any index) and deletes them after enough time has passed for those contents to be no longer referenced by any cache.
  • index-compaction - merges multiple smaller index blobs (n) into larger ones

Full maintenance

The main purpose of full maintenance is to perform garbage collection of contents that are no longer needed after snapshots they belong to get deleted or age out of the system.

  • snapshot-gc - finds all contents (files and directory listings) that are no longer reachable from snapshot manifests and marks them as deleted. It also undeletes contents that are in use and have been marked as deleted before (due to unavoidable race between snapshot gc and snapshot create possible when multiple machines are involved).

NOTE: This is the most costly operation as it requires scanning all directories in all snapshots that are active in the system. The good news is that all this data is in q blobs and thanks to the quick maintenance it was kept nice and compact and quick to access, so this phase does not usually take that long (e.g. currently ~25 seconds on my 720 GB repository with >1.5M contents).

  • full-drop-deleted-content - removes contents that have been marked for deletion long enough from the index. This creates “holes” in pack blobs and/or makes blobs completely unused and subject to deletion.

  • full-rewrite-contents - same as quick-rewrite-contents but acts on all blobs (p and q)

  • full-delete-blobs - same as quick-delete-blobs but acts on all blobs (p and q)

There are additional safety measures built into the maintenance routine to make it safe to run even when other kopia clients on other machines are executing snapshots concurrently. For example {quick}-delete-blobs will not run if less than X amount of time has passed since last content rewrite and full-drop-deleted-content will only drop contents if enough time has passed between full maintenance cycles.

The recommendation is to run quick maintenance as frequently as it makes sense for your repository (hourly is typically fine). The entire quick cycle should take <10 seconds, even for big repositories.

Full maintenance cycle runs every 24h and can be spread apart further (weekly or even monthly is probably fine) or stopped completely if somebody does not want or care to reclaim unused space.

9

Thanks for the detailed explanation! Understood.

So, in a very summarized / simplistic way:

  • Quick Maintenance = maintenance of metadata
  • Full maintenance = content maintenance / deletion

I’m still thinking about the need for hourly maintenance of metadata …

Allow me to make a comparison: I have been a Duplicacy user for many years. In some ways the design / architecture is very similar.

In Duplicacy there is prune for old snapshots (which, as I understand it, is equivalent to Kopia full maintenance). But there is absolutely no need to do so.

In fact, I have backups with hundreds of snapshots (which Duplicacy calls revisions) stored in B2, and there is no noticeable performance problem.

Of course, if I eliminated some older revisions, the listing (and other operations, like checks) of the backup revisions would be faster, but in my use case, it is not noticeable.

So I just don’t do prunes, for the simple reason that storage is so cheap that it’s something that doesn’t justify my time to create / manage scripts for it.

So in my case, backup is something like “send the new files to B2 and that’s it, forget about them”. Very convenient.

10

You really need quick maintenance for index compaction, otherwise it will become slow very quickly. as indexes become fragmented and merging too many of them at runtime is costly. We could add essential maintenance mode that only runs index-compaction task which would run every hour? Then somebody could totally disable quick maintenance as long as essential runs periodically.

11

Got it.

In this case, IMHO, Duplicacy has a simpler and more robust architecture, which requires less - or none - maintenance.

For Duplicacy there is also the equivalent of the indexes (which in its case are the revision files that reference the chunks), but they do not need any maintenance at all. They are only deleted if your prune policy so defines it.

On the other hand, a very useful feature that Kopia has - and Duplicacy doesn’t - is mount. :sunglasses:

12

That link is dead, is there a new one?

13
1 Like
14

My repository is on a simple external USB hard drive that would probably live longer if it didn’t run 24/7 or, even worse, jump on every hour for 10 seconds to then go back into standby. In this case, would it make sense to turn off automatic maintenance and instead have quick maintenance, followed by full maintenance run as Before Snapshot Actions?

That way, the hard drive would jump on once a day, do quick maintenance, then full maintenance, then the daily backup run, and then go back into standby.

The repo is 5TB and it’s only one client backing up to it once a day.

15

Then what is the point running maintenance more often than once a day?:slight_smile:

For sure in such case the best option IMO is to run quick maintenance before backup and full after. In reality does not matter much in which order.

Myself I turned auto maintenance completely and run it manually (from within my backup script). The reason is like yours - I run backup only once a day. Before/after snapshots actions could do - never tried. Let us know if it works this way.

16

Sidenote: Quick maintenance was never run on Kopia versions < 0.18.0 even if enabled.

If your data doesn’t change a lot and maintenance therefor doesn’t free up much needed space, I’d probably set quick maintenance to run every 12-24 hours and full maintenance to run once a week.

17

Thank you @kapitainsky and @dimejo! I will test and report back here in a little while.