I have some difficulty understanding the retention policy in Kopia. I’ve read the whole thread of Trying to understand retention policies , but still can’t be 100% sure.
Suppose I make a snapshot every day at 0:00. I want to have
Latest 4 days’ snapshots.
After that, only retain 4 weekly snapshots.
So to be precise after running the schedule for a while, I should have exactly 7 snapshots (day1 and week1, day2, day3, day4, week2, week3, week4). However, it looks like what actually happens is that after the day4 snapshot, the rest three snapshots are never created. Jarek said in the thread that “After each snapshot is taken, kopia will assign retention tag to snapshots and delete snapshots that don’t have any assigned tag.” So I’m understanding that once a snapshot is 5 days old, since it can’t be tagged by any policy, it is always deleted. Since there is never any snapshot survive more than 5 days, week2, 3 and 4 are also never created.
So basically in Kopia, it only makes sense if smaller retention tags have durations that can cover larger retention tags. For example, given X weekly retention, user should always have at least (X-1)*7 daily retentions.
Am I understanding it correctly? If yes, this is slightly different to other backup systems. I think this definitely worth some in-depth explanation in the user guide.
I cannot say, that this is the same for me. Have you set set the weekly, monthly and yearly retension periods explicitly, or did you leave untouched - still showing the # instead of setting it explicitly to 0?
Try setting latest to 1 and the ones you don’t want… leave them alone or delete 0, should you have set one.
Another easy way to demonstrate is to set Latest to say 4, and Hourly to 4. Leave the rest on 0. With this policy, if more than 4 snapshots are created within one hour (e.g. set scheduling to 10 minutes), there will never be a “hourly-2” snapshot. It will be just “latest-1/hourly-1”, “latest-2”, “latest-3”, “latest-4”.
On the other hand, if setting scheduling to 30 minutes, then there will be “latest-1/hourly-1”, “latest-2”, “latest-3/hourly-2”, “latest-4”, “hourly-3”.
What I want to achieve is frequent backups up close and loose backups far away. Latest just provide evenly distributed backups based on snapshot interval.
I think Kopia’s retention method making it difficult to achieve that. I’m OK with that, just need to change strategy from now on with the knowledge.
I get that. However, just think of the retensions like applying labels to each snapshot. Snapshots with labels stay, the ones without are deleted. I played with the policies a bit and that was the best I could come up with.
I have been running Kopia for a couple of weeks and my goal was to have
Based on the screenshot, I assume your snapshot interval is around 24 hours? My concern is that, if you happen to have that interval to be less than 24 hours, you will end up having exactly one (the “latest-1”) snapshot. All other labels will not stay. “Different snapshot interval invalidates retention” is gonna surprise users.
On to retention specifically. Your 7 daily covers a week. That’s why you have weekly-2. But if you change that daily to 6, your weekly and monthly retention won’t do anything anymore.
Also, 4 weekly does not necessarily cover a month, so I bet you might not have a monthly-3 at page 2 (unless for February).
There is nowhere that mentions such strict requirement for the relationship between those numbers.
I don’t know, where you get the impression that more frequent snapshots will end you up with less to keep. As you can see, there are 3 snapshots on the March 9th, where I performed two additional manual snapshots for speed testing. Well today, the earlier two snapshots @08:15 and @01:00 have gone, but that was to be expected.
I don’t see the difference between having snapshots being automatically run or manually. So as of today, the repo looks like this:
However, I have to admit, that I don’t know, why it lists the hourly ones, since Hourly has been unset in the policy. Altough, I had played with the Hourly setting as well. Maybe the policy has been partially cached somewhere, or that might be a bug.
When I gave you impression that more frequent backup leads to less snapshots, I meant to start from scratch. If you already have this tons of backups, sure Kopia will not remove old ones already having an appropriate label. It’d be difficult for you to test, but if you expire every backup except the latest one, set interval to 23 hours, I think 3 months later you will still have only one or two backups, overwriting themselves over and over.
And the hourly, I don’t know what you mean by “unset”. Is it explicitly “0”? Otherwise, it might use it’s “parent” policy’s default value. I suggest you set it to 0 for determinism.
About the March 9th, you do see the two removed backups have no label on them right? I don’t know how Kopia treats manually triggered backups. Maybe they have special retention rules, because Jarek said unlabeled backups can’t stay. If user expects them stay because they explicitly manually trigger, this should be documented.
Well, I had this 1l/7d/4w/3m retension in right from the start, so its nothing I imposed on my repo afterwards. However, I do find it a bit strange, that it labels hourly snapshots, when I didn’t set up a retention policy for those.
As far as multiple snapshots per day go - I’d think, that Kopia always labels the latest snapshot of a retention “class”. So, if you have daily snapshots set, it will keep always the latest snapshot of that day. Makes sense, since the one you mentioned actually was one of the manual ones I did that day.
I have set up another repo to better monitor the snapshot labeling. This small repo has a snapshot interval of 6 hours, so 4 snapshots a day - and it has the same 1/7/4/3 retention policy as my main repo. I will monitor this for a few days, but what I can tell right away is, that it also labels the hourly ones, so this might be a bug.
So instead of using an explicit “0”, you left the hourly empty. I think Kopia will just use parent’s or pre-defined default values, which is 48. That’s why you have hourly label. If you set to “0”, you should not get hourly label anymore.
Anyway, IMO this retention system is too complicated for average users to fully comprehend and tweak with. And there isn’t simple way to do tiered retention without bloating numerous labels. I think Jarek should consider something like Duplicati’s retention policy.
I think the UX around policies could use improvements, for sure. Definitely something that will preview the impact of changes before they happen (in both CLI and UI). Ability to specify the retention spec as a single string is also a good idea.
I know this is an old thread and CrendKing might not be looking for answers anymore, but I had to spend a bunch of time carefully checking to find that their descriptions of the retention behavior do not follow from what was in the previous thread.
The key idea is: A snapshot is born protected by every kind of retention tag. When the last tag of one kind moves off of it, another kind can still protect it, so it will not be deleted yet. It is deleted when every kind of retention tag moves off of it.
Take the example with
snapshots every 10 minutes
retain latest 4
retain hourly 4
all other retention set to 0
Imagine starting the whole system from nothing at 0:00, with the files constantly changing (so we ignore the question of identical snapshots at present).
At 0:30, the number of snapshots retained reaches 4.
At 0:40, the latest-4 tag shifts off of snapshot 0:00 and it is deleted. The number of snapshots retained stays 4.
At 0:50, the latest-4 tag shifts off of snapshot 0:10 and it is deleted.
At 1:00, the hourly-1 tag shifts off of snapshot 0:50 but it is replaced by the hourly-2 tag. The latest-4 tag shifts off of snapshot 0:20 and it is deleted.
…
At 1:30, the latest-4 tag shifts off of snapshot 0:50, but it still has the hourly-2 tag. Nothing is deleted. The number of snapshots retained reaches 5.
…
At 2:00, the hourly-1 tag shifts off of snapshot 1:50, but it is replaced by the hourly-2 tag, which shifts off of snapshot 0:50, but it is replaced there by the hourly-3 tag. (And the latest-4 tag shifts off of snapshot 1:20 and it is deleted.)
…
At 2:30, the latest-4 tag shifts off of snapshot 1:50, but it still has the hourly-2 tag. Nothing is deleted. The number of snapshots retained reaches 6.
Now how do identical snapshots complicate this? They don’t. The retention policy treats them just like any other. When you stop changing the files, the retention tags keep marching forward.
