Hello,
as seen in the docs I created an access key without permission to delete objects and bucket, is possible that this error that I am having while creating the first snapshot is related to this?
It looks like your policy is wrong as you blocked deleting objects.
Post your policy here.
I did create a restricted access key as follows
my kopia policy is the standard one only with the zstd compression override
Citing the docs:
The easiest solution is to generate a key without any
deletepermissions, and to configure Kopia to use that key
Enabling restricted keys does not require any changes in your Kopia workflow, since Kopia does not need to change its behavior at all.
What am I missing 
?
iDrive (I use it too) does not allow to create policies granular enough for this to work. As what you need is:
"Effect": "Deny",
"Action": [
"s3:DeleteBucket",
"s3:DeleteBucketPolicy",
"s3:DeleteBucketWebsite",
"s3:DeleteObjectVersion"
],
You have to deny deleting s3:DeleteObjectVersion but not object itself. In iDrive you deny all including s3:DeleteObject
kopia docs are not clear enough about it.
What works with iDrive perfectly is Object Lock. Enable COMPLIANCE lock mode on your bucket and set life cycle rules for old versions - then follow kopia guide.
1 Like
oh thats great and good to know!
I created a new bucket with object lock (I left powered off the “Object Retention” slider since as I understood thats the default behavior and kopia will do by itself with the passed flags in the docs --retention-mode COMPLIANCE --retention-period <retention time>
May I ask you if you set some lifecycle rules over here?
And yes, i guess that with iDrive we dont get the complexity of aws s3 but we also dont get the granularity and personalization
you should set rules otherwise your repo will grow forever:)
Let’s say you want protection for 100 days, set 100 days lock and then here set Rules for older versions to 101 days. It means that after 101 days if there are any old versions out of lock they will be deleted automatically.
the 100 days lock is the kopia cli command right? Nothing to do on iDrive
I was going to set the Rules for current version (instead of old), I guess that would have deleted all my files then!
correct.
What kopia does is maintaining repo (business as usual) - in addition it renews 100 days lock on all objects when you run full maintenance so do not forget:
Enable Object Lock extension
Run: kopia maintenance set --extend-object-locks true Note that the full-interval must be at least 1 day shorter than the retention-period or Kopia will not allow you to enable Object Lock extension
and make sure you run full maintenance. You are actually protected for 100 days from last full maintenance run
Otherwise some old objects after 100 days would leave lock and could be deleted.
Now when some object(s) is deleted (by kopia as part of pruning or by evil actor) it becomes version (protected by lock for 100 days). After 100 days living as a version it can be deleted by lifecycle rules.
and make sure you run full maintenance.
if I am not wrong kopia already has a full scheduled every 24h so nothing further to do by my side (?)
the option for the current version so should not be set at all, the one for old versions is enough?
Yes this is default but only enforced when you use GUI. In case you use CLI you have to make sure you run it. It is enough to run daily backup - maintenance will be triggered if needed by schedule.
But you have to enable Object Lock extension.
Yes. Only set it for old versions.
My tip is to play with this including restore using --point-in-time with some small dataset first. you can login to iDrive web UI and delete all objects from bucket - then try to restore something. Use short lock period for experiments.
The reason to be careful is that it is not possible to delete any locked objects. Only way is to terminate your account… So if you set lock for 365 and upload 10TB of data and things are not working properly you have to wait 1 year to delete it.
1 Like
Also please share your experience and findings using it. I am very interested how it works for others.
Sure, here I am, I was doing a bit of research, still in the learning phase
So since I wanted to switch to a object lock bucket I had to delete my whole bucket and reupload everything, I did this, deleted the kopia container appdata and created a new repository (object lock can be set cli only)
kopia repo create s3 --endpoint= --bucket mybucket --access-key= --secret-access-key= --retention-mode COMPLIANCE --retention-period 12d
after executing this command I kept having this error here
ERROR cannot initialize repository: unexpected error when checking for format blob: unable to complete GetBlob(kopia.repository,0,-1) despite 10 retries: The specified method is not allowed against this resource.
now, sadly, I do not know how to reproduce this, since I kept having this and the only thing I could do was re-clearing the appdata, after it just worked, I doubt was a repository error since I also tried different access keys (with full permissions) or different new buckets. Then I just thought idrive didnt support object lock.
Then after a further reinstallation it just worked, I went with 12days of retention
kopia repo create s3 --bucket <bucket name> --access-key=<access key> --secret-access-key=<secret access key> --retention-mode COMPLIANCE --retention-period <retention time>
with kopia repo status I double checked retention was enabled:
Blob retention mode: COMPLIANCE
Blob retention period: 288h0m0s
enabled the object lock plugin
kopia maintenance set --extend-object-locks true
and checked with kopia maintenance info if the full interval was short enough and full was enabled at all
root@kopia:/app/config# kopia maintenance info
Owner: root@kopia
Quick Cycle:
scheduled: false
Full Cycle:
scheduled: true
interval: 24h0m0s
next run: 2024-01-27 20:40:13 CET (in 22h10m8s)
full seems already enabled and scheduled every 24h (I left 24h as it is) but noticed quick is disabled, so I went with
kopia maintenance set --enable-quick true
I can say this has been my whole journey in this, as always thank you very much for the precious help
edit: actually just for safety I checked everything was working with kopia repository validate-provider , a very useful command in my opinion
The reason to be careful is that it is not possible to delete any locked objects. Only way is to terminate your account… So if you set lock for 365 and upload 10TB of data and things are not working properly you have to wait 1 year to delete it.
And about this I did not know really, also in the docs the part in which ransomware flooding the repository with locked data was new and informative to me
2 Likes
All together at least for kopia users it is a bit unchartered waters as you discovered initially with trying to setup “simple” delete policies. Not many people go there:)
Documentation is not really up to date and even worse project feels a bit abandoned.
I am not sure myself if it is worth to take risk using kopia. It is fantastic idea and initial implementation but now it looks like death by thousand cuts. Dying project:(
I have to say I noticed, the forum is quite empty and the slack has many questions not answered, many doubts I have I did find already posted threads unanswered and not reported in the docs (for instance this), little doubs, the big ones you were very kind to answer but without your replies I would have stopped at the first important non-answered doubt.
We also have to say there are few alternatives, kopia is the fastest and has a ton of features, restic is not so rich of features, duplicati is not so trusted and duplicacy seems to be the new de facto standard? Really there are so few options in backup frontends I would not even know which alternative to consider
May I ask you why you have this feeling / do not suggest it? Since as I said as of today there are really no competitors
edit: I also guess is the first time in the open source world that I would have really loved to have the skills to contribute
1 Like
I also use rustic (restic written in rust) - it is command line only but have some cool features like cold storage support (AWS Glacier).
In general for important data I prefer to use two different programs - you never know what bugs can surface one day etc.
And I hope I am wrong about kopia:) and it will have great future. It is really nice piece of software - probably the best in cloud backup category.
Redundancy is always the golden rule, I saw restic but they dont support many things (6-7 years to support compression, lock not supported etc)
As you see I dont even care so much about gui even bacuse also kopia’s gui is not such rich, for me a clear cli is the most important baseline
Does rustic merge well with idrive? So you have 2 buckets with same data to my understanding
Yeah restic is very conservative and its development is very slow - it has some advantages though. It is rock solid and very careful with implementing new features. rustic was created I think due to frustrations how slow restic dev is. It is fully compatible with restic - so you can switch them anytime. And has a lot of features added which are still pending in restic e.g. can resume restore, works lock free (similarly like kopia), supports config files etc. And its developer is always happy to answer questions and investigate issues:)
It is worth to try IMO - here some highlights - FAQ - rustic user documentation
rustic uses rclone for all its connectivity - works perfectly with iDrive S3. rclone has built in restic API so this integration is very close and efficient.
Thanks for the heads up, I’ll look into rustic too even if is still on beta (just as kopia I’d say)