Experimenting with rclone

Support
1

Hi,

I’m experimenting kopia with rclone to try to backup data to 1fichier, a very cheap file hosting provider that rclone can talk with:
https://rclone.org/fichier/

here is the result:

kopia repository create rclone --remote-path 1fichier:/din/kopia/test
Enter password to create new repository: 
Re-enter password for verification: 
Initializing repository with:
  block hash:          BLAKE2B-256-128
  encryption:          AES256-GCM-HMAC-SHA256
  splitter:            DYNAMIC-4M-BUZHASH
kopia: error: cannot initialize repository: unable to write format blob: unable to write format blob: MOVE kopia.repository.f-5577006791947779410: 403, try --help

I checked on 1fichier console that the file was created in the right place.
For what it’s worth, I used the same remote to do the same thing with restic and it worked.

Is there something I can try to solve this?

Thanks

2

I tried to run kopia with rclone to b2 backblaze and it works fine. So the issue seems to be storage provider specific. Maybe I should talk to the rclone guys. But can you think of a specific feature that kopia needs and that 1fichier could not provide ?

I use kopia 0.7.3 with rclone 1.53.3-1+b2 on debian bullseye

3

with the new 0.8.0-beta1 release:

./kopia repository create rclone --remote-path 1fichier:din/kopia/test2 --log-level=info
Enter password to create new repository: 
Re-enter password for verification: 
Initializing repository with:
  block hash:          BLAKE2B-256-128
  encryption:          AES256-GCM-HMAC-SHA256
  splitter:            DYNAMIC-4M-BUZHASH
kopia: error: cannot initialize repository: unable to write format blob: unable to write format blob: unable to complete PutBlob(kopia.repository) despite 10 retries, try --help
4

Using kopia 0.7.3 and rclone 1.53.3-1+b2:

In order to bypass the MOVE, I created a local repo that I synched to its final online destination (1fichier) with rclone.
I could then get a connection with:

kopia repository connect rclone --remote-path 1fichier:din/kopia/test

but then:

kopia snapshot create /data/test/dir/
 | 8 hashing, 0 hashed (23.1 KB), 0 cached (0 B), 0 uploaded (0 B), 0 errors, es / 0 hashing, 916 hashed (21.2 MB), 0 cached (0 B), 0 uploaded (0 B), 0 errors, estimated 22.1 MB (96.1%) 8s lefterror running maintenance: error updating maintenance schedule: MOVE kopia.maintenance.f-3916589616287113937: 403
kopia: error: unable to close repository: error flushing: error writing pending content: error writing pack content: error writing pack: can't save pack data content: MOVE p80/f7d/7acd2a050046839c9e42a65ce02.f-6334824724549167320: 403, try --help

Data is uploaded, but obviously, 1fichier doesn’t allow MOVE. I can think of two ways to work around this:

  • One solution could be to backup locally and then sync online.

  • Or: according to Overview of cloud storage systems, rclone already has a procedure to deal with that issue. Would it be possible to use it with kopia?

5

So the lack of MOVE would be a problem with current implementation, because move is used for atomic upload - Kopia will first upload a file named something.tmp and when done it will rename/move it to something.f. Only files ending with .f are treated as fully uploaded for safety reasons (in case the client died half-way through we don’t want to rely on partial contents).

Somebody could try to add an option in PutBlobInPath in webdav_storage.go to do direct upload to an .f file bypassing MOVE, but that would probably be unsafe - I’m not sure how rclone would handle atomicity of webdav uploads. If rclone webdav (which kopia uses as its abstract API) is translated as atomic upload to the 1fichier backend which is also atomic, this will in fact work and be safe.

6

Actually, I took a look at rclone page which says:

1Fichier does not support modification times.

This is actually a dealbreaker today, since Kopia relies on modification times that follow wall clock time for safety of its operations.

7

End of story.

Thanks a lot for having a look! You’re doing a amazing job.