Newbie questions

General
1

Found Kopia recently and start loving it. I have some basic questions and suggestions

  1. Is Kopia UI meant to be the only tool to support multi repository paradigm in the future? What if I’m using Kopia in a terminal only environment? Do I have to use --config-file every time?
  2. Does Kopia UI run the scheduled backup and maintenance jobs for all connected repositories? So essentially I should autorun and keep Kopia UI in notification area as a “service”? What if I’m in Linux server? Do I have to setup cron job myself?

And if Kopia UI is considered essential part of the Kopia system, can you consider adding it to the scoop bucket?

  1. The only option for scheduled snapshotting is interval, but I can’t specify the time of day, e.g. 23:30. Are users required to use OS scheduling tools to do so?
  2. I think the “max pack length” of a repository is hardcoded as 20 MiB. What if the data source of that repository are large videos (which are all big and uncompressable)? Wouldn’t allowing user to choose a bigger pack length be more efficient?
  3. Similar to 3, what would be more efficient hash and splitter algorithms for that repository? Or should be default BLAKE2B-256-128 and DYNAMIC-4M-BUZHASH be sufficed and users should not worry about it?
  4. In Kopia UI, the “global” policy of each repository has username and host being *, which I always connect to a repository with a specific username and host. I was assuming when I create a policy for directory I could just use * that applies to any connected users. But after many tries, I think * is not support for directory policies. Am I correct?
  5. Need a little help on a ignore rule. Suppose I have a directory /a/b. This directory has a bunch of cache subdirectories that I want to ignore. They are /a/b/Cache, /a/b/PicCache, /a/b/CachedImages. Basically all have word Cache in the name. And I want to limit the rule just to /a/b. I tried /a/b/*Cache*/ but it doesn’t work. Only *Cache*/ works, but it applies to any parent.
  6. It is common to have some exclusively opened lock files. They can’t be backed up. So I have to set ignore file errors to yes. But this may shroud real backup errors. Is there an option to just ignore exclusively opened files, or maybe supporting VSS?
  7. And a little bug: if I run any command --help, I always get empty cli-logs and content-logs in the current directory. They should only be created and updated in user home directory, no?

Thanks.

1 Like
2

Thanks for your interest in Kopia. What great questions. Let me answer them one by one.

  1. Yes, when using kopia CLI you’re expected to pass --config-file to indicate which repository you’re working with.

  2. Yes, KopiaUI does scheduled backup and maintenance. It does it by launching kopia server with some flags in the background, which does maintenance, scheduled backups and serves HTML UI. On Linux you can either use cron or run kopia server as a daemon.

  3. Setting time of day should be supported in the CLI, it it not be available in the UI yet, but patches to add it are welcome.

  4. Max pack length is slight misnomer, it’s a threshold above which the pack will be saved to the storage, so packs will almost always be slightly bigger. Kopia will keep multiple buffers for packs in memory, while the packs are being assembled and asynchronously flushed to storage, so if you have lots of RAM in all your clients you may benefit from having slightly larger packs. OTOH I would not go overboard because Kopia will sometimes rewrite packs that become fragmented, so that will require downloading them all which may be more efficient with smaller packs.

  5. You can measure that using kopia benchmark crypto and kopia benchmark splitter. The defaults should be good for most use cases.

  6. There are 4 kinds of policies, progressively more precise.

global (@:)
@host:
user@host:
user@host:/path

There is currently no policy for *@host:/path, if that’s what you’re asking for.

  1. Yeah, those ignore rules are tricky. You can always put policies directly in the affected folders, if that’s any help. Alternatively you can put file .kopiaignore (similar format to .gitignore) in the affected directory. BTW. There’s kopia snapshot estimate which can help when setting up those rules.

  2. We currently don’t have option to selectively ignore errors based on the error code, but that sounds like a good idea. BTW. I’m working on support for hooks which should be able to support custom pre&post folder actions, which should (eventually) allow things like VSS.

  3. Would you mind filing a bug on Github for that?

3

Thanks for the reply.

For 5. I did that at beginning. Actually all the defaults are the fastest for me, so no change needed. I was using Duplicati and it has the “remote volume size” option. I chose a bigger size for the repository whose files are mainly videos (large, uncompressible) so was wondering the equivalent in Kopia. But I wasn’t sure if there is any since the storage abstraction between the two are different.

For 6. I was actually looking for *@*:/path. Didn’t know about the hierarchy. Now you mentioned it, I think the UI could be changed to reflect the hierarchy (e.g. You can’t put in path without first filling out user and host).

For 7. Oh I thought the policy path must match the snapshot path. Turns out they don’t need to. So I could, as you said, put *Cache*/ under /a/b and snapshot / and it would work. Nice.

For 9. Will do.

I know you are aware of that, but all your information here should go into the documentation, especially 5, 6 and 7. Could save users lots of time on trial and error.

Thank you again for the wonderful piece of work.

4

Oh, forgot to mention. In Windows, the mount function does not seem to work. Here’s what I did:

  1. kopia snapshot list. Pick a snapshot ID.
  2. kopia mount <snapshot_id> x: --log-level=debug
  3. Log:
2020-12-14 13:14:18.596 D [manifest_manager.go:288] listing manifest contents
2020-12-14 13:14:18.602 D [mount_webdav.go:32] creating webdav server...
2020-12-14 13:14:18.604 D [mount_net_use.go:41] running c:\windows\system32\net.exe [net use x: http://127.0.0.1:3132]
2020-12-14 13:14:18.623 D [mount_net_use.go:44] net use finished with  exit status 2
2020-12-14 13:14:18.623 D [mount_webdav.go:60] web server finished with http: Server closed
2020-12-14 13:14:19.124 D [logging_storage.go:28] [STORAGE] GetBlob("kopia.maintenance",0,-1)=({2141 bytes}, <nil>) took 0s
2020-12-14 13:14:19.124 D [maintenance_run.go:80] full manintenance cycle not enabled
2020-12-14 13:14:19.124 D [maintenance_run.go:92] quick manintenance cycle not enabled
2020-12-14 13:14:19.124 D [maintenance_run.go:171] not due for maintenance
2020-12-14 13:14:19.125 D [logging_storage.go:87] [STORAGE] Close()=<nil> took 0s
2020-12-14 13:14:19.126 D [memory_tracking.go:31] in use heap 3375104 (delta 3375104 max 0) stack 524288 (delta 524288 max 0)

I also went to made sure net use itself is working by sharing a folder and net use map to a drive. Something might be wrong with webdav.

Should I create ticket in GitHub?

5

Thanks, I’ll be happy to review and merge PRs that improve the documentation. Just saying :slight_smile:

6

Hi @jkowalski. I have two more questions.

  1. What happens if I change the compression policy after a snapshot is taken? For example, I create a snapshot with no compression, change the global policy to use s2-default compression, then create another snapshot. Does Kopia retrospectively go back to compress everything with s2-default, or only compress new content?
  2. Is there a command to show the after-compression size of each directory? It could be useful if some directories are not compressible at all then I’d just disable compression for them.