How can I optimize Restore efficiency?

Support
1

I’m running btrfs and using Timeshift to snapshot my system, including my @home subvolume.
With Kopia, I’m backing up /run/timeshift/ to Google Drive via rclone. This works great, happens very fast every time. Speed and efficiency is top notch.

This weekend I wanted to test my backup/restore system to make sure it works as intended. I had a mishap previously, where my backups were actually no help in restoring.
So, I made a bare metal image of my machine, booted a live system, and installed a fresh distro with Timeshift.

Then I fired up Kopia, connected to my repo, and began restoring /run/timeshift.

This was taking an incredibly long time to finish- in the area of multiple hundreds of hours. I canceled the task after it had restored 3% in 8 hours.

I’ve got something like 195GB to restore, and I was using --parallel=32. I see something called shallow restore, but the docs don’t say much that I could find, so I don’t know what that’s about. I didn’t use it, but maybe it would help?

Basically, is there anything I can do to speed up the restore procedure?
Maybe change caching, or use some shallow restore, or something else?

As it is now, it’s almost more efficient to run a bare metal backup once a week. That’s a hassle because it isn’t automated, but at least I can pull it off over the course of about 10 hours. I tried uploading these 256GB images using Kopia, but haven’t noticed deduping to work on these (Encrypted drive, so dedupe probably can’t help anyway).

One final note, since this is a btrfs timeshift, I think it’s just using hardlinks all over the place, but Kopia sees the hard links as their full apparent size, which substantially increases the size of these backups.

EDIT:
Maybe, would it help to reduce the number of snapshots I keep in history? Maybe run maintenance and reduce the entire repo size?

I’m just trying to figure out what would improve the speed at which I can get these files restored.

2

Restore generally benefits from low-latency access to repository. Google Drive is not really known to be fast, and rclone will add its own latency.

We actually have an experimental google drive provider in the works, if you want to try it - it may be faster. It’s not published yet, but you can grab early builds at: GitHub - kopia/kopia-test-builds: Pre-releases for Kopia UI

3

BTW, I have been thinking of automatically adjusting caching behavior during restore, to download bigger chunks at a time instead of lots of small requests. This has some trade-offs, but should in principle speed up the restore by a lot, especially for slow backends.

(to be clear, this is just an idea, not something that exists)

4

Thanks, I do like to test your builds when they come out. Snapshot sizes are cool.

I’ll check out the native Google drive soon.

Also, when interacting with Google drive via rclone, are you passing --fast-list? I think there’s a lot of opportunity to leverage the way rclone interfaces with GDrive already, to maybe build a temporary index/mapfile of the chunks needed for this specific restore, then grabbing those with rclone in the kopia cache. Maybe there’s something there.

I don’t think this has anything to do with it, but could it be faster to mount a snapshot and copy it with rsync -av or something?

I do like your idea on building a cache on restore. I like that idea quite a bit.

5

Other than using a different backup medium, are there any optimizations I can make during the restore process? If the problem truly lies with using Google Drive, I’m not sure I see the benefit in using cloud storage for backup unless you only expect to retrieve an occasional file rather than perform a full restore.

Any flags I can pass this thing to speed it along? Do something with caching? Is “Shallow” restore something that would help? (Where are the docs on shallow restore? I don’t understand what it is)

1 Like
6

Google Cloud is a very different class of cloud storage than say S3, GCS, B2 or Azure. It’s not optimized for programmatic access and by its nature much slower. I would recommend one of the mentioned providers which are much faster and very scalable.

7

Ah, very interesting, thank you.

If I move from Google Drive to Google Cloud, do you have any kind of transactional data estimates? For example, ingress is free, egress is charged.

I’d be interested to know how much it’ll cost me to snapshot and perform maintenance- not necessarily restore.

I realize that’s probably difficult to estimate, so maybe I just need to experiment on my own and find out.

I appreciate the advice, thank you very much.

Final note, do you have any personal opinion on which of those might be fastest/lowest cost relative to the others?

8

I personally have been a long-time Wasabi user, because they have a simple pricing model - you only pay for storage (with minimum storage duration) not for data transfer or API calls, so when you need to restore a lot, you’re not faced with a one-time major egress cost (typically around 10 cents/GB).

The list price is $5.99/TB/month and that’s close to what I’ve been paying monthly for my main repository which is around 800-900 GiB now.

For comparison, with egress cost added, restoring that amount of data would cost $90.

9

I am a happy Wasabi customer aswell. Just wanted to note that Wasabi’s guidelines do not allow endless egress. Basically you are asked to not download more than your current active storage volume.

https://wasabi.com/paygo-pricing-faq/#free-egress-policy

10

Thank you for the advice.

I had already signed up for B2 when you mentioned Wasabi. I began snapshotting, and saw that my API and daily download usage were rising, but that my entire repo size was only 24 GB with my 207GB snapshot. While I don’t like the idea of paying for egress, I think that might be cheaper than 5.99/month at Wasabi since I’m storing such a small amount.

I went back to my Gdrive repo to look at what was going on over there, and found that I had migrated my snapshots and that an old (unused) user was the owner of maintenance, and I hadn’t been performing maintenance for something like a month. I bet that was ruining my restore performance.

Also, I may just keep this repo on Gdrive and keep my snapshots as efficient as possible. This is just crash protection for me, so I don’t need snapshots from way back in the past. I just need a quick and reliable recovery solution in the event that my work machine goes down.

The plan is that I’ll snapshot to gdrive, then if I need to restore, I’ll rclone copy the whole repo down to a flash drive and connect to that for the restore operation.

I would kopia snapshot directly to a flash drive, but I’m concerned that I’ll wear out the flash too quickly and risk failure of my backups when I need them. What are your thoughts on this?

11

How to restore a copy from “Google Drive/Google Drive Shared Drives” extremely fast?

  1. Download Gdrive for Windows

  2. Mount Google Drive as Disk with Letter available from Windows Explorer.

  3. Connect the copy to filesystem repository and point to the path to the repository on the mounted drive.

For a repository that has 500 snapshots and 10TB, recovering a 2GB folder was reduced from 3 hours to 5 minutes.