GCS "download worldwide destinations" charges

Support
1

Hi. We have 9.5tb in nearline storage.
This month’s bill from Google has £90 GBP in “Download Worldwide Destinations” charges.
I haven’t done any verifying or downloading.
Is this expected, from maintenance or something?

Here is the maintenance info. Is full-rewrite-contents downloading lots of data?

Owner: root@backup
Quick Cycle:
  scheduled: false
Full Cycle:
  scheduled: true
  interval: 24h0m0s
  next run: 2022-08-04 09:31:41 BST (in 22h12m14s)
Log Retention:
  max count:       10000
  max age of logs: 720h0m0s
  max total size:  1 GiB
Recent Maintenance Runs:
  full-rewrite-contents:
    2022-08-03 10:07:01 BST (1h7m29s) SUCCESS
    2022-08-02 14:21:04 BST (2h9m27s) SUCCESS
    2022-07-29 15:10:31 BST (2h9m26s) SUCCESS
    2022-07-27 09:17:58 BST (2h18m15s) SUCCESS
    2022-07-24 14:35:12 BST (2h50m34s) SUCCESS
  snapshot-gc:
    2022-08-03 09:31:41 BST (35m19s) SUCCESS
    2022-08-02 13:44:59 BST (36m4s) SUCCESS
    2022-07-31 13:53:56 BST (34m28s) SUCCESS
    2022-07-29 14:33:32 BST (36m58s) SUCCESS
    2022-07-28 14:14:23 BST (34m34s) SUCCESS
  cleanup-epoch-manager:
    2022-08-03 11:17:46 BST (0s) SUCCESS
    2022-08-02 16:30:33 BST (2s) SUCCESS
    2022-07-31 14:31:17 BST (1s) SUCCESS
    2022-07-29 17:19:58 BST (0s) SUCCESS
    2022-07-28 14:51:57 BST (1s) SUCCESS
  cleanup-logs:
    2022-08-03 11:17:46 BST (0s) SUCCESS
    2022-08-02 16:30:33 BST (0s) SUCCESS
    2022-07-31 14:31:17 BST (0s) SUCCESS
    2022-07-29 17:19:58 BST (0s) SUCCESS
    2022-07-28 14:51:56 BST (0s) SUCCESS
  full-delete-blobs:
    2022-08-03 11:14:30 BST (3m15s) SUCCESS
    2022-07-31 14:28:25 BST (2m51s) SUCCESS
    2022-07-28 14:48:58 BST (2m58s) SUCCESS
    2022-07-25 14:46:29 BST (4m15s) SUCCESS
    2022-07-23 14:35:20 BST (3m35s) SUCCESS
  full-drop-deleted-content:
    2022-08-03 11:14:30 BST (0s) SUCCESS
    2022-08-02 16:30:32 BST (0s) SUCCESS
    2022-07-31 14:28:25 BST (0s) SUCCESS
    2022-07-29 17:19:58 BST (0s) SUCCESS
    2022-07-28 14:48:58 BST (0s) SUCCESS

Here’s the billing from Google:

image
image974×86 3.35 KB

I am a bit staggered at a cost of £86 GBP (approx $105 USD) to download 881gb of data.

2

Ah, you have discovered the scam of the Big 3 cloud providers: excessive egress (downloading) costs!

To my knowledge, Kopia uses egress in two ways. First is through repository compaction, where Kopia rewrites blobs when cleaning up the repository after you have deleted files. This is done during the daily full maintenance. Second is when Kopia runs snapshot verify, where Kopia downloads the metadata. This is run by Kopia daily during full maintenance also.

I suspect your egress use (881.511 GB downloads during the month) is a combination of the two: you are backing up files that change regularly (which then leads to a lot of rewriting blobs) and, given your repo size, you have a large amount of metadata. It is probably more of the former than the latter.

It is worth having a conversation whether Kopia should have a “low egress” full maintenance option where Kopia less aggressively dose compaction and/or less frequently runs snapshot verify. What do you think, @jkowalski?

Beyond changing the mechanics of Kopia, there are a few things you can do.

First, make sure you are on the latest version of Kopia. Kopia introduced automatic compression of metadata in v0.10.7. If you are on an earlier version of Kopia, it is likely that your metadata is uncompressed. Compressing it will reduce the amount of egress snapshot verify uses.

Second, you can either disable full maintenance or run it less frequently. See the page Maintenance | Kopia (you will need to do this via CLI). Running full maintenance less frequently will reduce the number of times snapshot verify is run, but it will not reduce the amount of compaction. Disabling full maintenance will stop both. Disabling full maintenance is not recommended because it is essential to ensuring your repo stays clean and compact, which improves performance and decreases the chances of data corruption.

Finally, you can consider moving to a cloud storage provider that does not have excessive egress costs. Amazon, Azure, and Google Cloud Storage all charge $0.08 or higher per GB of egress (Amazon and Azure give free 100GB egress every month, and Google Cloud Storage will give free 100GB egress every month starting in October). Other providers either do not charge anything for egress or the egress cost is much lower:

  • Cloudflare R2 – no egress costs
  • Wasabi – no egress costs (subject to their fair use policy, but Kopia will never use enough egress during full maintenance to violate that policy)
  • Oracle Cloud Infrastructure – 10TB per month of free egress
  • BackBlaze B2 – $0.01 per GB of egress
  • Scaleway – €0.01 per GB of egress
  • IDrive E2 – no egress costs (subject to their fair use policy, but Kopia will never use enough egress during full maintenance to violate that policy)
  • Synology C2 – no egress costs (subject to their fair use policy, but Kopia will never use enough egress during full maintenance to violate that policy)
  • OVH – $0.011 per GB of egress

Beyond egress, most, if not all, of these cloud storage providers also charge less for storage and/or retrieval than the Big 3.

Good luck.

2 Likes
3

Thank you very much for such a thorough reply. Yes I did find out that ‘network egress’ is astonishingly expensive with the top 3. I can’t believe just how expensive it would be to restore a few terabytes of data. I have set up an account with BackBlaze already, but I will look at the others too.

Thanks once again, it’s greatly appreciated.