Kopia eating 100% of CPU

compumatter · July 23, 2024, 8:22pm

I do not have any added compression options turned on and have not tweaked any settings that would effect performance. This is out of the box.

I’m running

a decent CPU AMD EPYC 7232P 8-Core Processor
NVME operating system drive

The snapshot target is a motherboard connected SATA Platter drive

This is the script I’ve started:
kopia snapshot create /MYDATA --log-file=/var/log/internal.log --config-file=/CM/backups/internal.config -p mypassword

This is where my CPU is riding and sometimes higher…

Current Policies:

Policy for (global):

Retention:
  Annual snapshots:                     3   (defined for this target)
  Monthly snapshots:                   24   (defined for this target)
  Weekly snapshots:                     4   (defined for this target)
  Daily snapshots:                      7   (defined for this target)
  Hourly snapshots:                    48   (defined for this target)
  Latest snapshots:                    10   (defined for this target)
  Ignore identical snapshots:       false   (defined for this target)

Files policy:
  Ignore cache directories:          true   (defined for this target)
  No ignore rules:
  Read ignore rules from files:             (defined for this target)
    .kopiaignore
  Scan one filesystem only:         false   (defined for this target)

Error handling policy:
  Ignore file read errors:          false   (defined for this target)
  Ignore directory read errors:     false   (defined for this target)
  Ignore unknown types:              true   (defined for this target)

Scheduling policy:
  Scheduled snapshots:
    None.
  Manual snapshot:                  false   (defined for this target)

Uploads:
  Max parallel snapshots (server/UI):   1   (defined for this target)
  Max parallel file reads:              -   (defined for this target)
  Parallel upload above size:      2.1 GB   (defined for this target)

Compression disabled.

No actions defined.

OS-level snapshot support:
  Volume Shadow Copy:               never   (defined for this target)

Logging details (0-none, 10-maximum):
  Directory snapshotted:                5   (defined for this target)
  Directory ignored:                    5   (defined for this target)
  Entry snapshotted:                    0   (defined for this target)
  Entry ignored:                        5   (defined for this target)
  Entry cache hit:                      0   (defined for this target)
  Entry cache miss:                     0   (defined for this target)

Any idea on how I can limit CPU usage or offload some of this to memory?

What sayeth the group?

dimejo · July 24, 2024, 12:50pm

Which algorithms are you using for your repository? Any non-default settings? Please show kopia repo status and if possible run kopia benchmark hashing and kopia benchmark crypto when the system is not under load.

Try setting --parallel=1 and increase if needed.

compumatter · August 3, 2024, 3:12am

My sincere apologies for this delay in responding to your help. The reality is I’ve been scraping every bit of knowledge I can gleen on backing up a typical business environment (server, multiple clients, internal backup drive and cloud b2 backblaze - on the same machine) and as a result have had an elephant by the tail.

That said, I need to circle back to this because I continue to run into it. At first I thought maybe the problem had to do with running snapshots from the systemd environment so I used systemd commands like this to force down the load.

[Service]
CPUWeight=90
CPUQuota=50%
IOWeight=60

That does keep the CPU numbers at or below 50ish percent with a single snapshot running. I’m guessing it just ‘slows down’ the kopia functionality lock step. However if there are 2 snapshots running ie; internal repo and cloud repo, it hits 100% between them and of course any other windows clients on the network I’m guessing would bump CPU further yet.

Unlike systemd, Kopia has the view from the top. So I am hoping there is a solution within Kopia to fine grain its resource consumption as client demands increase.

First I will answer I have made no changes to the out of the box algorithms. No non-default settings (scheduling and retention aside)

Now onto your thoughtful suggestions.

–parallel=1

I have not been certain about where to apply this setting. I have found a few kopia policy settings that look interesting. The first one may have the effect you recommended. Please confirm. Limiting parallel snapshots might make sense but I can see where that might be an issue when 10 client devices are scheduled to do snapshots. Staggering times would be required at the least.

For your info, I decided to take a look at the current policy settings for internal/external backups and it shows:

kopia policy show --global --config-file=/backups/kopia/configs/internal.config | grep arallel
  Max parallel snapshots (server/UI):   1   (defined for this target)
  Max parallel file reads:              -   (defined for this target)
  Parallel upload above size:      2.1 GB   (defined for this target)
kopia policy show --global --config-file=/backups/kopia/configs/external.config | grep parallel
  Max parallel snapshots (server/UI):   1   (defined for this target)
  Max parallel file reads:              -   (defined for this target)
  Parallel upload above size:      2.1 GB   (defined for this target)

The above surprises me. Max parallel file reads says -. This page Frequently Asked Questions | Kopia indicates it’s supposed to be max logical cores of my machine’s cpu which in this case is 16.

Anyway, I’ll start with the --max-parallel-file-reads=1 and keep testing the results.

–max-parallel-file-reads
–max-parallel-snapshots
–parallel-upload-above-size-mib

kopia repo status (–config-file=internal/external)

I use local and cloud backups on the same machine and so have had to create dedicated config files and not the default config file that kopia repo status responds to.

So I generated two kopia repo status files (one for each)

kopia repo status --config-file=/backups/kopia/configs/internal.config

Config file:         /backups/kopia/configs/internal.config

Description:         SM Internal Backup
Hostname:            servermatter
Username:            root
Read-only:           false
Format blob cache:   15m0s

Storage type:        filesystem
Storage capacity:    21.9 TB
Storage available:   4.5 TB
Storage config:      {
                       "path": "/BACKUPS/DATA",
                       "fileMode": 384,
                       "dirMode": 448,
                       "dirShards": null
                     }

Unique ID:           a988958e68ee7dff9a23557427a6245b42c3445da9fc8e690e3388a152da4152
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      3
Content compression: true
Password changes:    true
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 4

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin:    4h0m0s
Epoch checkpoint every:  7 epochs

kopia repo status --config-file=/backups/kopia/configs/external.config

Config file:         /backups/kopia/configs/external.config

Description:         SM External Backup
Hostname:            servermatter
Username:            root
Read-only:           false
Format blob cache:   15m0s

Storage type:        b2
Storage capacity:    unbounded
Storage config:      {
                       "bucket": "compumatter",
                       "keyID": "004fd3af409900000000001",
                       "key": "*******************************"
                     }

Unique ID:           5e9b51d591f36590a9c1d4f88bbf6dd71d3693af44f750ac536afe93dd2c6bcd
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      3
Content compression: true
Password changes:    true
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 4

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin:    4h0m0s
Epoch checkpoint every:  7 epochs

kopia benchmark hashing

Benchmarking hash 'BLAKE2B-256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2B-256-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA224' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-224' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-256' (100 x 1048576 bytes, parallelism 1)
     Hash                 Throughput
-----------------------------------------------------------------
  0. BLAKE3-256-128       10.8 GB / second
  1. BLAKE3-256           10.7 GB / second
  2. HMAC-SHA256          4.9 GB / second
  3. HMAC-SHA256-128      4.9 GB / second
  4. HMAC-SHA224          4.9 GB / second
  5. BLAKE2B-256-128      2.6 GB / second
  6. BLAKE2B-256          2.4 GB / second
  7. BLAKE2S-256          1.9 GB / second
  8. BLAKE2S-128          1.9 GB / second
  9. HMAC-SHA3-224        1 GB / second
 10. HMAC-SHA3-256        1 GB / second
-----------------------------------------------------------------
Fastest option for this machine is: --block-hash=BLAKE3-256-128

kopia benchmark crypto

Benchmarking hash 'BLAKE2B-256' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2B-256' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2B-256-128' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2B-256-128' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-128' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-128' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-256' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-256' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256-128' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256-128' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA224' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA224' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256-128' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256-128' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-224' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-224' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-256' and encryption 'AES256-GCM-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-256' and encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (100 x 1048576 bytes, parallelism 1)
     Hash                 Encryption                     Throughput
-----------------------------------------------------------------
  0. BLAKE3-256-128       AES256-GCM-HMAC-SHA256         1.4 GB / second
  1. BLAKE3-256           AES256-GCM-HMAC-SHA256         1.4 GB / second
  2. BLAKE3-256           CHACHA20-POLY1305-HMAC-SHA256  1.1 GB / second
  3. BLAKE3-256-128       CHACHA20-POLY1305-HMAC-SHA256  1.1 GB / second
  4. HMAC-SHA256          AES256-GCM-HMAC-SHA256         0.9 GB / second
  5. HMAC-SHA256-128      AES256-GCM-HMAC-SHA256         0.9 GB / second
  6. HMAC-SHA224          AES256-GCM-HMAC-SHA256         0.9 GB / second
  7. HMAC-SHA256-128      CHACHA20-POLY1305-HMAC-SHA256  815.4 MB / second
  8. HMAC-SHA256          CHACHA20-POLY1305-HMAC-SHA256  814.9 MB / second
  9. HMAC-SHA224          CHACHA20-POLY1305-HMAC-SHA256  813.8 MB / second
 10. BLAKE2B-256-128      AES256-GCM-HMAC-SHA256         610.5 MB / second
 11. BLAKE2B-256-128      CHACHA20-POLY1305-HMAC-SHA256  560 MB / second
 12. BLAKE2B-256          CHACHA20-POLY1305-HMAC-SHA256  556.2 MB / second
 13. BLAKE2S-256          AES256-GCM-HMAC-SHA256         471.6 MB / second
 14. BLAKE2S-128          AES256-GCM-HMAC-SHA256         470.7 MB / second
 15. BLAKE2S-128          CHACHA20-POLY1305-HMAC-SHA256  440.4 MB / second
 16. BLAKE2S-256          CHACHA20-POLY1305-HMAC-SHA256  440 MB / second
 17. BLAKE2B-256          AES256-GCM-HMAC-SHA256         391.3 MB / second
 18. HMAC-SHA3-224        AES256-GCM-HMAC-SHA256         282.2 MB / second
 19. HMAC-SHA3-224        CHACHA20-POLY1305-HMAC-SHA256  279.1 MB / second
 20. HMAC-SHA3-256        AES256-GCM-HMAC-SHA256         273 MB / second
 21. HMAC-SHA3-256        CHACHA20-POLY1305-HMAC-SHA256  266.5 MB / second
-----------------------------------------------------------------
Fastest option for this machine is: --block-hash=BLAKE3-256-128 --encryption=AES256-GCM-HMAC-SHA256

I hope this helps.

Just an FYI, though I clearly need to solve this for “this” server. I am hoping to learn how to solve this for other servers if the symptom reveals itself elsewhere. I sell servers to my customers and there server might not always be quite as robust as mine ie 16 cores.

Thanks for weighing in on this

Jay

tessus · August 6, 2024, 10:10pm

Also see

github.com/kopia/kopia

CPU usage on macOS over the roof (> 400%)

opened 03:38AM - 25 Mar 24 UTC

tessus

app ux keep-open

I've been using kopia for a while now and I thought it might be a bug and waited… for newer releases, but it didn't help. I've seen this behavior since I've been using kopia, which is around 0.14.0. It also happens with the latest release: 0.16.0 ![image](https://github.com/kopia/kopia/assets/223439/94705a30-0d93-45b1-8b5b-fe875046368f) Whenever a snapshot is in progress the CPU usage is constantly over 400% on a 6-core machine. Is there a way to limit the CPU usage to maybe 75%? P.S.: A backup should run in the background and not use all available resources on a system.

Even with --parallel=1 and --max-parallel-file-reads=1 and all other smart ideas, my CPU usage is between 130% and 190% when kopia runs.

Topic		Replies	Views
Slow backup process Support	7	908	March 30, 2023
Global policies Support	5	1328	January 15, 2021
Performance improvement tips? Support	3	1466	July 26, 2021
KopiaUI not finishing snapshot Support	4	519	February 16, 2021
Disk consumption larger than source Support	8	377	November 24, 2023

Kopia eating 100% of CPU

Related Topics