Hello, I am new to Kopia. I have two differents servers using Ubuntu server 24.04, and I want to use Kopia to backup the data from one server (data-server) to store it in my backup-server (quite the classic use-case !)
On my data server, I have an old xeon 1225 v3 and 32GB of DDR3 which has 4 threads.
On the backup server, I created a Kopia repository and created an Ubuntu service which automatically launches kopia server.
On the data server, I installed Kopia and I am trying different scenarios to create a snapshot that is uploaded to the repository of the backup server using kopia snapshot create, for a set of 28.9GB of data (mainly pictures and videos). I first had an extremely slow process (2h for 10GB) by using the following command :
sudo kopia snapshot create /data
I stopped the snapshot and cleaned the data with listing the incomplete snapshot and running
kopia snapshot delete
And then on the backup-server I run
kopia maintenance run --full --safety=none
Which from my understanding delete everything and makes my repository clean.I run theses command after every test.
I then started to make some tests playing with the parameter --parallel.
Here are the result I had
–parallel 5000
- 0 hashing, 2222 hashed (28.9 GB), 0 cached (0 B), uploaded 25.3 GB, estimated 28.9 GB (100.0%) 0s left
Created snapshot with root k823d94524065b5293f43db74c4724d6b and ID 550380de7d2c76cd1ad0aaa595df794e in 32m49s
With --parallel = 4
- 0 hashing, 2222 hashed (28.9 GB), 0 cached (0 B), uploaded 14 GB, estimated 28.9 GB (100.0%) 0s left
Created snapshot with root kb9847720104737d0b9de255e9376e066 and ID fd7355d07be45b9f4186d03ee6073ddc in 57m35s
With --parallel 100000
- 0 hashing, 2222 hashed (28.9 GB), 0 cached (0 B), uploaded 28.8 GB, estimated 28.9 GB (100.0%) 0s left
Created snapshot with root k4fbfc9a54143eb752218ad5657402607 and ID 79ff4c010272da9f18df256f0b6ae9bd in 20m31s
When I look at the different data I have while the process in ongoing, with a higer parameter for --parallel, the hashing is going much faster and it starts to upload at full speed directly. When it’s low (4) or when the parameter is by default (0), the hashing seems to takes forever. The upload is really low (probably because the data is not ready to be uploaded ? ). Shouldn’t the default parameter be using more of the CPU capabilities ? Is it because the processor is quite old ?
Another interogation I have is why the value of “uploaded” is different in each test I am doing ? I checked on the backup-server how much data was added using the command “df -h” and it matchs with the value of "uploaded ". How is it possible that with the same command with only the parameter --parallel, and without changing anything compression related, it uploads differents size ?
I also have this warning in the end of the snapshot :
WARN unable to send notification {"err":"unhandled session request","errVerbose":"unhandled session request\ngithub.com/kopia/kopia/repo.errorFromSessionResponse\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:660\ngithub.com/kopia/kopia/repo.unhandledSessionResponse\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:666\ngithub.com/kopia/kopia/repo.(*grpcInnerSession).SendNotification\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:496\ngithub.com/kopia/kopia/repo.(*grpcRepositoryClient).SendNotification.func1\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:473\ngithub.com/kopia/kopia/repo.inSessionWithoutRetry[...]\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:606\ngithub.com/kopia/kopia/repo.maybeRetry[...]\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:572\ngithub.com/kopia/kopia/repo.(*grpcRepositoryClient).SendNotification\n\t/home/runner/work/kopia/kopia/repo/grpc_repository_client.go:472\ngithub.com/kopia/kopia/notification.Send\n\t/home/runner/work/kopia/kopia/notification/notification_send.go:118\ngithub.com/kopia/kopia/cli.(*commandSnapshotCreate).run\n\t/home/runner/work/kopia/kopia/cli/command_snapshot_create.go:153\ngithub.com/kopia/kopia/cli.(*App).repositoryWriterAction.func1.1\n\t/home/runner/work/kopia/kopia/cli/app.go:496\ngithub.com/kopia/kopia/repo.WriteSession\n\t/home/runner/work/kopia/kopia/repo/repository.go:413\ngithub.com/kopia/kopia/cli.(*App).repositoryWriterAction.func1\n\t/home/runner/work/kopia/kopia/cli/app.go:492\ngithub.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).maybeRepositoryAction.func2\n\t/home/runner/work/kopia/kopia/cli/app.go:579\ngithub.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).maybeRepositoryAction.(*App).baseActionWithContext.func3.1.1\n\t/home/runner/work/kopia/kopia/cli/app.go:564\ngithub.com/kopia/kopia/cli.(*profileFlags).withProfiling\n\t/home/runner/work/kopia/kopia/cli/profile.go:45\ngithub.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).maybeRepositoryAction.(*App).baseActionWithContext.func3.1\n\t/home/runner/work/kopia/kopia/cli/app.go:559\ngithub.com/kopia/kopia/cli.(*App).runAppWithContext.func1\n\t/home/runner/work/kopia/kopia/cli/app.go:530\ngithub.com/kopia/kopia/cli.(*App).runAppWithContext\n\t/home/runner/work/kopia/kopia/cli/app.go:531\ngithub.com/kopia/kopia/cli.(*App).repositoryWriterAction.(*App).maybeRepositoryAction.(*App).baseActionWithContext.func3\n\t/home/runner/work/kopia/kopia/cli/app.go:558\ngithub.com/alecthomas/kingpin/v2.(*actionMixin).applyActions\n\t/home/runner/go/pkg/mod/github.com/alecthomas/kingpin/v2@v2.4.0/actions.go:28\ngithub.com/alecthomas/kingpin/v2.(*Application).applyActions\n\t/home/runner/go/pkg/mod/github.com/alecthomas/kingpin/v2@v2.4.0/app.go:568\ngithub.com/alecthomas/kingpin/v2.(*Application).execute\n\t/home/runner/go/pkg/mod/github.com/alecthomas/kingpin/v2@v2.4.0/app.go:398\ngithub.com/alecthomas/kingpin/v2.(*Application).Parse\n\t/home/runner/go/pkg/mod/github.com/alecthomas/kingpin/v2@v2.4.0/app.go:230\nmain.main\n\t/home/runner/work/kopia/kopia/main.go:77\nruntime.main\n\t/opt/hostedtoolcache/go/1.22.7/x64/src/runtime/proc.go:271\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.22.7/x64/src/runtime/asm_amd64.s:1695"}