Unable to list kopia snapshots

Hello,

I am using velero for backing up my k8s PVC’s. Recently, Velero deprecated Restic, so I am currently trying to migrate our environment from using Restic to using Kopia.

So far, if I describe a backup, I would get the following status:

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups - kopia:
    Completed:
      qs-splunk/splunk-qs-cm01-cluster-manager-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-lm-license-manager-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-mc-monitoring-console-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-deployer-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-search-head-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-search-head-1: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-search-head-2: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-deployer-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-search-head-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-search-head-1: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-search-head-2: pvc-etc, pvc-var
      qs-splunk/splunk-sh01-standalone-0: pvc-etc, pvc-var

→ It shows that the PVCs have been successfully backed up.

And this is the s3 bucket structure:

~ % s3cmd ls -c .qss3cfg  s3://qs-kopia-snapshots/         
                          DIR  s3://qs-kopia-snapshots/backups/
                          DIR  s3://qs-kopia-snapshots/kopia/
2025-02-10 13:13          766  s3://qs-kopia-snapshots/_log_20250210151349_1235_1739193229_1739193230_1_927e42ff61e336f99c991922520a0236
2025-02-10 13:48         1396  s3://qs-kopia-snapshots/_log_20250210154827_523c_1739195307_1739195311_1_eb788bf91a97de11775d9cc062daaddf
2025-02-10 13:13           30  s3://qs-kopia-snapshots/kopia.blobcfg
2025-02-10 13:48         1152  s3://qs-kopia-snapshots/kopia.maintenance
2025-02-10 13:13         1101  s3://qs-kopia-snapshots/kopia.repository
2025-02-10 13:13         4298  s3://qs-kopia-snapshots/qb5819eab5f3fb88afe6916799481a170-sca46a09fbbba2a00131
2025-02-10 13:13          143  s3://qs-kopia-snapshots/xn0_c71d4cb84529678f3fc78be1c0c1d466-sca46a09fbbba2a00131-c1

~ % s3cmd ls -c .qss3cfg  s3://qs-kopia-snapshots/backups/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250211020006/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250212020007/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250213020009/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250214020010/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250215020011/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250216020013/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250217020014/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250218020015/

 ~ % s3cmd ls -c .qss3cfg  s3://qs-kopia-snapshots/kopia/qs-splunk/

-> here are lots of log files
2025-02-10 13:24           30  s3://qs-kopia-snapshots/kopia/qs-splunk/kopia.blobcfg
2025-02-18 06:37         5037  s3://qs-kopia-snapshots/kopia/qs-splunk/kopia.maintenance
2025-02-10 13:24         1075  s3://qs-kopia-snapshots/kopia/qs-splunk/kopia.repository
2025-02-12 02:00     24140605  s3://qs-kopia-snapshots/kopia/qs-splunk/p0000d84ab53f413c809b814631d57f4a-s4c132af118bad779131
2025-02-10 13:25     24121647  s3://qs-kopia-snapshots/kopia/qs-splunk/p00019366802e2c50c74975ca90459516-s46e19a024baaf3c1131
2025-02-14 02:00     21007549  s3://qs-kopia-snapshots/kopia/qs-splunk/p000a3f7dc32e81b7a598b77870c682b0-s9bbb989b428366ea131
2025-02-14 02:01     21001800  s3://qs-kopia-snapshots/kopia/qs-splunk/p00103ba1d6090528280dca1378577b63-s9bbb989b428366ea131
2025-02-12 02:00     26841952  s3://qs-kopia-snapshots/kopia/qs-splunk/p001a5cca75d14cb2114b2a71afda2799-s6b712e25c91d59ac131
2025-02-15 02:01     26162049  s3://qs-kopia-snapshots/kopia/qs-splunk/p001fcc590e9949eb81397932d57700ab-s5f627b7d08d9d188131
2025-02-10 13:30     25170190  s3://qs-kopia-snapshots/kopia/qs-splunk/p0020d8e9a516940218c7f32aa87e88d2-s4942d19d50907c80131
2025-02-16 02:00     27173233  s3://qs-kopia-snapshots/kopia/qs-splunk/p0029dd2be2d97fd470209348690caddc-s2307f1854270777e131
2025-02-10 13:25     23982449  s3://qs-kopia-snapshots/kopia/qs-splunk/p0030665e791e1cfa6b5626325a14562e-s250bde811d1ea75f131
2025-02-16 02:01     21072917  s3://qs-kopia-snapshots/kopia/qs-splunk/p003d45686b44a0c69054614c7c19c938-sf8d441bef157d2cf131
...
{lots of those}}
...
2025-02-18 02:01         4298  s3://qs-kopia-snapshots/kopia/qs-splunk/qf9061e340fd6818ac2c799c7b5323d48-s1d161690d63fc0d6132
2025-02-18 02:01       282650  s3://qs-kopia-snapshots/kopia/qs-splunk/qfd8535d6aeb760bbf51fc6242de581d0-sc593a1bcf4bac9d7132
2025-02-18 02:00         4298  s3://qs-kopia-snapshots/kopia/qs-splunk/qfe9679932e00866a2bae419029ea0bcf-s3ddff6cd9e6e068f132
...
{lots of those}}
...
2025-02-15 03:37           12  s3://qs-kopia-snapshots/kopia/qs-splunk/xe3
2025-02-17 03:37           12  s3://qs-kopia-snapshots/kopia/qs-splunk/xe4
2025-02-14 02:00         1467  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_0181dc7542126514e0b2aa61b3e8769a-sea148ffaaad58e66131-c1
2025-02-14 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_0572472c6f79b12626f165aa0c074ed7-s083b9ddf2fcecafc131-c1
2025-02-15 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_078823a2af25b8d9c12d81ec9cfe3d91-sf286fa73aecf469d131-c1
2025-02-15 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_096daf419325bcef8908ffe666a368eb-sd247fa24b4b60297131-c1
2025-02-18 02:01        50546  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_dc7914fe601bc2a2d156f24c2986c1d1-s297399a701591680132-c1
2025-02-18 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_efcfa58cf6078a1530c596796eaa6feb-s97db82b5a9b1ce5f132-c1
2025-02-18 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f192c29bc219b42f5936d71f5337fa72-se7beea585d07df92132-c1
2025-02-18 02:01          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f57a9f3969258955109481c677906290-sb25e6469daf83490132-c1
2025-02-18 02:02      3186481  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f6860b75bc315a97092e66f0341b891c-s76e5964bde01836b132-c1
2025-02-18 02:01          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f9ecfa93801ef716896956166d1615ec-s9c1b673851b7f4be132-c1
...
{lots of those}}
...
2025-02-13 03:37     13798561  s3://qs-kopia-snapshots/kopia/qs-splunk/xs0_0266ab5307fd167a600a17adb81650d2-s899815072f638230-c1
2025-02-15 04:37     15107687  s3://qs-kopia-snapshots/kopia/qs-splunk/xs1_5c68000b38fdbd85d1db195cffc0266d-s5527e343025a3682-c1
2025-02-17 04:37     11392502  s3://qs-kopia-snapshots/kopia/qs-splunk/xs2_9f4d9199ac368fb8d03b95dbf9e7140e-s74c7c7a0aca38578-c1
2025-02-17 16:37           18  s3://qs-kopia-snapshots/kopia/qs-splunk/xw1739716633

here is repo status:

~ % kopia repository status
Enter password to open repository: 

Config file:         /Users/chakarovs/Library/Application Support/kopia/repository.config

Description:         Repository in S3: object.storage.eu01.onstackit.cloud qs-kopia-snapshots
Hostname:            sit-smbp-kkdnmd
Username:            chakarovs
Read-only:           false
Format blob cache:   15m0s

Storage type:        s3
Storage capacity:    unbounded
Storage config:      {
                       "bucket": "qs-kopia-snapshots",
                       "endpoint": "{s3 endpoint}",
                       "accessKeyID": "***********************************",
                       "secretAccessKey": "****************************************",
                       "sessionToken": ""
                     }

Unique ID:           ID
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      3
Content compression: true
Password changes:    true
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 0

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin:    4h0m0s
Epoch checkpoint every:  7 epochs
~ % kopia blob list
Enter password to open repository: 

_log_20250210151349_1235_1739193229_1739193230_1_927e42ff61e336f99c991922520a0236        766 2025-02-10 15:13:51 EET
_log_20250210154827_523c_1739195307_1739195311_1_eb788bf91a97de11775d9cc062daaddf       1396 2025-02-10 15:48:31 EET
kopia.blobcfg                                                                  30 2025-02-10 15:13:46 EET
kopia.maintenance                                                            1152 2025-02-10 15:48:31 EET
kopia.repository                                                             1101 2025-02-10 15:13:46 EET
qb5819eab5f3fb88afe6916799481a170-sca46a09fbbba2a00131                       4298 2025-02-10 15:13:49 EET
xn0_c71d4cb84529678f3fc78be1c0c1d466-sca46a09fbbba2a00131-c1                  143 2025-02-10 15:13:50 EET
backups/                                                                        0 0001-01-01 01:33:16 LMT
kopia/                                                                          0 0001-01-01 01:33:16 LMT

However, when I run snapshot list, I get nothing:

~ % kopia snapshot list --all
Enter password to open repository: 

 ~ %

It looks like the snapshots are created and are there, but I cant find them… Do you guys have any idea what the issue might be?

Thanks in advance!

I’m absolutely not familiar with Velero but it seems like there are multiple repositories (multiple kopia.repository files) in your bucket. The repository in your main directory (s3://qs-kopia-snapshots/ doesn’t have any data in it (blobs with p prefix), hence no snapshots. Try connecting to the repository under s3://qs-kopia-snapshots/kopia/qs-splunk/.