Unable to list kopia snapshots

Support
1

Hello,

I am using velero for backing up my k8s PVC’s. Recently, Velero deprecated Restic, so I am currently trying to migrate our environment from using Restic to using Kopia.

So far, if I describe a backup, I would get the following status:

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups - kopia:
    Completed:
      qs-splunk/splunk-qs-cm01-cluster-manager-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-lm-license-manager-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-mc-monitoring-console-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-deployer-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-search-head-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-search-head-1: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc01-search-head-2: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-deployer-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-search-head-0: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-search-head-1: pvc-etc, pvc-var
      qs-splunk/splunk-qs-shc02-search-head-2: pvc-etc, pvc-var
      qs-splunk/splunk-sh01-standalone-0: pvc-etc, pvc-var

→ It shows that the PVCs have been successfully backed up.

And this is the s3 bucket structure:

~ % s3cmd ls -c .qss3cfg  s3://qs-kopia-snapshots/         
                          DIR  s3://qs-kopia-snapshots/backups/
                          DIR  s3://qs-kopia-snapshots/kopia/
2025-02-10 13:13          766  s3://qs-kopia-snapshots/_log_20250210151349_1235_1739193229_1739193230_1_927e42ff61e336f99c991922520a0236
2025-02-10 13:48         1396  s3://qs-kopia-snapshots/_log_20250210154827_523c_1739195307_1739195311_1_eb788bf91a97de11775d9cc062daaddf
2025-02-10 13:13           30  s3://qs-kopia-snapshots/kopia.blobcfg
2025-02-10 13:48         1152  s3://qs-kopia-snapshots/kopia.maintenance
2025-02-10 13:13         1101  s3://qs-kopia-snapshots/kopia.repository
2025-02-10 13:13         4298  s3://qs-kopia-snapshots/qb5819eab5f3fb88afe6916799481a170-sca46a09fbbba2a00131
2025-02-10 13:13          143  s3://qs-kopia-snapshots/xn0_c71d4cb84529678f3fc78be1c0c1d466-sca46a09fbbba2a00131-c1

~ % s3cmd ls -c .qss3cfg  s3://qs-kopia-snapshots/backups/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250211020006/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250212020007/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250213020009/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250214020010/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250215020011/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250216020013/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250217020014/
                          DIR  s3://qs-kopia-snapshots/backups/velero-qs-splunk-backup-20250218020015/

 ~ % s3cmd ls -c .qss3cfg  s3://qs-kopia-snapshots/kopia/qs-splunk/

-> here are lots of log files
2025-02-10 13:24           30  s3://qs-kopia-snapshots/kopia/qs-splunk/kopia.blobcfg
2025-02-18 06:37         5037  s3://qs-kopia-snapshots/kopia/qs-splunk/kopia.maintenance
2025-02-10 13:24         1075  s3://qs-kopia-snapshots/kopia/qs-splunk/kopia.repository
2025-02-12 02:00     24140605  s3://qs-kopia-snapshots/kopia/qs-splunk/p0000d84ab53f413c809b814631d57f4a-s4c132af118bad779131
2025-02-10 13:25     24121647  s3://qs-kopia-snapshots/kopia/qs-splunk/p00019366802e2c50c74975ca90459516-s46e19a024baaf3c1131
2025-02-14 02:00     21007549  s3://qs-kopia-snapshots/kopia/qs-splunk/p000a3f7dc32e81b7a598b77870c682b0-s9bbb989b428366ea131
2025-02-14 02:01     21001800  s3://qs-kopia-snapshots/kopia/qs-splunk/p00103ba1d6090528280dca1378577b63-s9bbb989b428366ea131
2025-02-12 02:00     26841952  s3://qs-kopia-snapshots/kopia/qs-splunk/p001a5cca75d14cb2114b2a71afda2799-s6b712e25c91d59ac131
2025-02-15 02:01     26162049  s3://qs-kopia-snapshots/kopia/qs-splunk/p001fcc590e9949eb81397932d57700ab-s5f627b7d08d9d188131
2025-02-10 13:30     25170190  s3://qs-kopia-snapshots/kopia/qs-splunk/p0020d8e9a516940218c7f32aa87e88d2-s4942d19d50907c80131
2025-02-16 02:00     27173233  s3://qs-kopia-snapshots/kopia/qs-splunk/p0029dd2be2d97fd470209348690caddc-s2307f1854270777e131
2025-02-10 13:25     23982449  s3://qs-kopia-snapshots/kopia/qs-splunk/p0030665e791e1cfa6b5626325a14562e-s250bde811d1ea75f131
2025-02-16 02:01     21072917  s3://qs-kopia-snapshots/kopia/qs-splunk/p003d45686b44a0c69054614c7c19c938-sf8d441bef157d2cf131
...
{lots of those}}
...
2025-02-18 02:01         4298  s3://qs-kopia-snapshots/kopia/qs-splunk/qf9061e340fd6818ac2c799c7b5323d48-s1d161690d63fc0d6132
2025-02-18 02:01       282650  s3://qs-kopia-snapshots/kopia/qs-splunk/qfd8535d6aeb760bbf51fc6242de581d0-sc593a1bcf4bac9d7132
2025-02-18 02:00         4298  s3://qs-kopia-snapshots/kopia/qs-splunk/qfe9679932e00866a2bae419029ea0bcf-s3ddff6cd9e6e068f132
...
{lots of those}}
...
2025-02-15 03:37           12  s3://qs-kopia-snapshots/kopia/qs-splunk/xe3
2025-02-17 03:37           12  s3://qs-kopia-snapshots/kopia/qs-splunk/xe4
2025-02-14 02:00         1467  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_0181dc7542126514e0b2aa61b3e8769a-sea148ffaaad58e66131-c1
2025-02-14 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_0572472c6f79b12626f165aa0c074ed7-s083b9ddf2fcecafc131-c1
2025-02-15 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_078823a2af25b8d9c12d81ec9cfe3d91-sf286fa73aecf469d131-c1
2025-02-15 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn2_096daf419325bcef8908ffe666a368eb-sd247fa24b4b60297131-c1
2025-02-18 02:01        50546  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_dc7914fe601bc2a2d156f24c2986c1d1-s297399a701591680132-c1
2025-02-18 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_efcfa58cf6078a1530c596796eaa6feb-s97db82b5a9b1ce5f132-c1
2025-02-18 02:00          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f192c29bc219b42f5936d71f5337fa72-se7beea585d07df92132-c1
2025-02-18 02:01          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f57a9f3969258955109481c677906290-sb25e6469daf83490132-c1
2025-02-18 02:02      3186481  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f6860b75bc315a97092e66f0341b891c-s76e5964bde01836b132-c1
2025-02-18 02:01          143  s3://qs-kopia-snapshots/kopia/qs-splunk/xn4_f9ecfa93801ef716896956166d1615ec-s9c1b673851b7f4be132-c1
...
{lots of those}}
...
2025-02-13 03:37     13798561  s3://qs-kopia-snapshots/kopia/qs-splunk/xs0_0266ab5307fd167a600a17adb81650d2-s899815072f638230-c1
2025-02-15 04:37     15107687  s3://qs-kopia-snapshots/kopia/qs-splunk/xs1_5c68000b38fdbd85d1db195cffc0266d-s5527e343025a3682-c1
2025-02-17 04:37     11392502  s3://qs-kopia-snapshots/kopia/qs-splunk/xs2_9f4d9199ac368fb8d03b95dbf9e7140e-s74c7c7a0aca38578-c1
2025-02-17 16:37           18  s3://qs-kopia-snapshots/kopia/qs-splunk/xw1739716633

here is repo status:

~ % kopia repository status
Enter password to open repository: 

Config file:         /Users/chakarovs/Library/Application Support/kopia/repository.config

Description:         Repository in S3: object.storage.eu01.onstackit.cloud qs-kopia-snapshots
Hostname:            sit-smbp-kkdnmd
Username:            chakarovs
Read-only:           false
Format blob cache:   15m0s

Storage type:        s3
Storage capacity:    unbounded
Storage config:      {
                       "bucket": "qs-kopia-snapshots",
                       "endpoint": "{s3 endpoint}",
                       "accessKeyID": "***********************************",
                       "secretAccessKey": "****************************************",
                       "sessionToken": ""
                     }

Unique ID:           ID
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      3
Content compression: true
Password changes:    true
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 0

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin:    4h0m0s
Epoch checkpoint every:  7 epochs

~ % kopia blob list
Enter password to open repository: 

_log_20250210151349_1235_1739193229_1739193230_1_927e42ff61e336f99c991922520a0236        766 2025-02-10 15:13:51 EET
_log_20250210154827_523c_1739195307_1739195311_1_eb788bf91a97de11775d9cc062daaddf       1396 2025-02-10 15:48:31 EET
kopia.blobcfg                                                                  30 2025-02-10 15:13:46 EET
kopia.maintenance                                                            1152 2025-02-10 15:48:31 EET
kopia.repository                                                             1101 2025-02-10 15:13:46 EET
qb5819eab5f3fb88afe6916799481a170-sca46a09fbbba2a00131                       4298 2025-02-10 15:13:49 EET
xn0_c71d4cb84529678f3fc78be1c0c1d466-sca46a09fbbba2a00131-c1                  143 2025-02-10 15:13:50 EET
backups/                                                                        0 0001-01-01 01:33:16 LMT
kopia/                                                                          0 0001-01-01 01:33:16 LMT

However, when I run snapshot list, I get nothing:

~ % kopia snapshot list --all
Enter password to open repository: 

 ~ %

It looks like the snapshots are created and are there, but I cant find them… Do you guys have any idea what the issue might be?

Thanks in advance!

2

I’m absolutely not familiar with Velero but it seems like there are multiple repositories (multiple kopia.repository files) in your bucket. The repository in your main directory (s3://qs-kopia-snapshots/ doesn’t have any data in it (blobs with p prefix), hence no snapshots. Try connecting to the repository under s3://qs-kopia-snapshots/kopia/qs-splunk/.