[TESTERS NEEDED] Major improvements to repository index format

Announcements
21

I first did some testing with version 0.8.4 and 20210719.0.215556 build: 4f93a1778229be426b26081b8a2a59cfb6e38e7a from: kopia/kopia on existing repos for filesytem and sftp, no problems found.

I then made a copy of my filesystem repo and upgraded it.
I then tested again, also 0.8.4 gave no problems with the upgraded repo.

Then I created a new filesystem repo with the --index-version=2 --enable-index-epochs flags.
I tested with the new version and found no problem, but now 0.8.4 was unable to list the snapshots, no error, just no snapshots.
I then tried to restore a snapshot from the new repo with 0.8.4 but got an error that the snapshot could not be found (of cause the new version had no problems with the restore).
I then created a new sftp repo with the --index-version=2 --enable-index-epochs flags and repeated above tests with the same result.

So my conclusion: the new version of kopia gave no problems with both old and new repo version for filesystem and sftp.

22

Just sharing my experience of using v20210724.0.130450 with the new features.

I had no issues backing up roughly 1TB of data spread across two boxes to B2 in terms of speed, and experienced seemingly no data corruption from testing restoring a few large files that were shared between boxes (compressed and non-compressed). The two boxes shared a lot of data (~700GB) and had s2-default compression enabled on some files.

Looking at the repository sizes, one machine has a repo with ~70GB data (all unique afaik), and one with 812GB. The other box has two small <1GB repos with unique data, and one with 779GB which should theoretically be almost entirely shared data with the two repos on the other machine.
By my count, usage should roughly add up to 884GB, but my bucket usage in B2 is 944GB. I could be wrong, but I don’t think that much data has changed recently to make up for the discrepancy, and maintenance has been running consistently through kopia server.

Is there a way to check the usage amounts of non-latest snapshots?

Edit:
Just read the above conversation, and decided to delete all but the latest snapshot for all repos then run maintenance. The safety time hasn’t passed yet - so will report back after - but during maintenance, the following error occured:

[...]
  Processed 788108 contents, discovered 796408...
Looking for unreferenced contents...
Found safe time to drop indexes: 2021-07-28 14:22:36.600037343 +0000 UTC
Dropping contents deleted before 2021-07-28 14:22:36.600037343 +0000 UTC
Rewriting contents from short packs...
unable to add 6ae864cc190065cf05622d1e8e20ac65 to content cache: unable to complete PutBlobInPath:/home/me/.cache/kopia/3a145bfee2cfb4a0/contents/6a/e864cc190065cf05622d1e8e20ac65.f despite 10 retries
Skipping blob deletion because not enough time has passed yet (59m59s left).
Finished full maintenance.

Here’s the output of kopia blob ls --prefix=x before and after maintenance/snapshot cleaning:

xe27                                                                           12 2021-07-28 21:25:18 UTC
xe28                                                                           12 2021-07-29 07:29:32 UTC
xn27_1733c51d80d4956006c86dd1254d865c-s027004a2245ad409106-c1               16835 2021-07-29 00:04:19 UTC
xn27_1adaff83ac9579932a6f15b0cb18d67c-scb44300663963c7d106-c1                 367 2021-07-29 03:00:30 UTC
xn27_27dda36d27ac7b41286ec63b441d9bbe-sb819725ccfaf2612106-c1               35387 2021-07-29 03:27:52 UTC
xn27_28999b48965fd65df69147b38a0b294a-sa54ecf1d2f2b4402106-c1                 367 2021-07-29 06:00:22 UTC
xn27_31c27f2b09f5d4768d23d67c434d3c24-s2e207b55047943a5106-c1                 367 2021-07-29 07:01:42 UTC
xn27_3caf276e09a8c0a02b07eb9d7fa9b123-sf2686e8d780ae131106-c1               23785 2021-07-29 00:09:06 UTC
xn27_6addc5d7b96bfa214805b0bff50f118e-s541662d179996de5106-c1                 367 2021-07-29 04:00:27 UTC
xn27_81106330503bfdb011710b1dd3254c85-s19cdcf40dc4c2e85106-c1                 143 2021-07-29 00:01:39 UTC
xn27_878997b1dfccfd32b9f937aaa62fd749-s41b761663ebd3cd1106-c1               20677 2021-07-28 22:00:47 UTC
xn27_9e699c287c67e05c835ce4694332ec32-sd7270a18684b1a3f106-c1                 143 2021-07-29 00:01:21 UTC
xn27_adbe19a63f72dbf8ca684d1fc2be446d-s2e5b0baf1c6720de106-c1                 367 2021-07-29 05:00:25 UTC
xn27_b1eedddef178c6544044b76b84dad34f-s1367ae6349149dca106-c1                 549 2021-07-29 01:00:27 UTC
xn27_bdd828a99aa6896d03514bfb215cc709-sadc8303ad11af8cc106-c1               17567 2021-07-29 01:27:01 UTC
xn27_d06a8aa9a7f6a790510d0565c6e74fa2-sfa5d6a4216917e0f106-c1                8902 2021-07-29 02:10:18 UTC
xn27_da82f68217b8fce3d114863ace299b73-sd14e612614656994106-c1               36014 2021-07-29 05:28:54 UTC
xn27_df0ac0a14fd4d535380295368b63591f-s22440cdbf9ec171a106-c1                 447 2021-07-29 02:11:14 UTC
xn27_e3adc89b3e2641adf67a58234dfc2b3b-sad4ac6e3d63fc8e1106-c1               36344 2021-07-29 07:29:28 UTC
xn27_eeccdce5600d7eddf95f911647685b9c-s03ca4add6661b190106-c1               21255 2021-07-28 23:00:47 UTC
xn27_f04c57b117f81ff821acd07a2475a64f-sb9e6348e36de0514106-c1               16874 2021-07-28 23:25:53 UTC
xn27_f686a86ef2a854e36092dfe1a6f56ea8-s6bb2a88caf910dd5106-c1               16903 2021-07-29 00:04:33 UTC
xn28_036ea9242e9031aedb518a5a7bcda357-s274180050b83c367106-c1               37037 2021-07-29 11:30:21 UTC
xn28_12ac7b8de28613f31549290861b3251e-sf428290fa960d9c6106-c1                 367 2021-07-29 13:00:13 UTC
xn28_1df2acc1ff8bd5e5a8b55d3a3ca20481-sde303e63302f16b3106-c1               43736 2021-07-29 13:30:47 UTC
xn28_35350bf0d7f4f8ebc7a9ae178515ce1b-s6b0a942a31335462106-c1                 379 2021-07-29 14:00:12 UTC
xn28_62a9bb64e49a5e85e2d6aa9aac9944ef-sc29dbc3e7808c00a106-c1                 367 2021-07-29 12:00:12 UTC
xn28_9ae930202777f92a6f67aea71d3ba9e3-sc0e62f3933d232a9106-c1               36245 2021-07-29 09:29:59 UTC
xn28_a00140c74cf6e10ef5f4907e532a4a32-s6d819bac2082c60c106-c1               22554 2021-07-29 12:01:59 UTC
xn28_a1c6935e5c189ca3057142cf827ea630-se9d8a3325eae5bd7106-c1               14011 2021-07-29 11:16:53 UTC
xn28_a25c2ea052304ac15d1ea3430c677083-s8af77c6e93920cb8106-c1                 367 2021-07-29 09:00:16 UTC
xn28_aee9c88693d1924879e028e9d2f898fb-s9401e0aacb093897106-c1                 367 2021-07-29 08:00:20 UTC
xn28_ba699447b7ed61a04de818cf1e3b2d13-scc06891fc0cbe800106-c1               35245 2021-07-29 15:32:36 UTC
xn28_cd4d4a236a84c713f76556cfd8e7567d-s6e2d9f27a2114ca0106-c1                 143 2021-07-29 13:52:03 UTC
xn28_d0b5766b535d0a701aa7733ab0faecad-s22058769e8c6827c106-c1                 367 2021-07-29 11:00:13 UTC
xn28_d3751c65aeeff5ae8061b58e348aa7fa-s523068a0996b1cef106-c1                 367 2021-07-29 10:00:20 UTC
xn28_d8730d4becb2acfbf4097ad80ac50c6f-sb9ff3b2a2d23eaa7106-c1                1236 2021-07-29 15:01:37 UTC
xn28_e03e87a06a12f2be91a3b66b7bf16541-sf062d84da9efff4b106-c1                 367 2021-07-29 16:01:11 UTC
xn28_e3ef09711cd0426e8c046e7185349002-s841dd9978678d84b106-c1              122499 2021-07-29 15:31:57 UTC
xn28_f7aa953f4e0bf45384795eb69ae4b41b-s0c5b16b52f4f269f106-c1                1025 2021-07-29 08:00:32 UTC
xr0_7_a51491a5adf4586b89812dac6079b2c6-s749180735e0c3863-c1              29769670 2021-07-21 00:08:54 UTC
xr16_23_8c1e98dac7caa9f5f3122fe37751db8f-s25a8ddb24e26a2d0-c1             2418091 2021-07-28 00:13:33 UTC
xr8_15_7af99ee0e821c7c510c6c6a630923f19-sf751c849cb4e8a64-c1              8964883 2021-07-24 13:36:29 UTC
xs0_65bc61e435c6e0310c0f3cdf4b37f7c7-s7dac514995098f5c-c1                 2502545 2021-07-18 19:34:30 UTC
xs0_65bc61e435c6e0310c0f3cdf4b37f7c7-s8368faca34516d9e-c1                 2502545 2021-07-18 19:38:33 UTC
xs0_65bc61e435c6e0310c0f3cdf4b37f7c7-sb58117e00facd7bc-c1                 2502545 2021-07-18 19:39:09 UTC
xs10_0611177c35a41d35043b7f61002fc65a-s6fe540e3f123e24b-c1                2175819 2021-07-21 19:29:59 UTC
xs10_0611177c35a41d35043b7f61002fc65a-s70df1ad4e7737c60-c1                2175819 2021-07-21 19:25:12 UTC
xs10_0611177c35a41d35043b7f61002fc65a-sb3a7d029b0e49028-c1                2175819 2021-07-21 19:29:29 UTC
xs11_443b4f6b3cd2a0ceefac3ca536ea64e1-sf89d526f56bcf520-c1                 910412 2021-07-22 01:57:01 UTC
xs12_4c91d0e82a209b0d839401165f50080c-s0f230a26d47c3294-c1                1212378 2021-07-22 11:01:42 UTC
xs12_4c91d0e82a209b0d839401165f50080c-s921d7bad5f9c1e97-c1                1212378 2021-07-22 11:01:42 UTC
xs12_4c91d0e82a209b0d839401165f50080c-saaafe96a86a0664e-c1                1212378 2021-07-22 11:06:20 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-s565d89651902142d-c1                1066869 2021-07-22 18:38:11 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-sa1e6ac95c7d88d62-c1                1066869 2021-07-22 18:33:01 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-sc2f31dc5b4e99722-c1                1066869 2021-07-22 18:33:11 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-sd6eb72ac6c2be892-c1                1066869 2021-07-22 18:43:11 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-se3965dcf00b87e15-c1                1066869 2021-07-22 18:28:12 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-s46100eb98a29e137-c1                 406184 2021-07-23 17:39:58 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-s7746901b5295984f-c1                 406184 2021-07-23 17:29:57 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-s82ed4a70739848b6-c1                 406184 2021-07-23 17:34:50 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-sbb2f6b1f28a03798-c1                 406184 2021-07-23 17:25:05 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-sbee518b021596663-c1                 406184 2021-07-23 17:34:57 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s01a962beec41234e-c1                 376321 2021-07-24 13:46:17 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s390e0bdcf804c249-c1                 376321 2021-07-24 13:46:09 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s46cdbde36fe09d8e-c1                 376321 2021-07-24 13:36:28 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s4b22d11fc7d72573-c1                 376321 2021-07-24 13:51:17 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-sbf03056e173796ca-c1                 376321 2021-07-24 13:36:28 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-sdfd7ab84c16d8696-c1                 376321 2021-07-24 13:41:17 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s20ab68055bd33ec0-c1                 135222 2021-07-24 21:16:32 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s279164dba5f52c2a-c1                 135222 2021-07-24 21:01:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s349087eecb82c1d6-c1                 135222 2021-07-24 21:06:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s4edeb401eaaac819-c1                 135222 2021-07-24 21:11:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s7174f521767765e5-c1                 135222 2021-07-24 21:16:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-sb7cee941ba87a7e3-c1                 135222 2021-07-24 21:01:32 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-s7cab4d79c628c22b-c1                 139956 2021-07-25 09:27:22 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-s7ff5b2afbdefa983-c1                 139956 2021-07-25 09:22:30 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-s8c174a5d95b54b17-c1                 139956 2021-07-25 09:32:22 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-sba367a6f2279c5d3-c1                 139956 2021-07-25 09:31:46 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-sebdfadabdfc4d3a5-c1                 139956 2021-07-25 09:37:22 UTC
xs18_1697185502810f85c252be5960e1e982-s394e00ccd43808a1-c1                  44344 2021-07-25 18:31:55 UTC
xs18_1697185502810f85c252be5960e1e982-s50ed18d05f1840c5-c1                  44344 2021-07-25 18:27:44 UTC
xs18_1697185502810f85c252be5960e1e982-s67b996f4be5f4694-c1                  44344 2021-07-25 18:32:45 UTC
xs18_1697185502810f85c252be5960e1e982-sf2c36950ac7a3a87-c1                  44344 2021-07-25 18:37:45 UTC
xs18_1697185502810f85c252be5960e1e982-sffdfa67d65144e61-c1                  44344 2021-07-25 18:22:54 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s2c58a47f17494a2f-c1                1982778 2021-07-26 05:18:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s33bd3c7ac756b0fa-c1                1982778 2021-07-26 05:13:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s46d2c32562c2acdf-c1                1982778 2021-07-26 05:03:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s5c144f379019b994-c1                1982778 2021-07-26 05:08:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s9c0deaa4d8cbdfa2-c1                1982778 2021-07-26 05:17:05 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-sc524f88d6df7a081-c1                1982778 2021-07-26 05:03:07 UTC
xs1_3371b43e01e0ca9013e89f5499f8cf38-s01a9640da4047515-c1                11664987 2021-07-19 10:10:21 UTC
xs1_3371b43e01e0ca9013e89f5499f8cf38-s5b5ff3f5b175ef7e-c1                11664987 2021-07-19 10:14:46 UTC
xs1_3371b43e01e0ca9013e89f5499f8cf38-s616f5b68e3824bef-c1                11664987 2021-07-19 10:10:21 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-s92c239caa39419fb-c1                 184581 2021-07-26 15:13:37 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-s934af1db00aa78b5-c1                 184581 2021-07-26 15:23:37 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-saa903d3dc10bf3b5-c1                 184581 2021-07-26 15:17:12 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-sbbecb6139e106d87-c1                 184581 2021-07-26 15:08:44 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-sf4db485e7c74b2ab-c1                 184581 2021-07-26 15:18:37 UTC
xs21_7e65a99a3113db0337302df039a74b79-s41aac18478a8b7dc-c1                 164816 2021-07-27 02:34:06 UTC
xs21_7e65a99a3113db0337302df039a74b79-s4e465fe2239475a0-c1                 164816 2021-07-27 02:32:29 UTC
xs21_7e65a99a3113db0337302df039a74b79-s6d86a0b4ad34ef9a-c1                 164816 2021-07-27 02:39:06 UTC
xs21_7e65a99a3113db0337302df039a74b79-s71f97d63243b6110-c1                 164816 2021-07-27 02:24:11 UTC
xs21_7e65a99a3113db0337302df039a74b79-sd544e60ea312e21a-c1                 164816 2021-07-27 02:29:05 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-s1699aa399523dab2-c1                  43157 2021-07-27 13:11:47 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-s63aaf109ac1f0525-c1                  43157 2021-07-27 13:16:48 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-sa5670f9ffbdd7bf9-c1                  43157 2021-07-27 13:11:32 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-sc346e7ade1c87b97-c1                  43157 2021-07-27 13:26:33 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-scf77473bf4f75097-c1                  43157 2021-07-27 13:21:49 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-se99e24e9a2749b66-c1                  43157 2021-07-27 13:06:47 UTC
xs23_e7fd20cc21d8dd4fc804bc2b6a5adab1-scbf8e11f8cca04af-c1                  84706 2021-07-28 00:13:33 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s31652e2463d951df-c1                  96230 2021-07-28 11:17:58 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s551bcc9398650602-c1                  96230 2021-07-28 11:00:35 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s73356840ce807865-c1                  96230 2021-07-28 11:10:26 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s8bd3863ff6ff8a35-c1                  96230 2021-07-28 11:02:58 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-saf0dcab4fb8d4f35-c1                  96230 2021-07-28 11:05:26 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-sede157bb0415e2ce-c1                  96230 2021-07-28 11:15:26 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s1528eb2dd7b3a4e4-c1                  33499 2021-07-28 21:53:10 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s627ec9f1a8df335b-c1                  33499 2021-07-28 21:43:11 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s7cf734f0abb990ee-c1                  33499 2021-07-28 21:57:42 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s8c57bbd9bca13eb7-c1                  33499 2021-07-28 21:48:10 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-sa06a82d0c8a30997-c1                  33499 2021-07-28 21:58:10 UTC
xs26_ca421e8782cd1249ef893b537a07e7c7-s3ce7de8612bf8dde-c1                 469793 2021-07-29 07:51:51 UTC
xs26_ca421e8782cd1249ef893b537a07e7c7-s8c439fdc8e36b982-c1                 469793 2021-07-29 07:56:36 UTC
xs2_46298f31265b2cd6aca20f22836b6b8b-sc7c652f0646d21ad-c1                 1975209 2021-07-19 16:25:16 UTC
xs2_46298f31265b2cd6aca20f22836b6b8b-sdc66bfab64399c02-c1                 1975209 2021-07-19 16:25:16 UTC
xs2_46298f31265b2cd6aca20f22836b6b8b-sf8ab2223a1698fb0-c1                 1975209 2021-07-19 16:30:13 UTC
xs3_a799b8987de3d14a7d22311f9f113bde-s2b0eb5a2f3bfbb0e-c1                 4640181 2021-07-19 22:48:17 UTC
xs4_1c9a42aa338d34cfabeec4ccb0a0b870-s9ae5ce7fb46a442a-c1                 1846760 2021-07-20 05:03:37 UTC
xs5_9379d0834ee6f2fbeb5bd2db06444b47-s33088d96adf59ac3-c1                 5326131 2021-07-20 11:19:04 UTC
xs6_8b978096465b9c5e5857d10e88df7e31-s9c9f9a291f9cd917-c1                 2358714 2021-07-20 17:47:04 UTC
xs6_8b978096465b9c5e5857d10e88df7e31-sc7bc34663316f45c-c1                 2358714 2021-07-20 17:42:03 UTC
xs7_56359e4a359a77a53869909126e42c80-s3b56d7b203d86d4d-c1                 3176625 2021-07-21 00:08:53 UTC
xs7_56359e4a359a77a53869909126e42c80-sa8b4133ef9465287-c1                 3176625 2021-07-21 00:08:53 UTC
xs8_bd00f4b25e76caf4958072a9d70a63cd-sf431b69bd26e46cb-c1                 2342842 2021-07-21 06:39:43 UTC
xs9_043a65c6f33f1c6c87396546ad58adc1-s4dc9c9239364261c-c1                 1768573 2021-07-21 13:04:33 UTC
xs9_043a65c6f33f1c6c87396546ad58adc1-s5d56840861a5fa2c-c1                 1768573 2021-07-21 12:59:33 UTC
xs9_043a65c6f33f1c6c87396546ad58adc1-sa28120775aec8584-c1                 1768573 2021-07-21 12:59:19 UTC

xe28                                                                           12 2021-07-29 07:29:32 UTC
xe29                                                                           12 2021-07-29 16:44:07 UTC
xn27_1733c51d80d4956006c86dd1254d865c-s027004a2245ad409106-c1               16835 2021-07-29 00:04:19 UTC
xn27_1adaff83ac9579932a6f15b0cb18d67c-scb44300663963c7d106-c1                 367 2021-07-29 03:00:30 UTC
xn27_27dda36d27ac7b41286ec63b441d9bbe-sb819725ccfaf2612106-c1               35387 2021-07-29 03:27:52 UTC
xn27_28999b48965fd65df69147b38a0b294a-sa54ecf1d2f2b4402106-c1                 367 2021-07-29 06:00:22 UTC
xn27_31c27f2b09f5d4768d23d67c434d3c24-s2e207b55047943a5106-c1                 367 2021-07-29 07:01:42 UTC
xn27_3caf276e09a8c0a02b07eb9d7fa9b123-sf2686e8d780ae131106-c1               23785 2021-07-29 00:09:06 UTC
xn27_6addc5d7b96bfa214805b0bff50f118e-s541662d179996de5106-c1                 367 2021-07-29 04:00:27 UTC
xn27_81106330503bfdb011710b1dd3254c85-s19cdcf40dc4c2e85106-c1                 143 2021-07-29 00:01:39 UTC
xn27_878997b1dfccfd32b9f937aaa62fd749-s41b761663ebd3cd1106-c1               20677 2021-07-28 22:00:47 UTC
xn27_9e699c287c67e05c835ce4694332ec32-sd7270a18684b1a3f106-c1                 143 2021-07-29 00:01:21 UTC
xn27_adbe19a63f72dbf8ca684d1fc2be446d-s2e5b0baf1c6720de106-c1                 367 2021-07-29 05:00:25 UTC
xn27_b1eedddef178c6544044b76b84dad34f-s1367ae6349149dca106-c1                 549 2021-07-29 01:00:27 UTC
xn27_bdd828a99aa6896d03514bfb215cc709-sadc8303ad11af8cc106-c1               17567 2021-07-29 01:27:01 UTC
xn27_d06a8aa9a7f6a790510d0565c6e74fa2-sfa5d6a4216917e0f106-c1                8902 2021-07-29 02:10:18 UTC
xn27_da82f68217b8fce3d114863ace299b73-sd14e612614656994106-c1               36014 2021-07-29 05:28:54 UTC
xn27_df0ac0a14fd4d535380295368b63591f-s22440cdbf9ec171a106-c1                 447 2021-07-29 02:11:14 UTC
xn27_e3adc89b3e2641adf67a58234dfc2b3b-sad4ac6e3d63fc8e1106-c1               36344 2021-07-29 07:29:28 UTC
xn27_eeccdce5600d7eddf95f911647685b9c-s03ca4add6661b190106-c1               21255 2021-07-28 23:00:47 UTC
xn27_f04c57b117f81ff821acd07a2475a64f-sb9e6348e36de0514106-c1               16874 2021-07-28 23:25:53 UTC
xn27_f686a86ef2a854e36092dfe1a6f56ea8-s6bb2a88caf910dd5106-c1               16903 2021-07-29 00:04:33 UTC
xn28_036ea9242e9031aedb518a5a7bcda357-s274180050b83c367106-c1               37037 2021-07-29 11:30:21 UTC
xn28_12ac7b8de28613f31549290861b3251e-sf428290fa960d9c6106-c1                 367 2021-07-29 13:00:13 UTC
xn28_1df2acc1ff8bd5e5a8b55d3a3ca20481-sde303e63302f16b3106-c1               43736 2021-07-29 13:30:47 UTC
xn28_35350bf0d7f4f8ebc7a9ae178515ce1b-s6b0a942a31335462106-c1                 379 2021-07-29 14:00:12 UTC
xn28_62a9bb64e49a5e85e2d6aa9aac9944ef-sc29dbc3e7808c00a106-c1                 367 2021-07-29 12:00:12 UTC
xn28_79bf012a94162bbc0fc0de00f68df7e5-sc2739fd6eaba9cb7106-c1                 143 2021-07-29 16:44:08 UTC
xn28_9ae930202777f92a6f67aea71d3ba9e3-sc0e62f3933d232a9106-c1               36245 2021-07-29 09:29:59 UTC
xn28_a00140c74cf6e10ef5f4907e532a4a32-s6d819bac2082c60c106-c1               22554 2021-07-29 12:01:59 UTC
xn28_a04cec540216a63308d97c8282516421-scecbbeb619889b09106-c1                 143 2021-07-29 16:43:42 UTC
xn28_a1c6935e5c189ca3057142cf827ea630-se9d8a3325eae5bd7106-c1               14011 2021-07-29 11:16:53 UTC
xn28_a25c2ea052304ac15d1ea3430c677083-s8af77c6e93920cb8106-c1                 367 2021-07-29 09:00:16 UTC
xn28_aee9c88693d1924879e028e9d2f898fb-s9401e0aacb093897106-c1                 367 2021-07-29 08:00:20 UTC
xn28_ba699447b7ed61a04de818cf1e3b2d13-scc06891fc0cbe800106-c1               35245 2021-07-29 15:32:36 UTC
xn28_bb28f9b2a3a5d605cbfa7b62fb42afdf-s21243e193654bf05106-c1                 143 2021-07-29 16:43:18 UTC
xn28_cd4d4a236a84c713f76556cfd8e7567d-s6e2d9f27a2114ca0106-c1                 143 2021-07-29 13:52:03 UTC
xn28_d0b5766b535d0a701aa7733ab0faecad-s22058769e8c6827c106-c1                 367 2021-07-29 11:00:13 UTC
xn28_d3751c65aeeff5ae8061b58e348aa7fa-s523068a0996b1cef106-c1                 367 2021-07-29 10:00:20 UTC
xn28_d8730d4becb2acfbf4097ad80ac50c6f-sb9ff3b2a2d23eaa7106-c1                1236 2021-07-29 15:01:37 UTC
xn28_e03e87a06a12f2be91a3b66b7bf16541-sf062d84da9efff4b106-c1                 367 2021-07-29 16:01:11 UTC
xn28_e3ef09711cd0426e8c046e7185349002-s841dd9978678d84b106-c1              122499 2021-07-29 15:31:57 UTC
xn28_f7aa953f4e0bf45384795eb69ae4b41b-s0c5b16b52f4f269f106-c1                1025 2021-07-29 08:00:32 UTC
xn29_1323be714ffba1acd212bb16a7adc667-s3813aac1cad2ac64106-c1                 143 2021-07-29 16:49:44 UTC
xn29_23149a7d198079478a16e7c7d2536735-sf3feeb0c81717219106-c1                 345 2021-07-29 16:50:59 UTC
xn29_278df814652b0c4cd9cad12fc8461741-s13e7382007badd22106-c1                1261 2021-07-29 16:47:44 UTC
xn29_27a6746124413787ffc837ed51b5e81b-s6d573190214d48d0106-c1                 143 2021-07-29 16:47:35 UTC
xn29_8b478f28ab724221a9ee493d5deaeed3-s435137ec84f43dc4106-c1                1261 2021-07-29 16:45:54 UTC
xn29_8e10cc036226833854b624e6659bc767-sf2ab0739ee112e0c106-c1                 143 2021-07-29 16:48:07 UTC
xn29_aa0ba6830515f970e2fdcb55040ec93e-sd481a29162331665106-c1              257105 2021-07-29 16:50:38 UTC
xn29_d2f03cfcfcd2740161407c3ee513145c-s7f80e92a62d17b84106-c1                 143 2021-07-29 16:45:49 UTC
xr0_7_a51491a5adf4586b89812dac6079b2c6-s749180735e0c3863-c1              29769670 2021-07-21 00:08:54 UTC
xr16_23_8c1e98dac7caa9f5f3122fe37751db8f-s25a8ddb24e26a2d0-c1             2418091 2021-07-28 00:13:33 UTC
xr8_15_7af99ee0e821c7c510c6c6a630923f19-sf751c849cb4e8a64-c1              8964883 2021-07-24 13:36:29 UTC
xs0_65bc61e435c6e0310c0f3cdf4b37f7c7-s7dac514995098f5c-c1                 2502545 2021-07-18 19:34:30 UTC
xs0_65bc61e435c6e0310c0f3cdf4b37f7c7-s8368faca34516d9e-c1                 2502545 2021-07-18 19:38:33 UTC
xs0_65bc61e435c6e0310c0f3cdf4b37f7c7-sb58117e00facd7bc-c1                 2502545 2021-07-18 19:39:09 UTC
xs10_0611177c35a41d35043b7f61002fc65a-s6fe540e3f123e24b-c1                2175819 2021-07-21 19:29:59 UTC
xs10_0611177c35a41d35043b7f61002fc65a-s70df1ad4e7737c60-c1                2175819 2021-07-21 19:25:12 UTC
xs10_0611177c35a41d35043b7f61002fc65a-sb3a7d029b0e49028-c1                2175819 2021-07-21 19:29:29 UTC
xs11_443b4f6b3cd2a0ceefac3ca536ea64e1-sf89d526f56bcf520-c1                 910412 2021-07-22 01:57:01 UTC
xs12_4c91d0e82a209b0d839401165f50080c-s0f230a26d47c3294-c1                1212378 2021-07-22 11:01:42 UTC
xs12_4c91d0e82a209b0d839401165f50080c-s921d7bad5f9c1e97-c1                1212378 2021-07-22 11:01:42 UTC
xs12_4c91d0e82a209b0d839401165f50080c-saaafe96a86a0664e-c1                1212378 2021-07-22 11:06:20 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-s565d89651902142d-c1                1066869 2021-07-22 18:38:11 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-sa1e6ac95c7d88d62-c1                1066869 2021-07-22 18:33:01 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-sc2f31dc5b4e99722-c1                1066869 2021-07-22 18:33:11 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-sd6eb72ac6c2be892-c1                1066869 2021-07-22 18:43:11 UTC
xs13_229aeb12b654c6aaade116bd9b196c43-se3965dcf00b87e15-c1                1066869 2021-07-22 18:28:12 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-s46100eb98a29e137-c1                 406184 2021-07-23 17:39:58 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-s7746901b5295984f-c1                 406184 2021-07-23 17:29:57 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-s82ed4a70739848b6-c1                 406184 2021-07-23 17:34:50 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-sbb2f6b1f28a03798-c1                 406184 2021-07-23 17:25:05 UTC
xs14_2e1c0a11c49dfd60d44bb75c9dec8c47-sbee518b021596663-c1                 406184 2021-07-23 17:34:57 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s01a962beec41234e-c1                 376321 2021-07-24 13:46:17 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s390e0bdcf804c249-c1                 376321 2021-07-24 13:46:09 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s46cdbde36fe09d8e-c1                 376321 2021-07-24 13:36:28 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-s4b22d11fc7d72573-c1                 376321 2021-07-24 13:51:17 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-sbf03056e173796ca-c1                 376321 2021-07-24 13:36:28 UTC
xs15_55ddde2b781cb0f8617dc3450dee3a52-sdfd7ab84c16d8696-c1                 376321 2021-07-24 13:41:17 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s20ab68055bd33ec0-c1                 135222 2021-07-24 21:16:32 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s279164dba5f52c2a-c1                 135222 2021-07-24 21:01:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s349087eecb82c1d6-c1                 135222 2021-07-24 21:06:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s4edeb401eaaac819-c1                 135222 2021-07-24 21:11:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-s7174f521767765e5-c1                 135222 2021-07-24 21:16:41 UTC
xs16_ce0d80950b1f12d4e9d4ed0a2d075501-sb7cee941ba87a7e3-c1                 135222 2021-07-24 21:01:32 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-s7cab4d79c628c22b-c1                 139956 2021-07-25 09:27:22 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-s7ff5b2afbdefa983-c1                 139956 2021-07-25 09:22:30 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-s8c174a5d95b54b17-c1                 139956 2021-07-25 09:32:22 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-sba367a6f2279c5d3-c1                 139956 2021-07-25 09:31:46 UTC
xs17_144e9af23d2d88fdbb1f87115e457c2e-sebdfadabdfc4d3a5-c1                 139956 2021-07-25 09:37:22 UTC
xs18_1697185502810f85c252be5960e1e982-s394e00ccd43808a1-c1                  44344 2021-07-25 18:31:55 UTC
xs18_1697185502810f85c252be5960e1e982-s50ed18d05f1840c5-c1                  44344 2021-07-25 18:27:44 UTC
xs18_1697185502810f85c252be5960e1e982-s67b996f4be5f4694-c1                  44344 2021-07-25 18:32:45 UTC
xs18_1697185502810f85c252be5960e1e982-sf2c36950ac7a3a87-c1                  44344 2021-07-25 18:37:45 UTC
xs18_1697185502810f85c252be5960e1e982-sffdfa67d65144e61-c1                  44344 2021-07-25 18:22:54 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s2c58a47f17494a2f-c1                1982778 2021-07-26 05:18:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s33bd3c7ac756b0fa-c1                1982778 2021-07-26 05:13:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s46d2c32562c2acdf-c1                1982778 2021-07-26 05:03:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s5c144f379019b994-c1                1982778 2021-07-26 05:08:13 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-s9c0deaa4d8cbdfa2-c1                1982778 2021-07-26 05:17:05 UTC
xs19_84bc374987c3be5d1eaeaeb2f852cf2c-sc524f88d6df7a081-c1                1982778 2021-07-26 05:03:07 UTC
xs1_3371b43e01e0ca9013e89f5499f8cf38-s01a9640da4047515-c1                11664987 2021-07-19 10:10:21 UTC
xs1_3371b43e01e0ca9013e89f5499f8cf38-s5b5ff3f5b175ef7e-c1                11664987 2021-07-19 10:14:46 UTC
xs1_3371b43e01e0ca9013e89f5499f8cf38-s616f5b68e3824bef-c1                11664987 2021-07-19 10:10:21 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-s92c239caa39419fb-c1                 184581 2021-07-26 15:13:37 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-s934af1db00aa78b5-c1                 184581 2021-07-26 15:23:37 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-saa903d3dc10bf3b5-c1                 184581 2021-07-26 15:17:12 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-sbbecb6139e106d87-c1                 184581 2021-07-26 15:08:44 UTC
xs20_18a34ac6b831b7a92602dc23b6c4c6e3-sf4db485e7c74b2ab-c1                 184581 2021-07-26 15:18:37 UTC
xs21_7e65a99a3113db0337302df039a74b79-s41aac18478a8b7dc-c1                 164816 2021-07-27 02:34:06 UTC
xs21_7e65a99a3113db0337302df039a74b79-s4e465fe2239475a0-c1                 164816 2021-07-27 02:32:29 UTC
xs21_7e65a99a3113db0337302df039a74b79-s6d86a0b4ad34ef9a-c1                 164816 2021-07-27 02:39:06 UTC
xs21_7e65a99a3113db0337302df039a74b79-s71f97d63243b6110-c1                 164816 2021-07-27 02:24:11 UTC
xs21_7e65a99a3113db0337302df039a74b79-sd544e60ea312e21a-c1                 164816 2021-07-27 02:29:05 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-s1699aa399523dab2-c1                  43157 2021-07-27 13:11:47 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-s63aaf109ac1f0525-c1                  43157 2021-07-27 13:16:48 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-sa5670f9ffbdd7bf9-c1                  43157 2021-07-27 13:11:32 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-sc346e7ade1c87b97-c1                  43157 2021-07-27 13:26:33 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-scf77473bf4f75097-c1                  43157 2021-07-27 13:21:49 UTC
xs22_952ab5c0e3a05f0d89303352cccb01ca-se99e24e9a2749b66-c1                  43157 2021-07-27 13:06:47 UTC
xs23_e7fd20cc21d8dd4fc804bc2b6a5adab1-scbf8e11f8cca04af-c1                  84706 2021-07-28 00:13:33 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s31652e2463d951df-c1                  96230 2021-07-28 11:17:58 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s551bcc9398650602-c1                  96230 2021-07-28 11:00:35 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s73356840ce807865-c1                  96230 2021-07-28 11:10:26 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-s8bd3863ff6ff8a35-c1                  96230 2021-07-28 11:02:58 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-saf0dcab4fb8d4f35-c1                  96230 2021-07-28 11:05:26 UTC
xs24_562981201a9fad8f4de0ef73992d01ff-sede157bb0415e2ce-c1                  96230 2021-07-28 11:15:26 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s1528eb2dd7b3a4e4-c1                  33499 2021-07-28 21:53:10 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s627ec9f1a8df335b-c1                  33499 2021-07-28 21:43:11 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s7cf734f0abb990ee-c1                  33499 2021-07-28 21:57:42 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-s8c57bbd9bca13eb7-c1                  33499 2021-07-28 21:48:10 UTC
xs25_1f3f5f2251f03471b50ac4a19ab83269-sa06a82d0c8a30997-c1                  33499 2021-07-28 21:58:10 UTC
xs26_ca421e8782cd1249ef893b537a07e7c7-s3ce7de8612bf8dde-c1                 469793 2021-07-29 07:51:51 UTC
xs26_ca421e8782cd1249ef893b537a07e7c7-s8c439fdc8e36b982-c1                 469793 2021-07-29 07:56:36 UTC
xs27_94764b64e725d75beecced68caecd153-sc186fe18c6a30252-c1                 119748 2021-07-29 16:44:45 UTC
xs2_46298f31265b2cd6aca20f22836b6b8b-sc7c652f0646d21ad-c1                 1975209 2021-07-19 16:25:16 UTC
xs2_46298f31265b2cd6aca20f22836b6b8b-sdc66bfab64399c02-c1                 1975209 2021-07-19 16:25:16 UTC
xs2_46298f31265b2cd6aca20f22836b6b8b-sf8ab2223a1698fb0-c1                 1975209 2021-07-19 16:30:13 UTC
xs3_a799b8987de3d14a7d22311f9f113bde-s2b0eb5a2f3bfbb0e-c1                 4640181 2021-07-19 22:48:17 UTC
xs4_1c9a42aa338d34cfabeec4ccb0a0b870-s9ae5ce7fb46a442a-c1                 1846760 2021-07-20 05:03:37 UTC
xs5_9379d0834ee6f2fbeb5bd2db06444b47-s33088d96adf59ac3-c1                 5326131 2021-07-20 11:19:04 UTC
xs6_8b978096465b9c5e5857d10e88df7e31-s9c9f9a291f9cd917-c1                 2358714 2021-07-20 17:47:04 UTC
xs6_8b978096465b9c5e5857d10e88df7e31-sc7bc34663316f45c-c1                 2358714 2021-07-20 17:42:03 UTC
xs7_56359e4a359a77a53869909126e42c80-s3b56d7b203d86d4d-c1                 3176625 2021-07-21 00:08:53 UTC
xs7_56359e4a359a77a53869909126e42c80-sa8b4133ef9465287-c1                 3176625 2021-07-21 00:08:53 UTC
xs8_bd00f4b25e76caf4958072a9d70a63cd-sf431b69bd26e46cb-c1                 2342842 2021-07-21 06:39:43 UTC
xs9_043a65c6f33f1c6c87396546ad58adc1-s4dc9c9239364261c-c1                 1768573 2021-07-21 13:04:33 UTC
xs9_043a65c6f33f1c6c87396546ad58adc1-s5d56840861a5fa2c-c1                 1768573 2021-07-21 12:59:33 UTC
xs9_043a65c6f33f1c6c87396546ad58adc1-sa28120775aec8584-c1                 1768573 2021-07-21 12:59:19 UTC

It also seems that after that maintenance, the previously deleted snapshots returned from being deleted.

23

PSA: Just released a major update to this feature - this should fix a bug where maintenance in epoch mode did not correctly delete unused blobs and addresses few additional mostly cosmetic issues.

Please upgrade Kopia to Release v20210802.0.210854 · kopia/kopia-test-builds · GitHub or later from unstable APT/RPM channel.

Thank you and please report any issues here, as usual,

Jarek

24

Done, updated all systems. Thank you.

25

Seems to be working correctly for me:

Looking for unreferenced blobs...
[...]
Deleted total 1206 unreferenced blobs (27.1 GB)
26

I am having a strange issue, where after some time, a the folder own-writes seems to get missing from the cache directory on my Kopia server. I am reporting this here, since the error log indicate an issue with the epoch-manager:

2021-08-15 08:41:54.3289506 [epoch-manager] refresh attempt failed: error refreshing: error loading full checkpoints: error listing all blobs: error listing all blobs: error listing cached blobs: error listing all blobs: error processing directory shards: error reading directory: open /kopia/cache/ST7A/own-writes: no such file or directory, sleeping 15s before next retry

The result of this folder going missin is that the snapshot of my Kopia clients doesn’t finish, although it gets done workwise, its just that is doesn’t complete. When I simply created the missing folder, the still running snapshot would then complete after a couple of seconds.

Looking into the cache directory while the snapshot was still waiting to finish and after I manually created the missing folder and the snapshot finished, was quite interesting:

[root@kopia content-logs]# cd …
[root@kopia repos]# cd /kopia/cache/ST7A/
[root@kopia ST7A]# ls
metadata
[root@kopia ST7A]# mkdir own-writes
[root@kopia ST7A]# ls
blob-list indexes metadata own-writes
[root@kopia ST7A]# ls -l own-writes/
insgesamt 0

27

UI tried to fire quick maintenance by mistake today, which resulted in the following:

16:21:07.295	reloading committed manifest contents: rev=12 last=0
16:21:07.295	listing manifest contents
16:21:07.297	updating timestamp on /Users/zyradyl/Library/Caches/kopia/8dcc95c79a16f794/metadata/q0/a1360db61db90410bf862d8aa3fea1f-s23a8b6328674851a107.f to 2021-08-15 16:21:07.297715 -0500 CDT m=+1804.307710585
16:21:07.298	updating timestamp on /Users/zyradyl/Library/Caches/kopia/8dcc95c79a16f794/metadata/q3/a574b6edd365ece227ed958ee8bcfaf-sd3ef841af46c67a3107.f to 2021-08-15 16:21:07.298143 -0500 CDT m=+1804.308138293
16:21:07.301	updating timestamp on /Users/zyradyl/Library/Caches/kopia/8dcc95c79a16f794/metadata/qd/7b0699ed1354b223004f067774f34a7-s00c90cd6beb0853a107.f to 2021-08-15 16:21:07.301028 -0500 CDT m=+1804.311023876
16:21:07.302	updating timestamp on /Users/zyradyl/Library/Caches/kopia/8dcc95c79a16f794/metadata/q0/67d400ce51a8429d3701aab43d2739d-s178f3c31fcb5ec86107.f to 2021-08-15 16:21:07.302572 -0500 CDT m=+1804.312567543
16:21:07.303	updating timestamp on /Users/zyradyl/Library/Caches/kopia/8dcc95c79a16f794/metadata/q2/71d2f32874ebef1bf0e7e6df34c1f15-s8d70abfa577dc4b9107.f to 2021-08-15 16:21:07.303996 -0500 CDT m=+1804.313991418
16:21:07.304	updating timestamp on /Users/zyradyl/Library/Caches/kopia/8dcc95c79a16f794/metadata/qf/d295d3bc0edc0be855e93ec27f901b0-s5e3429b08561f028107.f to 2021-08-15 16:21:07.304382 -0500 CDT m=+1804.314377001
16:21:07.405	not due for full maintenance cycle until 2021-08-18 12:05:34.96295 -0500 CDT
16:21:07.405	due for quick maintenance cycle
16:21:07.442	scheduling next quick cycle at 2021-08-15 16:18:39.282476 -0500 CDT
16:21:07.739	Acquiring maintenance lock in file /Users/zyradyl/Library/Application Support/kopia/repository.config.mlock
16:21:07.741	Running quick maintenance...
16:21:08.450	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#0), sleeping for 100ms before retrying
16:21:08.451	got error ReadAll: context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#0), sleeping for 100ms before retrying
16:21:08.509	got error Put "https://s3.us-east-2.wasabisys.com/tokisaki/xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1": context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#0), sleeping for 100ms before retrying
16:21:08.551	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#1), sleeping for 150ms before retrying
16:21:08.551	got error ReadAll: context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#1), sleeping for 150ms before retrying
16:21:08.609	got error Put "https://s3.us-east-2.wasabisys.com/tokisaki/xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1": context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#1), sleeping for 150ms before retrying
16:21:08.701	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#2), sleeping for 225ms before retrying
16:21:08.701	got error ReadAll: context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#2), sleeping for 225ms before retrying
16:21:08.725	quick maintenance not required for epoch manager
16:21:08.725	Finished quick maintenance.
16:21:08.760	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#2), sleeping for 225ms before retrying
16:21:08.926	got error ReadAll: context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#3), sleeping for 337.5ms before retrying
16:21:08.927	got error ReadAll: context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#3), sleeping for 337.5ms before retrying
16:21:08.985	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#3), sleeping for 337.5ms before retrying
16:21:09.265	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#4), sleeping for 506.25ms before retrying
16:21:09.266	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1": context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#4), sleeping for 506.25ms before retrying
16:21:09.324	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#4), sleeping for 506.25ms before retrying
16:21:09.773	got error ReadAll: context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#5), sleeping for 759.375ms before retrying
16:21:09.773	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#5), sleeping for 759.375ms before retrying
16:21:09.831	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#5), sleeping for 759.375ms before retrying
16:21:10.533	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1": context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#6), sleeping for 1.1390625s before retrying
16:21:10.533	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#6), sleeping for 1.1390625s before retrying
16:21:10.591	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#6), sleeping for 1.1390625s before retrying
16:21:11.673	got error ReadAll: context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#7), sleeping for 1.70859375s before retrying
16:21:11.673	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#7), sleeping for 1.70859375s before retrying
16:21:11.731	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#7), sleeping for 1.70859375s before retrying
16:21:13.383	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1": context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#8), sleeping for 2.562890625s before retrying
16:21:13.383	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1": context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#8), sleeping for 2.562890625s before retrying
16:21:13.440	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#8), sleeping for 2.562890625s before retrying
16:21:15.947	got error ReadAll: context canceled when GetBlob(xn21_6162bc1b3c20f964ad5e55394b43e053-scc28ef53654b311c107-c1,0,-1) (#9), sleeping for 3.844335937s before retrying
16:21:15.947	got error ReadAll: Get "https://s3.us-east-2.wasabisys.com/tokisaki/xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1": context canceled when GetBlob(xn22_94d5ba958ca89d8671416f64fcc33761-s2a85f2bd2cef7d79107-c1,0,-1) (#9), sleeping for 3.844335937s before retrying
16:21:16.004	got error context canceled when PutBlob(xs18_e4072da40b0e0491e9f2c46a32dc3d3d-s9e2a21eadea22f41-c1) (#9), sleeping for 3.844335937s before retrying

after that i took a look at kopia index epoch list and i’m missing epochs?

❯ kopia index epoch list
Current Epoch: 24
Epoch Started  2021-08-15 11:00:48 CDT
24 2021-08-15 11:01:00 CDT ... 2021-08-15 16:30:40 CDT, 79 blobs, 2.7 MiB, span 5h29m40s
23 2021-08-15 05:00:22 CDT ... 2021-08-15 11:10:11 CDT, 33 blobs, 742 KiB, span 6h9m49s
20: 2021-08-14 18:08:05 CDT single-epoch 1 blobs, 653.9 KiB
19: 2021-08-14 12:50:05 CDT single-epoch 1 blobs, 327 KiB
17: 2021-08-14 13:02:24 CDT single-epoch 1 blobs, 818.1 KiB
16: 2021-08-13 17:24:55 CDT single-epoch 1 blobs, 658.4 KiB
0-7: range, 1 blobs, 101 MiB
8-15: range, 1 blobs, 5.1 MiB

content verify passes, so nothing is damaged, but this seemed a little strange to me so i thought i’d let you know.

28

Thanks for the report. Can you re-run index epoch list and also kopia blob list --prefix=x ?

29

sure thing:

❯ kopia index epoch list
Current Epoch: 28
Epoch Started  2021-08-16 14:52:34 CDT
28 2021-08-16 15:10:10 CDT ... 2021-08-16 19:40:11 CDT, 31 blobs, 229.9 KiB, span 4h30m1s
27 2021-08-16 08:50:08 CDT ... 2021-08-16 15:00:12 CDT, 34 blobs, 270.1 KiB, span 6h10m4s
26: 2021-08-16 16:03:40 CDT single-epoch 1 blobs, 465.1 KiB
0-7: range, 1 blobs, 101 MiB
8-15: range, 1 blobs, 5.1 MiB
16-23: range, 1 blobs, 7.1 MiB

as for kopia blob list --prefix=x it’s huge, so here is a link to it http://miyuki.zyradyl.moe/kopia/prefix.log

i also went ahead and ran kopia maintenance set --enable-quick=false after this happened as just a preventative measure, i can enable it again and see if it happens once more?

30

Ok, this is just a UX issue, not that epochs are lost. Can you run kopia index epoch list couple times and see if settles?

31

seems that it has stabilized now, output is remaining the same as in my previous post after three runs. i’ll keep an eye on it for a little while and see if the issue pops back up.

32

Is v0.9 going to be out soon? I’m waiting for it to deploy Kopia in prod on some servers.

Also, is testers still needed? I could test on my desktop.

33

All the required changes are now in and I’m planning to cut 0.9.0-rc1 this weekend. Will post an announcement here.

1 Like
34

And there it is. Magnificent. Great job :slight_smile: Fantastic to see ransomware protection built in!

35

Hi,

I’m currently using RC3 with upgraded repo format, I now get the following line displayed in red at the end the output of kopia snap list:

default (64KiB) - allocated 26 chunks freed 25 alive 1 max 4 free list high water mark: 4

It was not the case with previous versions.
What does it mean?

Thanks

36

Same here. Is it possible to disable the allocator statistics?

37

It’s been removed in latest code. I’ll make v0.9.1 release over the weekend with this and few other changes.

1 Like
38

Yesterday I finally updated my main Kopia repo, which holds the snapshots of 358 individual Samba shares in approx. 60T from 0.8.4 to 0.9.5 and performed the update to the new repo format as well.

The update went quite smoothly in a couple of minutes and the follwing

kopia content verify

Took approx 18 or so hours and came out clean. What a great piece of software this is! :grinning:

1 Like
39

Well… there’s nothing in this world, which cannot be made even better, is there? I was lucky enough to have @jkowalski leading me through a series of steps to drastically improve the behaviour of

kopia content verify

and

kopia snapshot verify

on my 60TB repo which is hosted on a ZFS system. I am happy to report that the performance of e.g. kopia content verify has increased by 800+ times (not % !). The key to this success was to alter the sharding,the way kopia handles the shard files on the local file system and adjusting this to better suit ZFS behaviour. Also is there a switch to have kopia perform multiple list ops in parallel, which also helped a lot.

So, first bringing down the runtime from 18 hrs to 12 hrs by increasing the number of list ops, the ultimate hit was to have kopia reshard its files to a more flattened structure, which brought the runtime down from 12 hrs to unbeliveable 49s - and thats were times 800x is derived from. Love it! :heart: