Trying to understand the CLI progress bar.
Have a long running snapshot running since last night and when I checked it this AM noticed that the most current line had a number hashed lower than previous line. Is that because of checkpoint and need to start rehash? How about the percentage all the way to the right? is it lower because of the rehash?
Even I doubt the progess bar.
I was backing up a single vmdk file around 85gb the progess stated going above 87g 
I will test this in a fresh new repo later
I had come accross the post that the UI shows incorrect percentage there seems to be something even in the cli
Is there a verbose flag for debugging
Yes, progress bar was all messed up for long uploads due to checkpointing. I have a fix in the works. See this topic for info on upcoming improvements.
I tried this with and without the --progress flag in 0.07-rc1 and saw not difference in the output. It is not clear to me what the items are trying to say. The last % went to 100% and stayed there while the backup continued.
Besides when the backup is run in the background with systemd you donāt get to see the output. I have to pipe it to file and try to determine the progress by looking at the logs and compare the number of finished reading directory log outputs with the list length given by kopia snapshot estimate.
The percentage Kopia is showing is compared to last known snapshot of the directory, if it changed significantly or if the previous snapshot was incomplete, it will indeed sit at 100% for a long time.
What are you suggesting we do instead? I really donāt want to pollute the directory name to the output.
For a while I considered scanning the source tree in parallel to compute the actual source size which would make the progress bar accurate, but this will slow down the snapshotting process since weāll be reading each directory twice.
As a new user it isnāt clear what the numbers are saying. Now you clarify that it is a directory by directory comparison Iām not sure what the value is to the user. A user wants to know the overall progress in getting to the end of a snapshot. Directory based progress is secondary although maybe important for debugging or other purposes.
Even at a directory level of overall progress it doesnāt seem too computationaly consuming to calculate the number of directories (a one time pass at the start) that are to be processed and then counting them off as you go to give an overall percentage perhaps?
At the moment I have to use 2 logging files to capture the outputs. All logs to file are most important I think because with systemd running as a background process the file logs are all you have. The logging is also alot of work to set up and capture for every command as I like to store my logs with each repo for easier lookup. e.g. my create command and other commands have to look like this.
kopia snapshot create \
-p "$PW" \
--description "" \
--progress \
--force-hash 0 \
--log-dir "${LOGDIR}" \
--log-file "${LOGPATH}" \
--file-log-level debug \
--log-level info \
"${SOURCE[i]}" |& log
from which I get the overall progress by counting the directories completed.
I have a pending change to improve the progress indicator by computing the size of the directory tree to be uploaded in parallel to the actual upload.
This estimation tends to complete very quickly and usually after few seconds weāre able to display nice percentage progress and relatively precise ETA:
https://asciinema.org/a/O7ktcWSgaGUPfJwhzc65mMWM1
Let me know whether it makes sense. BTW, please donāt rely on parsing the logs as their contents are not guaranteed. Itās better to make the required changes in the code directly if you think those are going to be beneficial to others.
Seems to me that you are still only tallying up the progress of each directory not the overall progress of the job.
With what you displayed in the video I would change the arrangement from:
* 0 hashing, 27 hashed (1.8 MB), 139709 cached (22.3 GB), 3 uploaded (37.5 KB), 0 errors, estimated 22.3 GB (100.0%) 0s left
to (with the correct text added in where Iāve added a question e.g. [files/chuncks/?] or as appropriate). Also added an overall progress section.
* 0 hashing [files/chuncks/?], 27 hashed (1.8 MB), 139709 [files/chuncks/?] cached (22.3 GB / 22.3 GB (100.0%) 0s remaining), 3 uploaded [files/chuncks/?] (37.5 KB), 0 errors | Directories (or Files) processed: 2345 / 245000 (45%) 1d:3h:45m:10s remaining
My initial snapshots are taking anything up to 30 hours to complete so the overall progress is helpful.
The estimation process is tallying up the file count and total file size before any deduplication. This is exactly the same as during upload so should be accurate.
This is now released in v0.7.0-rc2
Oh, I think I understand what you meant by āSeems to me that you are still only tallying up the progress of each directory not the overall progress of the job.ā. This command is actually taking multiple snapshots in sequence, each of them has separate progress bar.
Is there some way this progress bar could be logged to its own file as a feature request?
Added with each update of its output to a new line?
In its current form it is difficult to capture with its ācarraige returnā (CR ā\rā) (0x0D) character to a file via a pipe.
(sorry for bumping an old thread - this is the thread that comes up when searching for carriage return or pipe or cli progress, and I think this will be helpful to others)
Iām running kopia snapshot in kubernetes and I was having trouble capturing progress logs also. The problems were that every progress update overwrites the previous (rather than being on a new line) and that the stdout and stderr was buffered so that I would get nothing and then all the progress reports at the end after it finished. This was slightly annoying for centralised logging.
The solution I found was to run kopia as:
kopia snapshot create [dir] --progress-update-interval 1m 2>&1 | stdbuf -oL -eL tr '\r' '\n'
This reduces the updates to 1 every minute, and then more importantly, stdbuf sets the stdout and stderr buffer to āline modeā so it prints after every line, and tr '\r' '\n' replaces the carriage return \r (used for overwriting a cli line) with a ātrueā newline, so that each progress line is printed on a new line. This makes the progress updates act in a way thatās more helpful for pipes and central logging.