# CLI progress bar **URL:** \ **Category:** Support\ **Created:** [September 8, 2020, 11:42am UTC](https://kopia.discourse.group/t/cli-progress-bar/97 "2020-09-08T11:42:31Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![francisco](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/francisco/32/40_2.png) [@francisco](https://kopia.discourse.group/u/francisco)\ **Post date:** [September 8, 2020, 11:42am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/1 "2020-09-08T11:42:31Z")
Trying to understand the CLI progress bar. ![Screen Shot 2020-09-08 at 7.26.03 AM](https://global.discourse-cdn.com/free1/uploads/kopia/original/1X/a787375db83c3f3d90ee21a892afa5e494206bd4.png) Have a long running snapshot running since last night and when I checked it this AM noticed that the most current line had a number hashed lower than previous line. Is that because of checkpoint and need to start rehash? How about the percentage all the way to the right? is it lower because of the rehash? ---
**Author:** ![saviodsouza](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/saviodsouza/32/50_2.png) [@saviodsouza](https://kopia.discourse.group/u/saviodsouza)\ **Post date:** [September 8, 2020, 5:38pm UTC](https://kopia.discourse.group/t/cli-progress-bar/97/2 "2020-09-08T17:38:42Z")
Even I doubt the progess bar. I was backing up a single vmdk file around 85gb the progess stated going above 87g šŸ¤” I will test this in a fresh new repo later I had come accross the post that the UI shows incorrect percentage there seems to be something even in the cli Is there a verbose flag for debugging ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 9, 2020, 6:07am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/3 "2020-09-09T06:07:16Z")
Yes, progress bar was all messed up for long uploads due to checkpointing. I have a fix in the works. See [this topic](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90) for info on upcoming improvements. Or this: [https://asciinema.org/a/eSHQUHBduFAZBmTCAyknx2Nkr](https://asciinema.org/a/eSHQUHBduFAZBmTCAyknx2Nkr) ---
**Author:** ![ted](https://avatars.discourse-cdn.com/v4/letter/t/db5fbb/32.png) [@ted](https://kopia.discourse.group/u/ted)\ **Post date:** [September 16, 2020, 12:49pm UTC](https://kopia.discourse.group/t/cli-progress-bar/97/4 "2020-09-16T12:49:42Z")
I tried this with and without the `--progress` flag in 0.07-rc1 and saw not difference in the output. It is not clear to me what the items are trying to say. The last % went to 100% and stayed there while the backup continued. Besides when the backup is run in the background with systemd you donā€™t get to see the output. I have to pipe it to file and try to determine the progress by looking at the logs and compare the number of `finished reading directory` log outputs with the list length given by `kopia snapshot estimate`. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 17, 2020, 5:26am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/5 "2020-09-17T05:26:48Z")
The percentage Kopia is showing is compared to last known snapshot of the directory, if it changed significantly or if the previous snapshot was incomplete, it will indeed sit at 100% for a long time. What are you suggesting we do instead? I really donā€™t want to pollute the directory name to the output. For a while I considered scanning the source tree in parallel to compute the actual source size which would make the progress bar accurate, but this will slow down the snapshotting process since weā€™ll be reading each directory twice. ---
**Author:** ![ted](https://avatars.discourse-cdn.com/v4/letter/t/db5fbb/32.png) [@ted](https://kopia.discourse.group/u/ted)\ **Post date:** [September 17, 2020, 11:49am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/6 "2020-09-17T11:49:11Z")
As a new user it isnā€™t clear what the numbers are saying. Now you clarify that it is a directory by directory comparison Iā€™m not sure what the value is to the user. A user wants to know the overall progress in getting to the end of a snapshot. Directory based progress is secondary although maybe important for debugging or other purposes. Even at a directory level of overall progress it doesnā€™t seem too computationaly consuming to calculate the number of directories (a one time pass at the start) that are to be processed and then counting them off as you go to give an overall percentage perhaps? At the moment I have to use 2 logging files to capture the outputs. All logs to file are most important I think because with systemd running as a background process the file logs are all you have. The logging is also alot of work to set up and capture for every command as I like to store my logs with each repo for easier lookup. e.g. my `create` command and other commands have to look like this. ```auto kopia snapshot create \ -p "$PW" \ --description "" \ --progress \ --force-hash 0 \ --log-dir "${LOGDIR}" \ --log-file "${LOGPATH}" \ --file-log-level debug \ --log-level info \ "${SOURCE[i]}" |& log ``` from which I get the overall progress by counting the directories completed. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 18, 2020, 2:21am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/7 "2020-09-18T02:21:51Z")
I have a pending change to improve the progress indicator by computing the size of the directory tree to be uploaded in parallel to the actual upload. This estimation tends to complete very quickly and usually after few seconds weā€™re able to display nice percentage progress and relatively precise ETA: [https://asciinema.org/a/O7ktcWSgaGUPfJwhzc65mMWM1](https://asciinema.org/a/O7ktcWSgaGUPfJwhzc65mMWM1) Let me know whether it makes sense. BTW, please donā€™t rely on parsing the logs as their contents are not guaranteed. Itā€™s better to make the required changes in the code directly if you think those are going to be beneficial to others. ---
**Author:** ![ted](https://avatars.discourse-cdn.com/v4/letter/t/db5fbb/32.png) [@ted](https://kopia.discourse.group/u/ted)\ **Post date:** [September 18, 2020, 5:13am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/8 "2020-09-18T05:13:02Z")
Seems to me that you are still only tallying up the progress of each directory not the overall progress of the job. With what you displayed in the video I would change the arrangement from: ```auto * 0 hashing, 27 hashed (1.8 MB), 139709 cached (22.3 GB), 3 uploaded (37.5 KB), 0 errors, estimated 22.3 GB (100.0%) 0s left ``` to (with the correct text added in where Iā€™ve added a question e.g. [files/chuncks/?] or as appropriate). Also added an overall progress section. ```auto * 0 hashing [files/chuncks/?], 27 hashed (1.8 MB), 139709 [files/chuncks/?] cached (22.3 GB / 22.3 GB (100.0%) 0s remaining), 3 uploaded [files/chuncks/?] (37.5 KB), 0 errors | Directories (or Files) processed: 2345 / 245000 (45%) 1d:3h:45m:10s remaining ``` My initial snapshots are taking anything up to 30 hours to complete so the overall progress is helpful. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 18, 2020, 5:47pm UTC](https://kopia.discourse.group/t/cli-progress-bar/97/9 "2020-09-18T17:47:30Z")
The estimation process is tallying up the file count and total file size before any deduplication. This is exactly the same as during upload so should be accurate. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 18, 2020, 6:51pm UTC](https://kopia.discourse.group/t/cli-progress-bar/97/10 "2020-09-18T18:51:48Z")
This is now released in v0.7.0-rc2 ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 19, 2020, 3:33pm UTC](https://kopia.discourse.group/t/cli-progress-bar/97/11 "2020-09-19T15:33:52Z")
Oh, I think I understand what you meant by ā€œSeems to me that you are still only tallying up the progress of each directory not the overall progress of the job.ā€. This command is actually taking multiple snapshots in sequence, each of them has separate progress bar. ---
**Author:** ![ted](https://avatars.discourse-cdn.com/v4/letter/t/db5fbb/32.png) [@ted](https://kopia.discourse.group/u/ted)\ **Post date:** [September 22, 2020, 11:57am UTC](https://kopia.discourse.group/t/cli-progress-bar/97/12 "2020-09-22T11:57:09Z")
Is there some way this progress bar could be logged to its own file as a feature request? Added with each update of its output to a new line? In its current form it is difficult to capture with its ā€œcarraige returnā€ (CR ā€˜\rā€™) (0x0D) character to a file via a pipe. ---
**Author:** ![bkanuka](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/bkanuka/32/918_2.png) [@bkanuka](https://kopia.discourse.group/u/bkanuka)\ **Post date:** [January 29, 2024, 1:48pm UTC](https://kopia.discourse.group/t/cli-progress-bar/97/13 "2024-01-29T13:48:02Z")
(sorry for bumping an old thread - this is the thread that comes up when searching for carriage return or pipe or cli progress, and I think this will be helpful to others) Iā€™m running `kopia snapshot` in kubernetes and I was having trouble capturing progress logs also. The problems were that every progress update overwrites the previous (rather than being on a new line) and that the stdout and stderr was buffered so that I would get nothing and then _all_ the progress reports at the end after it finished. This was slightly annoying for centralised logging. The solution I found was to run kopia as: ```auto kopia snapshot create [dir] --progress-update-interval 1m 2>&1 | stdbuf -oL -eL tr '\r' '\n' ``` This reduces the updates to 1 every minute, and then more importantly, `stdbuf` sets the stdout and stderr buffer to ā€œline modeā€ so it prints after every line, and `tr '\r' '\n'` replaces the carriage return `\r` (used for overwriting a cli line) with a ā€œtrueā€ newline, so that each progress line is printed on a new line. This makes the progress updates act in a way thatā€™s more helpful for pipes and central logging.