PROPOSAL: Kopia Telemetry Collection

jkowalski · October 5, 2022, 5:37am

FYI: I just created a GitHub issue to discuss telemetry collection in Kopia. Comments much appreciated.

PROPOSAL: Kopia Telemetry Collection

opened 05:31AM - 05 Oct 22 UTC

onboarding-experience discussion

## Summary In order to better understand how people use Kopia we’d like to ad…d telemetry feature which will collect and aggregate some basic information about installations and usage in a privacy-preserving manner. The data collected will be used to prioritize feature development and bug fixes. This document proposes the scope of data collection to get community consensus on what kind of information is ok to collect. Please post feedback as comments on this issue. ## Goals We will collect minimal amount of non-personally-identifying information to be able to answer the following questions: * Which versions of Kopia are active in the wild * What’s the respective usage of each version * How many repositories are active? * How big are the repositories, how many users typically share the repository, etc. * What is the age and size of the repositories * What providers are being used? * Which operating systems and CPU architectures are most commonly used * What’s the ratio of UI vs CLI usage All collected information will be made public in the form of public dashboards on https://kopia.io/ showing trends over time and charts showing respective usage percentages. # Collection Method Starting with v0.13.0 we will instrument Kopia clients (both CLI and UI) to `POST` the following information to central endpoint https://telemetry.kopia.io/ping no more than once every 24 hours from each client. The mechanism will be very similar to one currently used for upgrade checks. ``` { “repositoryInfo”: { “createDate”: “2022-10-05”, “repositoryUniqueId”: “<random-id-assigned-when-repository-is-created>” }, “clientInfo”: { “randomConnectionId”: “<random-id-assigned-per-connection-to-repository>”, “version”: “0.13.0”, “os”: “windows|darwin|linux|freebsd|openbsd”, “arch”: “amd64|arm64|arm”, “ui”: true|false, “isMaintenanceOwner”: true|false, }, “formatInfo”: { “provider”: “s3|b2|gcs|filesystem|...”, “hash”: “<algorithm>”, “encryption”: “<algorithm>”, “splitter”: “<algorithm>”, “formatVersion”: <number>, “indexFormatVersion”: <version>, }, “stats”: { “repositorySizeGB”: <number>, “numSnapshots”: <number>, // number of snapshots in the repository “numSources”: <number>, // number of unique username@hostname:/path “numPolicies”: <number>, // number of policies “numUsers”: <number>, // number of users sharing the repository “numHosts”: <number>, // number of hosts sharing the repository }, } ``` ## Opt-Out Method Collection will be enabled by default and clearly explained to the users giving them instructions for opting out at the moment of connection. The first ping will be sent no sooner than 24 hours after establishing connection to a repository. To opt out, the clients can: * Set environment variable KOPIA_TELEMETRY=false * Pass `--no-telemetry` flag on kopia invocation * Uncheck a checkbox in the kopia UI. ## Data Storage All collected data will be aggregated (for example on a daily, weekly and monthly basis) and individual ping records will be discarded after no more than 90 days with only aggregated counters stored beyond that.

Comments on Github are preferred, but if you don’t have an account, feel free to post comments in this thread as well.

Topic		Replies	Views
Kopia v0.9.0-rc1 is released Announcements	7	1002	October 1, 2021
Kopia v0.7.0-rc1 Announcements	5	442	September 16, 2020
Curious data and times Support	2	227	April 6, 2022
0.8.0-beta1 is out Announcements	12	619	March 14, 2021
Kopia v0.10.7 has been released Announcements	3	689	April 10, 2022

PROPOSAL: Kopia Telemetry Collection

Related Topics