I think the best explanation is, that Kopia is file-based backup tool. Thus, it doesn’t perfom system backups in a oob usable manner. You can surely backup the files from your system, but you won’t be able to do any systethe m image backup that way.
I guess borg comes closest usage-wise. dd and rsync are both basic tools that happen to ship with every Linux/Unix install I know… rsync performs file backups, while dd performs device backups, where “device” covers everything than can represent a device from the system’s view (files or real devices).
Kopia does a great - if not the best, job in file level backups. I am using it for backing up my personal files, as well as our big 100T file server and it does all that extremely efficient. However, Kopia can become somewhat involved, if you’re new to it and even more, if you don’t have a good understanding of its underlying systems. Be prepared to start over multiple times, before getting the results you want to achieve. But… it’s absolutely worth it!
Would be safe to say then that Kopia would work flawless on backing my /home or any folder that I set up replacing famous tools such as Borg, etc.?
But if I format my ssd I would have to install the system first and then recover the backup file created by Kopia?
Can I set up a daily backup to my Google Drive or just Google Cloud?
Per my understanding, Kopia works with incremental backup, correct?
If yes, would the backup be encrypted at Google level? i.e. Google or anyone else who could hack my Google account wouldn’t be able to access my backup data, at least directly.
Can I upload all my files that are under different partitions? My current set up has 3 OS (Pop + Fedora 33 + W10). I would use Kopia on my Fedora 33 which has access to all other partitions. File systems are BRTFS for Linux and NTFS for Windows.
It does for me. I have never used Borg, but restic - which has a similar approach like Kopia, I deem it stable enough to entrust it with a lot of valuable data already.
I don’t know, what you’re getting at here. You can actually install Kopia anywhere you like and get to your files, given that you can access the actual repository, where you had Kopia store the data. All you will need is the repository password and - if needed, the username, which performed the snapshots
Absolutely, however the actual way of achieving that will vary depending on your Kopia installation.
Yes. Kopia performs incremental backups/snapshots and performs dedupe in one go.
Kopia always encrypts data on its own, so it’s safe to use with any storage, where it can write to - no need to worry about any one else’s encryption at rest. If anyone should get hold of your saved blobs, they couldn’t do a thing with the data.
Yes - thats no issue. You can use the same repo for different “Users”. You can also perform multiple snapshots (folders) from the same user of yourse.
What I meant is Kopia won’t backup an image of my system which would include boot system, etc. Hence, as you mentioned that it’s a file based backup, in case a repair is needed I would first reinstall my OS, then I would recover my backup from my repository.
2.1) I understand that once recovering it would overwrite existing files and folders, right? In other words, fresh install, fresh /home, then recover has a /home, extract from backup file and replace/home.
My question was not clear, apologies. Does it connect to Google drive or only to Google cloud? Would you know? I need an online option, but it seems all of them are available only for companies…
I think it is important to clarify this. In my understanding, every backup (snapshot) is a full backup in the sense that it is a complete file catalogue and requires no other backup histories to restore. The deduplication feature means only one file fragment needs to be stored, thus saving space. The actual backup process only needs to copy the files/chunks that have not previously been backed up so in that sense it is incremental. The single snapshot is however a representation of the full selected file system.
I use Kopia for backup of multiple PCs over a network, one of which runs a Windows partition (not my choice). I run kopia on each Linux PC to backup my user files only, which makes copies to a local repo and to a central PC repo. To restore Linux I use just a fresh install disk but keep a record of packages I have installed.
The windows partitions I backup using ‘dd’ so I can get a system image (windows partitions are not as easy to restore from an install CD). Using dd I run it prior to running kopia. As windows has 3 partitions I use a bash fn which looks like (take this with a grain of salt as I’m not an authority on the best way to do this):
dd_backup()
{
dd if=/dev/sda1 conv=sync,noerror status=progress bs=512 | sudo gzip -c > "${1}/win_os_ddbackup_sda1_${DATE}.img.gz"
dd if=/dev/sda2 conv=sync,noerror status=progress bs=512 | sudo gzip -c > "${1}/win_os_ddbackup_sda2_${DATE}.img.gz"
dd if=/dev/sda3 conv=sync,noerror status=progress bs=64k | sudo gzip -c > "${1}/win_os_ddbackup_sda3_${DATE}.img.gz"
# Use gz instead as tar doesn't work with a pipe e.g.
# dd if=/dev/sda conv=sync,noerror status=progress bs=131072 | sudo tar -cnv --zstd -f "$1"/win_os_ddbackup"$DATE".img.tar.zst
# dd if=/dev/sda conv=sync,noerror status=progress bs=131072 | sudo tar -cnv --lzip -f "$1"/win_os_ddbackup"$DATE".img.tar.lz
# Save extra information about the drive geometry necessary in order to interpret
# the partition table stored within the image. The most important of which is the cylinder size.
fdisk -l /dev/sda1 > "$1"/"${HOSTNAME}"_fdisk_win_os_sda1_"${DATE}".info
fdisk -l /dev/sda2 > "$1"/"${HOSTNAME}"_fdisk_win_os_sda2_"${DATE}".info
fdisk -l /dev/sda3 > "$1"/"${HOSTNAME}"_fdisk_win_os_sda3_"${DATE}".info
}
where arg $1 is the target backup path location.
I tried restic (mmm) and borg (python yuk), but I much prefer kopia and had even less issues, even though I’ve had to write long scripts to automate my process with policy settings, and snapshots.
Well… first you state, that each snapshot is a full backup and then you state, that its actually an incremental one…even with a nice explanation, as of why it is incremental. Incremental means, that the source files get scanned for changes and files which have been changed or have not yet been saved, get saved/backed up. A full backup will always read/process any file present, regardless of it’s state. And this is exactly how Kopia operates.
How do you keep a record of linux packages installed? You probably have ended creating a script to install all of them at once, right?
Out of curiosity, and if that’s the case, why don’t you simple clone the whole disk to smooth the installation process instead of saving only the /home? The same you do with dd for W10.
For the windows partition, you cloned each partition but DD is not incremental, is it? You basically create a new image every time you back it up, potentially increasing the size of your backup disk.
Have you tried using Kopia for these mounted partitions on linux?
Arch linux has a package manager, “pacman” that has commands that will list various types of packages installed from its repos being the "AUR " or the core, community or extra libraries. By running the pacman command I get a list that I can save to file. I also list all my configuration files. This file can then be used to reinstall, provided you have an internet connection. I create a local repo for the AUR packages.
I have gone one step further and created an offline USB installer (using “archiso” that will bundle all these packages and reinstall from scratch.
I haven’t used dd for a linux system because from what I’ve read no one does, as you end up saving a whole bunch of temp data that is created live when you boot up. In any case the install files are readily available anyway. Also a new install is easier particularly if you have varying partition sizes between the old and the new HDD. So really all you need are the user files and configuration files to get going again on a new system. Admittably there will be some inconvienience unless you can streamline/speed up the process by using RAID or some other method (I mentioned mine above using a custom USB).
Also to use dd on a system you ideally should not be using it at the time of imaging. This is usually not convienient.
It is a complex project but in reality you should be able to do a new installl every 6 months just to prove you can do it, as data loss, hdd failures are inevitable so it should be part of a keen users system management process, in my opinion. Also the tools avialble to do this all well require somewhat of a learning curve to get there.
Yes dd is not incremental, but I don’t plan to keep too many of them to save disk space. I only use it once every few months or so as I don’t see the windows box as a critical system.
If you mean backup of windows files to a linux box, No I haven’t. Not sure how you would do that and maintain file integity. I’ve alway used imaging for system partitions, except for the linux method I described above.
From my understanding it sometimes means a user can only restore if they have all the incremental backups reight back to the last full backup. Kopia doesn’t require this as I see it.
Wow… hold on a sec… this refers to the times, when we were using tapes as storage media. Ever since VTLs got fancy or one found another way to ditch those tapes, this hasn’t been the case.
However, this is more of an academic discussion anyway, since most users of Kopia will not care about the knitty-gritty details. But as soon, as you start to use Kopia seriously, like backing up a 100TB server with 350+ shares, where each share gets its own snapshot, you’d need to get into the weeds…
Nice to see the power users making use of kopia in a meaningful way as it provides more confidence to the rest of us that it is a worthy tool even in its early stages. Yes some of us don’t fully understand it’s inner workings, but what I have seen it is quite elegant.
Well… yes. And given that the first “full” snapshots took approx 7 days to complete, performing the “incremental” ones in roughly 3 hrs, really can only be achieved, if you don’t have to rescan each and every file…
If you’ve toyed with it since then, you probably already know this by now, but there’s no forced-overwriting when restoring files, as you decide where you restore to.
As Buddy said,
Therefore, I suggest to start small, backing up a limited amount of data at first, and make use of estimate to figure out ignore rules.
rsync performs file backups, while dd performs device backups, where “device” covers everything than can represent a device from the system’s view (files or real devices).
this refers to the times, when we were using tapes as storage media. Ever since VTLs got fancy or one found another way to ditch those tapes, this hasn’t been the case.