I have a “backup server” were I rsync data to. One folder
/syncs/MUSIC
has data which I rsync about 3 days ago and other than that, the files are over 2 years old. When I run snapshot on the folder I am seeing:
130 uploaded (3 GB)
it is still running so may be higher… is that actually sending data? Or just reporting what is “Backed up”, but in reality it is not sending since data is already on repository?
you can see a diff between two snapshots to determine if anything was uploaded.
kopia diff kOLD kNEW
It is showing files “changed”, but nothing has changed in the folder for over 2 years (for the most recent additions). Most files in the folder and probably 5+ years.
Output of diff:
./flac sizes differ: 50006600991 50034838618
./flac/AliceCooper sizes differ: 1247169907 1275407534
./flac/AliceCooper/Trash sizes differ: 275021251 303258878
added file ./flac/AliceCooper/Trash/Trash - 03 - House Of Fire.flac (28237627 bytes)
./flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac modification times differ: 2013-09-08 00:21:27 -0400 EDT 2013-09-08 09:06:54 -0400 EDT
changed ./flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac at 2013-09-08 09:06:54 -0400 EDT (size 28228811 -> 28228811)
added file ./flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac (28228811 bytes)
./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 02 - Tu Reputaci�n.flac modification times differ: 2013-09-08 00:22:16 -0400 EDT 2013-09-08 09:06:55 -0400 EDT
changed ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 02 - Tu Reputaci�n.flac at 2013-09-08 09:06:55 -0400 EDT (size 30325931 -> 30325931)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 02 - Tu Reputaci�n.flac (30325931 bytes)
./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 06 - A�n Te Amo (Carta No. 1).flac modification times differ: 2013-09-08 00:22:46 -0400 EDT 2013-09-08 09:06:56 -0400 EDT
changed ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 06 - A�n Te Amo (Carta No. 1).flac at 2013-09-08 09:06:56 -0400 EDT (size 19536029 -> 19536029)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 06 - A�n Te Amo (Carta No. 1).flac (19536029 bytes)
./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 07 - Abarroter�a De Amor.flac modification times differ: 2013-09-08 00:23:26 -0400 EDT 2013-09-08 09:06:57 -0400 EDT
changed ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 07 - Abarroter�a De Amor.flac at 2013-09-08 09:06:57 -0400 EDT (size 25239467 -> 25239467)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 07 - Abarroter�a De Amor.flac (25239467 bytes)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 09 - Te Acuerdas De M� (Carta No. 2, 20 A�os Despu�s).flac (21412331 bytes)
./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 11 - Me Ense�aste.flac modification times differ: 2013-09-08 00:24:48 -0400 EDT 2013-09-08 09:07:00 -0400 EDT
changed ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 11 - Me Ense�aste.flac at 2013-09-08 09:07:00 -0400 EDT (size 30376018 -> 30376018)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 11 - Me Ense�aste.flac (30376018 bytes)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 13 - T�.flac (28441223 bytes)
added file ./flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - M�xico.flac (29252295 bytes)
./flac/Scorpions/Live Bites/Live Bites - 02 - Is There Anybody There�.flac modification times differ: 2013-09-08 00:27:12 -0400 EDT 2013-09-08 09:07:04 -0400 EDT
changed ./flac/Scorpions/Live Bites/Live Bites - 02 - Is There Anybody There�.flac at 2013-09-08 09:07:04 -0400 EDT (size 32321780 -> 32321780)
added file ./flac/Scorpions/Live Bites/Live Bites - 02 - Is There Anybody There�.flac (32321780 bytes)
./flac/Tracy Chapman/Tracy Chapman/Tracy Chapman - 08 - Why�.flac modification times differ: 2013-09-08 00:27:34 -0400 EDT 2013-09-08 09:07:05 -0400 EDT
changed ./flac/Tracy Chapman/Tracy Chapman/Tracy Chapman - 08 - Why�.flac at 2013-09-08 09:07:05 -0400 EDT (size 14049840 -> 14049840)
added file ./flac/Tracy Chapman/Tracy Chapman/Tracy Chapman - 08 - Why�.flac (14049840 bytes)
Looks like something is indeed modifying those files or could be some local file system corruption ?
From what I can tell seems to be an issue with kopia.
This is what I did:
No idea why, but somehow Kopia is thinking some files have changed, even though they haven’t (at least their MD5s haven’t). For sure their timestamps have not changed either… find -mtime doesn’t show anything having changed in over 2 years
when you mount the snapshots, are the files the same?
Just tried mounting 2 of the snapshots and did md5 on one of the files reported. The md5 on both snapshots matched the md5 of the file in the filesystem.
can you compare two snapshots:
kopia ls -r -l kOLD
kopia ls -r -l kNEW
and paste results for any file that’s different? (just few samples)
Samples from files that keep getting backed up
File 1
root@storage:~#kopia ls -r -l k27cfceac965a1535063aa6f1f8c81ebf |grep Queensry|grep Dama
-rwxr-xr-x 32414654 2012-07-13 22:26:21 EDT Ix33157b573c7ae33d8ee1fd1e7521d985 flac/Queensryche/Promised Land/Promised Land - 03 - Damaged.flac
root@storage:~#kopia ls -r -l k8efb4482ba81d7cd13c24521172d8efa |grep Queensry|grep Dama
-rwxr-xr-x 32414654 2012-07-13 22:26:21 EDT Ix33157b573c7ae33d8ee1fd1e7521d985 flac/Queensryche/Promised Land/Promised Land - 03 - Damaged.flac
File 2
root@storage:~#kopia ls -r -l k27cfceac965a1535063aa6f1f8c81ebf |grep “Land/Promised Land - 11 - Somebody Else”
-rwxr-xr-x 28228810 2012-07-13 22:26:45 EDT Ix7e166e4f6a76c1cd9e196a72d9c03b0d flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else.flac
-rwxr-xr-x 28228811 2013-09-08 09:06:54 EDT Ixb6c7d539253e0e733acd1e299a603a24 flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac
-rwxr-xr-x 28228811 2013-09-08 00:21:27 EDT Ix6410bad2947ff837bcfa5dc479110f18 flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac
root@storage:~#kopia ls -r -l k8efb4482ba81d7cd13c24521172d8efa |grep “Land/Promised Land - 11 - Somebody Else”
-rwxr-xr-x 28228810 2012-07-13 22:26:45 EDT Ix7e166e4f6a76c1cd9e196a72d9c03b0d flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else.flac
-rwxr-xr-x 28228811 2013-09-08 09:06:54 EDT Ix503e33518fe725e101d2396573e377ee flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac
-rwxr-xr-x 28228811 2013-09-08 00:21:27 EDT Ixe5a40f856c6a77600858f4be06a376b7 flac/Queensryche/Promised Land/Promised Land - 11 - Somebody Else�.flac
File 3
root@storage:~#kopia ls -r -l k27cfceac965a1535063aa6f1f8c81ebf |grep “Si El Norte Fuera El Sur - 14”
-rwxr-xr-x 29252294 2012-07-13 22:29:46 EDT Ixced60f0ad59d73cef95d353e9817f7e4 flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - Mexico.flac
-rwxr-xr-x 29252295 2013-09-08 09:07:02 EDT Ix03b96f9ed0a73ee35d2ac0b1f65ab280 flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - M�xico.flac
-rwxr-xr-x 29252295 2013-09-08 00:26:21 EDT Ix48c05e1690d0c682f181d78af5b16892 flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - M�xico.flac
root@storage:~#kopia ls -r -l k8efb4482ba81d7cd13c24521172d8efa |grep “Si El Norte Fuera El Sur - 14”
-rwxr-xr-x 29252294 2012-07-13 22:29:46 EDT Ixced60f0ad59d73cef95d353e9817f7e4 flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - Mexico.flac
-rwxr-xr-x 29252295 2013-09-08 09:07:02 EDT Ix48c05e1690d0c682f181d78af5b16892 flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - M�xico.flac
-rwxr-xr-x 29252295 2013-09-08 00:26:21 EDT Ix03b96f9ed0a73ee35d2ac0b1f65ab280 flac/Ricardo Arjona/Si El Norte Fuera El Sur/Si El Norte Fuera El Sur - 14 - M�xico.flac
I think I found the likely culprit
Si El Norte Fuera El Sur - 14 - Mexico.flac
Si El Norte Fuera El Sur - 14 - Mxico.flac
Si El Norte Fuera El Sur - 14 - M�xico.flac
Seems most of the issues is files with similar names that somehow appear the same to Kopia. Perhaps due to character encoding.
Will clean those up and then try and debug the ones that are single file.
I’m confused. So you have 3 separate files in one directory with almost identical names?
Yes. And most of the errors seem to be related to that.
Btw, try adding --trace-localfs when snapshotting, it may tell you useful info in the logs.
BTW. Can you ping me on Slack? - will be easier to DM instead of posting on the forum.