ssh-add (adding a key to the ssh-agent) can be told to add it with a timer, so that it expires after say, 5 minutes or so. If (as you guess and I think I agree) ssh only needs the key at early connection time, it won’t notice or care that the ssh-agent forgot about the key after 5 minutes and will keep on using the same ssh connection until the backup is done, and then it can’t (ab)use this key again.