State of repository after loss of connection while doing a backup

[felixschndr]

Hello everyone,

I tried to create a backup with borg via SSH to a remote server. While creating the backup the internet connection went down and the backup was not able to finish successfully

borg.remote.ConnectionClosed: Connection closed by remote host

and

BrokenPipeError: [Errno 32] Broken pipe

Now I am wondering about the state of the backup: After the failed finish the directory of the repo took about 11G of disk space. borg list did/does not show any backups (empty output) (Note: this was the first backup to this repo). Since the backup did not finish successfully I assumed that borg compact would delete all files and the repo size would go down to some bytes (for the key and metadata).
However, after borg compact the directory still takes about 9G of disk space. borg info shows the same status:

┌─[16:57:12]-[:)]-[root💀grievous]-[/home/felix/server/Backup/ (main)]
└──> BORG_PASSPHRASE=${BORG_REPOSITORY_KEY} borg info ${BORG_REPOSITORY}
Repository ID: ...
Location: ssh://felix@qnap/share/homes/felix/Backupv2
Encrypted: Yes (repokey)
Cache: /root/.cache/borg/...
Security dir: /root/.config/borg/security/...
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                9.51 GB              9.51 GB              9.58 GB

                       Unique chunks         Total chunks
Chunk index:                   14405                15842

borg delete shows that the repo contains one archive.

┌─[16:58:09]-[:(]-[root💀grievous]-[/home/felix/server/Backup/ (main)]
└──> BORG_PASSPHRASE=${BORG_REPOSITORY_KEY} borg delete ${BORG_REPOSITORY}
You requested to completely DELETE the repository *including* 1 archives it contains.
Type 'YES' if you understand this and want to continue:

So now I am wondering what the status of my repository is. Obviously the backup was unsuccessful, but why does the repo still take up 9G of space? I wanted to delete the unsuccessful backup to not have to init a new repo.

[ThomasWaldmann]

Try: borg list --consider-checkpoints REPO

borg creates valid, but incomplete checkpoint archives every checkpoint interval (1800s) to commit the progress while doing a longer backup. So if the connection breaks down or the backup can't complete due to other reasons, you still won't lose all progress.

Just start the backup again and it will very quickly process the data up to the checkpoint (because all the chunks are already in the repo). All remaining data after that will then be processed at normal speed.

Usually one doesn't delete unsuccessful (or incomplete .checkpoint) archives until after one has made a successful one. Even then manually deleting is not necessary because prune will get rid of .checkpoint archives also IF there is a more recent complete archive with the same basename.

Don't forget to run borg compact periodically. Only that will free repo space for borg >= 1.2.

[felixschndr]

So if the connection breaks down or the backup can't complete due to other reasons, you still won't lose all progress.

Just start the backup again and it will very quickly process the data up to the checkpoint (because all the chunks are already in the repo). All remaining data after that will then be processed at normal speed.

This is so cool! I love it! Until I started to use borg (some days ago) I only used tar and had the problem that I had to restart the backup from scratch after a disconnect which happened quite frequently with a backup of about 120G.

When starting the backup again shall I create a new archive or can/shall I append to the existing one? Is there a best practice?

---

Is there a way to tell borg to wait after a disconnect until the connection is back up? I read something in the borg serve docs but am unsure how this works and what the exact implications are. I'd like something like if there is a disconnect (due to network outage) wait for x minutes. If the connection comes back up continue. If the connections is still down after x minutes error out.

---

Don't forget to run borg compact periodically. Only that will free repo space for borg >= 1.2.

Yes, my script does that 👍🏻

---

Thank you for your time and effort, I really appreciate it =)

[ThomasWaldmann]

You can not append to archives. But due to dedup, you can just create a new archive without losing the progress made by the previous try.