Question on Symlinks

[henfri]

Hello,

I know that borg does not resolve symlinks.
But what about this:

/srv/Daten -> /srv/dev-disk-by-label-Daten
borg create /path/to/repo::my-documents /srv/Daten/myfolder

Will myfolder be backed up? I think it should, but I think it does not...

Best regards,
Hendrik

[ThomasWaldmann]

Yes, because myfolder is a starting point (recursion root) and (if myfolder is a normal folder), it will be backed up normally and also everything below it.

[henfri]

Thanks for the quick reply Thomas.

Apparently, it does not, for me. I run this command to backup:
borg create --exclude-from /etc/borgmatic/excludes --exclude-from /tmp/tmpx5khz_uh --exclude-caches --exclude-if-present .nobackup --compression zstd --one-file-system --remote-path borg --umask 63 --debug --show-rc /media/22cd0c22-c75b-432a-b92b-73229607a8bd/Backups/Borg/borgbackup/::{hostname}-documents-{now} / /srv/DataPool1/Dokumente /srv/DataPool1/Fotos /srv/DataPool1/HomeVideo /srv/DataPool1/Musik /srv/DataPool1/Paperless /srv/Daten/dockerconfig

Now, in my backup, I have a Symlink "Daten".
But I do not find the folder "dockerconfig" in it (I am inspecting the backup with borg mount)
Where would I find "dockerconfig", when browsing borg mount? I mean... my::repo/srv/Daten is a symlink, so my::repo/srv/Daten/dockerconfig cannot really exist

Of course, I have used --one-file-system, which I did not write in my original post.... Does that make a difference?

I tried extract with success. So it seems to be an issue with borg mount...

Best regards,
Hendrik

[ThomasWaldmann]

The problem with borg mount is that, although you have backed up dockerconfig(+contents), you did NOT back up the fs object "Daten".

And even if you did, I guess you'ld also need to backup the real path where dockerconfig is - to give the Daten symlink the correct target inside the archive in that case. borg mount must be able to construct a valid filesystem from whatever it has in the archive.

[ThomasWaldmann]

See #5282.

[ThomasWaldmann]

In general, if you think there is a problem that could be easily fixed, write a simple, re-usable reproducer script.

simple means:

• no absolute pathes, so it runs everywhere
• don't use "root" user
• not a lot of options that are not needed to reproduce the issue - just use the bare minimum to show / reproduce
• create what you need / remove it afterwards, so the script can run again and again

[ThomasWaldmann]

I find it strange that you say you have a symlink "Daten" in your FUSE fs. Considering you did not back that up, I am wondering where that should come from.

[henfri]

Hello,

I am also not sure why there is that symlink "Daten". I will have to investigate a bit more.
Anyway, as you suggested, here's an example, that makes the issue reproducible:

mkdir -p testdata/borg-bug/backup/folder
touch testdata/borg-bug/backup/folder/valid
ln -s testdata/borg-bug test

borg init -e none repo
borg create -v --filter AME --list --stats --one-file-system repo::backup test/backup/ ./test

echo "----------- borg list ------"
borg list repo::backup

echo "----------- Now via borg mount -----------"
mkdir -p mnt/test
borg mount repo::backup mnt/test
find mnt/test
umount mnt/test
# here, the file "missing" is missing
rm -r repo testdata mnt
rm test

the output here:

bash ./borg_test.sh
Creating archive at "repo::backup"
A test/backup/folder/valid
------------------------------------------------------------------------------
Archive name: backup
Archive fingerprint: cebcc4c86b58eadd30eae936883b1415801fbf6b5a2f1520377ea6bd67c2cfe3
Time (start): Fri, 2022-02-11 20:41:31
Time (end):   Fri, 2022-02-11 20:41:31
Duration: 0.00 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  963 B                717 B                717 B
All archives:                  963 B                717 B                717 B

                       Unique chunks         Total chunks
Chunk index:                       2                    2
------------------------------------------------------------------------------
----------- borg list ------
drwxr-xr-x root   root          0 Fri, 2022-02-11 20:41:30 test/backup
drwxr-xr-x root   root          0 Fri, 2022-02-11 20:41:30 test/backup/folder
-rw-r--r-- root   root          0 Fri, 2022-02-11 20:41:30 test/backup/folder/valid
lrwxrwxrwx root   root         17 Fri, 2022-02-11 20:41:30 test -> testdata/borg-bug
----------- Now via borg mount -----------
mnt/test
mnt/test/test

borg version is:

borg --version
borg 1.1.11

Best regards,
Hendrik

[ThomasWaldmann]

remove all options and stuff that are not needed for reproducing. also do not name the mountpoint same as the directory.

[henfri]

Hello,

here we go:

mkdir -p testdata/borg-bug/backup/folder
touch testdata/borg-bug/backup/folder/valid
ln -s testdata/borg-bug test

borg init -e none repo
borg create -v repo::backup test/backup/ ./test

echo "----------- borg list ------"
borg list repo::backup
# here, all files SEEM to be there.

echo "----------- Now via borg mount -----------"
# when the backup is mounted using borg mount, your unable to access the data stored in other/folder
mkdir -p mnt
borg mount repo::backup mnt
find mnt/
find mnt/ -exec ls -l {} \;
umount mnt
# here, the file "missing" is missing
rm -r repo testdata mnt
rm test

Best regards,
Hendrik

[ThomasWaldmann]

can you please remove all stuff that is not needed to reproduce?

[henfri]

Hello,

I think I did. What do you think may still be not needed?

Best regards,
Hendrik

[ThomasWaldmann]

You commented some code. Some other comments still refer to stuff in the code that is not executed any more.

Hmm, I see comments are gone now. Maybe a caching issue or you removed them while I read it.

[ThomasWaldmann]

also, is that --one-file-system required to reproduce?

[ThomasWaldmann]

# when the backup is mounted using borg mount, your unable to access the data stored in other/folder

^ there is no other/folder?

[ThomasWaldmann]

# here, the file "missing" is missing

^ there is no "missing" file created by that code.

[jdchristensen]

It works fine if you don't explicitly back up the symlink "./test". Since you back it up, but it doesn't point to a valid filesystem object, I'm not sure where you expect the file test/backup/folder/valid to appear in the mounted filesystem...

[ThomasWaldmann]

I don't think @henfri backs up the testdata fs object.

Neither the borg-bug fs object.

[ThomasWaldmann]

So, this is somehow similar to the issue I linked above.

Except that we maybe can get a better/simpler reproducer script here, esp. if we get rid of unneeded stuff.

[henfri]

Hello,

indeed I had removed some comment in the test-script above. But it was just comments.
I have now edited the script again and remove the --one-filesystem parameter. Still the same issue.

@jdchristensen The files that I am missing in the fuse-mount are:

drwxr-xr-x root   root          0 Fri, 2022-02-11 22:31:09 test/backup/folder
-rw-r--r-- root   root          0 Fri, 2022-02-11 22:31:09 test/backup/folder/valid

And I think the reason is rather clear: in the mount, there is the symlink "test" that points to "nowhere". There cannot be at the same time the folder "test" containing the subfolders that I have backed up.

In fact, I see another reason why borg mount should hide dead symlinks:
When I originally noticed this issue, the symlink was NOT dead, but pointed to a place outside of that backup:

/mnt/Drive --> /mnt/drive-dev-by-id-2394p234234

And in fact I was not even aware anymore that /mnt/Drive was a symlink.
When I browsed the backup I was shocked that in the Backup, the same files that I had deleted were also missing - but in fact /mnt/borg-fuse/Drive was pointing to /mnt/drive-dev-by-id-2394p234234 - my source drive...

Best regards,
Hendrik

[ThomasWaldmann]

still seeing outdated comments in the script...

also, if you want to tell that there is an issue with symlinks "pointing outside" the mount, show it in your script.

i very much appreciate correct comments and code showing issues in the script rather than "discussion" text here.

in the end, we could just create a github issue with that script and that would be a good starting point for someone who might want to fix this.

[henfri]

Hello,

sorry, I did not see that.
I have created an issue now:
#6287

Best regards,
Hendrik