OBS chroot and Firefox

How to debug Firefox from within OBS chroot

Debugging/developing Firefox in openSUSE/SLE

Most importantly, the wrapper script Firefox in openSUSE/SLE has an extra option -d/--debugger:

$ firefox -d gdb

means run under debugger gdb

Debugging/developing generally anything on Linux for a specific environment

The simplest way of debugging software is:

installing target test system
installing affected packages together with their corresponding debugging info packages (for SUSE typically the -debuginfo and -debugsource packages)
reproducing the problem in question
reproducing the problem in question with a debugger attached
fixing code
building fixed packages
looping back to step 1

This full cycle has rather big disadvantage: speed - or rather slowness, and generally speaking clumsiness. Especially in cases, where a problem only shows up in a specific environment (think distribution code streams).

It entirely removes the possibility of incremental builds (where only changed sources are recompiled/relinked) and introduces additional overhead of the packaging stuff that needs to be performed. On top of that some errors are generally only appearing on some versions of operating environment (read "system") - that would seem to require one to have at disposal all possible systems (and architectures).

To speed up the debugging/development process and to make it generally more flexible it seems convenient to use an already existing infrastructure, namely the build environment. The general workflow then allows making the cycles smaller, especially allows quick looping between points 3. and 4. above:

building package using osc build --root <buildroot> ... on an arbitrary system capable thereof
chrooting to <buildroot>
reproducing the problem in question
reproducing the problem in question with a debugger attached
fixing code in the build directory (typically found in /home/abuild/rpmbuild/BUILD in the chroot environment)
building fixed code - incrementally, since all binaries are still available
looping back to 4

This general algorithm usually requires several tweaks to the buildroot configuration and additional steps that provide continuity between the initial build and the incremental builds that happen from within the buildroot.

Things worth noting (aka "Problems")

There is a couple of things you will find out sooner or later yourself, but it might be faster to read them here to save yourself some head scratching

It doesn't (always) work

In some cases this doesn't entirely work - for example there on SLE-15 @ ppc64le

built binaries under /home/abuild/rpmbuild/BUILD/obj/ do crash in chroot
installed packages under /home/abuild/rpmbuild/RPMS/ do result in crashes in chroot as well
exactly those packages installed into the host system work

... ghost in the machine

Running software as root...

... is generally not a brilliant idea. Even when running chrooted, there are still many ways to compromise/kill the host system - after all, you are talking to the same kernel and unless you took extra care to harden the chroot environment (so that it actually is not a chroot any more but more something of a FreeBSD jail - on Linux available e.g. through firejail or some containering infrastructures, that may or may not be an overkill) running with UID 0 is exactly the same as if performed on the host system - potentially deadly.

Therefore it is a Good Idea™ to create a separate user account in the chroot used only for test running the software. Whether you decide to build the software as root or use any other account is not as important, although it may be worthwhile to build it under the same account as the original build process (on SUSE this would be abuild since at least SLE-12).

Completing the chroot environment

While the chroot environment (as set up by e.g. osc build) is complete, at least when it comes to building the software (and that usually means also for running it) there are several pieces missing.

Pseudo filesystems

Lots of software requires access to kernel information, most importantly for introspection through /proc/<getpid()>/.
Some programs use special devices directly, and thus expect /dev to contain the appropriate nodes.
Some programs require access to pseudoterminal devices in /dev/pts
Having sysfs mounted may be a reasonable as well
Some programs use shared memory. Those relying on the POSIX API (shm_open/shm_unlink) will most likely assume tmpfs is mounted on /dev/shm. Not having the mount is not necessarily a problem, however tmpfs has the unique treat that it is mounted writable for all by default. This fact may cause a rather unexpected behaviour - if you don't mount tmpfs on /dev/shm (in the chroot) and neither change the permissions for the directory, programs using POSIX shared memory API will work when running with UID 0 but will behave strangely otherwise. For example newer (at least 60+, but probably some older as well) Firefox will not render anything into its window, since the data is coming via shared memory from a separate process.
some programs will expect /run to behave in a similar way (typically various daemons us it as temporary storage)

Solution

Bind-mount (mount --bind) some of the above from the host system, separately mount the rest. Reasonable configuration seems to be:

# running in the host prior to chrooting
$ for n in dev dev/pts proc sys; do mount --bind /$n $CHROOT/$n; done
$ mount -t tmpfs tmpfs $CHROOT/dev/shm -o size=16M
$ mount -t tmpfs tmpfs $CHROOT/run -o size=16M

Mounting /dev/shm and /run separately isolates the chroot environment from the host system a bit. While the limits on tmpfs size may seem small, remember that in the chroot you should only need it for small number of programs/daemons. As a side effect, should there be any bug causing mad overallocating space in tmpfs, you'll hit it much faster.

Tools

Debugging usually can't happen without useful tools. While due to the nature of construction of the buildroot it contains the toolchain and all necessary header files, it is missing indispensable things like debugger or an editor (not to mention version control systems).

Solution

Use the command line option -x for osc or configuration file (~/config/osc/oscrc) option extra_pkgs to pull in anything you may need into the buildroot. Compared to installing extra packages in the buildroot by chrooting into it this has the advantage of automatically pulling in any dependencies. Plus on a re-run the packages will be installed again.

The real solution

Get the Devel:Desktop:Mozilla/buildroot-debug package (and its subpackages). It pulls in tools, their configuration and a script that can do the necessary bind-mounting (if you set up sudo properly it will work even for unprivileged user).

/dev/pts problems

Sometimes it happens that logging into the host system via ssh results in PTY allocation request failed ... error message being printed and that being seemingly it. Very likely this means two things:

devpts filesystem normally mounted on /dev/pts is not accessible
You are logged into the system, yet without the terminal emulator layer - you have a direct, untranslated connection to the remote shell's standard input/output/error. It should be easily fixable by a single command:
mount -t devpts devpts /dev/pts issued as root.

If you are switching from an unprivileged account using e.g. su, don't be surprised that the password you enter is echoed - remember, there is no terminal emulator, that could be set to not echo the characters received. This is likely also the reason for not seeing the prompt of your shell.

If you have troubles logging into the machine while getting this message, make sure your ssh is not explicitly requesting TTY allocation, since if that fails the connection is dropped.

Using tmpfs for building

tmpfs has a granularity of PAGE_SIZE - on many architectures this is 4KiB, but on some it is 64KiB - don't be surprised by that. This is especially important if you are on a machine with lots of RAM and want to use it by having the buildroot in tmpfs. The best option is to create a backing file in tmpfs holding a filesystem that is subsequently mounted somewhere and used as buildroot. Using XFS may help since it can use sector sizes under PAGE_SIZE (down to 512B), saving noticeable amount of space on small files, and can have multiple inode groups, thus speeding up build process which writes many files in parallel.

$ dd if=/dev/zero of=/dev/shm/FF.xfs
$ /sbin/mkfs.xfs -d agcount=64 -b size=1k /dev/shm/FF.xfs
$ mkdir /dev/shm/FF
$ mount /dev/shm/FF.xfs /dev/shm/FF