ubuntu noble (pre-24.04) experience

[eichin]

I've been working on a pre-24.04 system (2024-04-16 install about a week ago, but packages updated to this morning) and as part of looking at the visibility/display changes and getting oriented, I figured I'd walk through https://toga.readthedocs.io/en/stable/how-to/contribute-code.html#how-the-testbed-works on main and make sure that worked. (tl;dr: got down to 12 failed, 389 passed with some tweaking, but some of the problems aren't toga.)

webkit

gir1.2-webkit2-4.0 isn't available anymore, instead install gir1.2-webkit2-4.1 and update gtk/src/toga_gtk/libs/gtk.py to say i.require_version("WebKit2", "4.1") (otherwise you get RuntimeError: Unable to import WebKit2. Ensure that the system package providing Webkit2 and its GTK bindings have been installed. from the webview and mapview tests.) Probably should be a fallback or fallforward.

cairo

I don't have a test failure to match this one, but while I was stepping through gtk.py above, I caught

>>> import cairo
>>> gi.require_foreign("cairo")
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gi/__init__.py", line 172, in require_foreign
    _gi.require_foreign(namespace, symbol)
ModuleNotFoundError: No module named 'gi._gi_cairo'

Fixed this by installing the python3-gi-cairo package.

pre-commit

The walkthrough talks about pre-commit install and then changing a file; you might want to encourage starting from an unchanged state first, which means running pre-commit run --all-files. (Got 11 ...Passed out of that.)

tox

Worked fine :-)

===================================================== 
1640 passed in 9.32s 
======================================================
  py: OK (30.60=setup[0.09]+cmd[20.67,9.83] seconds)
  congratulations :) (30.68 seconds)

ast in pytest

briefcase dev --test pops up the window and blows up immediately, treating deprecation warnings as errors, with 35 instances of

  File "/home/eichin/tmp/clean-22.04/github-beeware-toga-main/venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py", line 940, in visit_Name
    inlocs = ast.Compare(ast.Str(name.id), [ast.In()], [locs])
                         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ast.py", line 584, in _new
    warnings._deprecated(
  File "/usr/lib/python3.12/warnings.py", line 530, in _deprecated
    warn(msg, DeprecationWarning, stacklevel=3)
DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead

with

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 35 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================== 35 errors in 0.94s =======================================================

To get past that, and the other DeprecationWarnings:

(venv) eichin@workbench:~/tmp/clean-22.04/github-beeware-toga-main/testbed$ sed -i -e 's/ast.Str/ast.Constant/g' ../venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py
(venv) eichin@workbench:~/tmp/clean-22.04/github-beeware-toga-main/testbed$ sed -i -e 's/ast.NameConstant/ast.Constant/g' ../venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py
(venv) eichin@workbench:~/tmp/clean-22.04/github-beeware-toga-main/testbed$ sed -i -e 's/ast.Num/ast.Constant/g' ../venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py

Not sure what to do with that from within toga (or briefcase) though.

actual toga failures

What's left is:

==================================================== short test summary info ====================================================
FAILED tests/test_window.py::test_secondary_window[second_window_kwargs0] - assert (1436, 2496) == (640, 480)
FAILED tests/test_window.py::test_secondary_window_with_args[second_window_kwargs0] - assert (1436, 2496) == (300, 200)
FAILED tests/test_window.py::test_visibility[second_window_kwargs0] - assert (1436, 2496) == (640, 480)
FAILED tests/test_window.py::test_move_and_resize[second_window_kwargs0] - assert (0, 30) == (150, 50)
FAILED tests/test_window.py::test_screen[second_window_kwargs0] - assert (0, 30) != (0, 30)
FAILED tests/widgets/test_button.py::test_text - assert 43 == 41
FAILED tests/widgets/test_canvas.py::test_write_text - Failed: Rendered image doesn't match reference (RMSE==0.5091498824952636)
FAILED tests/widgets/test_canvas.py::test_multiline_text - Failed: Rendered image doesn't match reference (RMSE==0.61239459779...
FAILED tests/widgets/test_scrollcontainer.py::test_vertical_scroll - assert 3543 == (2457 * 3)
FAILED tests/widgets/test_scrollcontainer.py::test_scroll_both - assert 604 == 1000
FAILED tests/widgets/test_switch.py::test_text - assert 33 == 31
FAILED tests/widgets/test_table.py::test_scroll - assert 1275 > (2457 * 2)
=============================== 12 failed, 389 passed, 57 skipped, 5 xfailed in 70.16s (0:01:10) ================================

[testbed] Test suite failed!

The assert (1436, 2496) == (640, 480) looks suspiciously like it's not handling my X environment: Xorg and i3-wm, tallscreen 1440x2560 (with a status-bar/tray so default windows are Width: 1436 and Height: 2496.) Not sure that explains the others - are there perhaps font dependencies that aren't called out?

[rmartin16]

Thanks for writing this up @eichin! I've been looking at Ubuntu Noble as well.

WebKit2 4.1

• I opened Use GTK WebKit2 4.1; else fallback to 4.0 #2529 for Toga to support actually using 4.1

cairo

• You should only need libcairo2-dev to install pycairo with pip
• The pip install process includes everything else that python3-gi-cairo brings along

ast in pytest

• The version of pytest in testbed/pyproject.toml isn't compatible with Python 3.12 given how old it is
• @freakboy3742, should testbed be added to dependabot.yml?

actual toga failures

• I, too, have failures when testing the testbed in my desktop Linux environment....but I haven't taken the time to understand yet

[freakboy3742]

Another vote of thanks for this writeup, @eichin.

The one overarching caveat that I'll put here is that Linux is, by far, the hardest platform for us to test GUI behavior, because pretty much by definition, everything is up for interpretation by the desktop environment. We would definitely like the test suite to pass without issue on every Linux, on every desktop configuration... but that's just not possible.

So - the only guarantee we really make at present is that the tests will pass on the configuration that is used in CI:

• Ubuntu 22.04
• Python 3.10
• A single XVFB screen measuring 2048x1536x24
• The Blackbox window manager. This is the most bare-bones window manager we could find that still behaves like a "real world" window manager - a lot of "minimalistic" window managers have some unusual restrictions on window sizing, min/max/fullscreen behavior etc.

The "official" way to run the testbed on Linux is:

$ Xvfb :99 -screen 0 2048x1536x24 &
$ DISPLAY=:99 blackbox &
$ DISPLAY=:99 briefcase run --test

This - or at least a note that Linux testbed testing has some limitations - should probably be mentioned in the contribution guide, as it's definitely one of the more esoteric parts of the testing process.

There are similar restrictions for other platforms - for example, iOS testing must be run on an iPhone SE 3rd Edition; Android must be on a phone with a Pixel 3a skin; but there are a lot more more moving parts on Linux. That said - we definitely aspire to the testbed suite passing everywhere, and we'd accept patches to make that the case.

• The version of pytest in testbed/pyproject.toml isn't compatible with Python 3.12 given how old it is
• @freakboy3742, should testbed be added to dependabot.yml?

Ideally, yes. However, there's at least 2 problems at present:

The first is that pytest-asyncio can't be updated - there's a subtle change in how async fixtures are used that prevents us from upgrading. This is a straightforward "we need to fix it" problem - see #2382.

The second is that upgrading any module with binary components (most notably, Pillow), requires that we also publish a binary wheel for iOS and Android. Since that's not happening automatically at present, it's a bit difficult to automate the upgrade. Fixing this one likely requires the cooperation of the rest of the ecosystem (or, at least Pillow). Getting the pieces in place for the ecosystem to be able to publish iOS/Android binary wheels is something Malcolm and I are working on for the rest of this year; Pillow will be an obvious candidate to target as a widely-used binary package to use as a "proof of concept" for this work. I'm planning to speak to Hugo about this at PyCon.

actual toga failures

• I, too, have failures when testing the testbed in my desktop Linux environment....but I haven't taken the time to understand yet

Based on the test report, most of those look like they're due to Xvfb or blackbox discrepancies (changing the size/position of windows, or windowing behavior). Patches welcome to modify any or all of these test conditions - even if it's in the form of test skips if the desktop environment doesn't match the expected values.

[eichin]

Thanks for the background (and looking for blackbox finds me .github/workflows/ci.yml where I see similar details for other platforms.) That gets me down to only one failure tests/widgets/test_canvas.py::test_multiline_text FAILED [37%]:

=========================================================== FAILURES ============================================================
______________________________________________________ test_multiline_text ______________________________________________________
Traceback (most recent call last):
  File "/home/eichin/tmp/clean-22.04/github-beeware-toga-main/testbed/tests/widgets/test_canvas.py", line 758, in test_multiline_text
    assert_reference(probe, "multiline_text", threshold=0.09)
  File "/home/eichin/tmp/clean-22.04/github-beeware-toga-main/testbed/tests/widgets/test_canvas.py", line 277, in assert_reference
    pytest.fail(f"Rendered image doesn't match reference (RMSE=={rmse})")
  File "/home/eichin/tmp/clean-22.04/github-beeware-toga-main/venv/lib/python3.12/site-packages/_pytest/outcomes.py", line 194, in fail
    raise Failed(msg=reason, pytrace=pytrace)
Failed: Rendered image doesn't match reference (RMSE==0.416446126178656)
----------------------------------------------------- Captured stdout call ------------------------------------------------------
Saving to /home/eichin/.local/share/testbed/canvas
==================================================== short test summary info ====================================================
FAILED tests/widgets/test_canvas.py::test_multiline_text - Failed: Rendered image doesn't match reference (RMSE==0.41644612617...
================================ 1 failed, 400 passed, 57 skipped, 5 xfailed in 69.38s (0:01:09) ================================

which is almost all the way there. Poking around a bit, if SYSTEM is system-ui, /etc/fonts/conf.d/60-latin.conf has

	<alias>
		<family>system-ui</family>
		<prefer>
			<family>Cantarell</family>
			<family>Noto Sans UI</family>
			<family>Segoe UI</family>
			<family>Segoe UI Historic</family>
			<family>Segoe UI Symbol</family>
		</prefer>
	</alias>

Installing font-cantarell, fonts-noto-extra and fonts-noto-ui-extra didn't change the result. ( test_write_text uses specific ttf files which look different from the system fonts on all platforms.; test_multiline_text uses Font(SYSTEM, size) for a range of size values.)

Looking at the outputs visually and with uprightdiff, it appears to be the same font - the differences in the test output are mostly

• more leading/inter-line spacing
• the reference has Tw and To ligatures (the w and o are "under" the bar of the T) while the test output has them as apparently-separate characters.

uprightdiff output below - the green is the src/testbed/resources/canvas/multiline_text-gtk.png reference, the red is the test run output. (edit: got the colors reversed the first time)

[freakboy3742]

Yeah - that test is one that causes headaches everywhere. Any changes in font definition causes the test to fail, because it's essentially doing an image diff like the one you've provided, and measuring the RMSE between the two images. We have similar problems between macOS upgrades if they make a minor tweak to font definitions.

I can see two possible approaches here:

1. Mark the test as a conditional XFAIL. Allow the test to fail if the runtime environment unless we can confirm we're running in CI. I'm guessing we'd need to set an environment variable in the CI environment, then have something in the probe to determine whether an XFAIL is allowed based on "known good but flaky" configurations.
2. Modify the test data to include samples from other configs - i.e., ship ubuntu22.04, ubuntu24.04 etc versions of the test image. We already have OS-specific variants, so this shouldn't be overly onerous; the catch is that we could potentially end up with a very long list of reference images. However, we could keep the list reasonably short if we're pragmatic (even keeping it to "just Ubuntu LTS releases" would be a good starting point).

I guess the implicit 3rd option is "find a different way to assert test compliance"... but I don't have any good ideas for what that approach would be.

At the very least, we should add the list of required/expected fonts to the description of test setup (along with the blackbox setup).