Yet another missing-logs-and-events flake: journald?

[edsantiago]

I've lost track of how many bugs I've opened for something-or-other like this. I'm going to lump together here the category of flakes seen in late 2024 where podman-remote logs is supposed to see something but doesn't.

Likely fix: change the tests so instead of podman wait; podman logs they do for 5 retries { podman logs; grep for what we want; retry if not there.

• fedora-40 : int remote fedora-40 root host sqlite [remote]
  • PR libpod: hasCurrentUserMapped checks for gid too #24167
    • 10-04 10:49 in Podman kube play should be able to run image where workdir is a symlink
• fedora-40-aarch64 : sys remote fedora-40-aarch64 root host sqlite [remote]
  • PR vendor: update c/common and buildah #24135
    • 10-02 13:03 in [sys] |200| podman pod create - hashtag AllTheOptions

x	x	x	x	x	x
sys(1)	remote(2)	fedora-40-aarch64(1)	root(2)	host(2)	sqlite(2)
int(1)		fedora-40(1)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Two on Thursday, but not remote, so I don't know if they're the same bug or something new. The total so far:

• fedora-40 : int remote fedora-40 root host sqlite [remote]
  • PR libpod: hasCurrentUserMapped checks for gid too #24167
    • 10-04-2024 10:49 in Podman kube play should be able to run image where workdir is a symlink
• fedora-40-aarch64 : sys remote fedora-40-aarch64 root host sqlite [remote]
  • PR Housekeeping: remove duplicates from success_task #24279
    • 10-15 12:40 in [sys] |195| podman test all namespaces
  • PR vendor: update c/common and buildah #24135
    • 10-02-2024 13:03 in [sys] |200| podman pod create - hashtag AllTheOptions
• fedora-41 : int podman fedora-41 root host sqlite
  • PR fix(deps): update module golang.org/x/crypto to v0.29.0 #24501
    • 11-07 19:30 in Podman kube play test with hostIPC
    • 11-07 19:30 in Podman kube play test with sysctl defined

x	x	x	x	x	x
int(3)	remote(3)	fedora-41(2)	root(5)	host(5)	sqlite(5)
sys(2)	podman(2)	fedora-40-aarch64(2)
		fedora-40(1)

[edsantiago]

This one is blowing up, and I'm tentatively blaming it on the recent VM update. Issue title changed accordingly.

• fedora-40 : int remote fedora-40 root host sqlite [remote]
  • PR libpod: hasCurrentUserMapped checks for gid too #24167
    • 10-04-2024 10:49 in Podman kube play should be able to run image where workdir is a symlink
• fedora-40-aarch64 : sys remote fedora-40-aarch64 root host sqlite [remote]
  • PR Housekeeping: remove duplicates from success_task #24279
    • 10-15-2024 12:40 in [sys] |195| podman test all namespaces
  • PR vendor: update c/common and buildah #24135
    • 10-02-2024 13:03 in [sys] |200| podman pod create - hashtag AllTheOptions
• fedora-41 : int podman fedora-41 root host sqlite
  • PR [v5.3] Bump to Buildah v1.38.0 #24528
    • 11-11 14:51 in Podman kube play test with HostNetwork
  • PR fix(deps): update module golang.org/x/crypto to v0.29.0 #24501
    • 11-07 19:30 in Podman kube play test with hostIPC
    • 11-07 19:30 in Podman kube play test with sysctl defined
• fedora-41 : sys podman fedora-41 root host sqlite
  • PR Yet another bump, to fixed f41 kernel #24500
    • 11-08 07:29 in [sys] [255] podman auto-update - label io.containers.autoupdate=image
• fedora-41-aarch64 : sys remote fedora-41-aarch64 root host sqlite [remote]
  • PR fix(deps): update module github.com/moby/sys/capability to v0.4.0 #24534
    • 11-12 07:38 in [sys] |195| podman test all namespaces
    • 11-11 17:03 in [sys] |090| events - docker compat
• rawhide : int podman rawhide root host sqlite
  • PR fix(deps): update module github.com/moby/sys/capability to v0.4.0 #24534
    • 11-12 07:46 in Podman kube play podman pod logs test
    • 11-12 07:46 in Podman kube play podman-remote pod logs test
    • 11-12 07:46 in Podman kube play test correct output
    • 11-11 17:10 in Podman kube play test with hostIPC
    • 11-11 17:10 in Podman kube play test with sysctl defined
• rawhide : sys podman rawhide rootless host sqlite
  • PR system tests: safer install_kube_template() #24515
    • 11-11 13:01 in [sys] [255] podman auto-update - label io.containers.autoupdate=image

x	x	x	x	x	x
int(9)	podman(10)	rawhide(6)	root(14)	host(15)	sqlite(15)
sys(6)	remote(5)	fedora-41(4)	rootless(1)
		fedora-40-aarch64(2)
		fedora-41-aarch64(2)
		fedora-40(1)

[edsantiago]

cirrus-vm-get-versions, trimmed to remove packages that can't possibly (?) be causing this. New:

	debian	prior-fedora	fedora	fedora-aws	rawhide
base	13.5	Generic	Generic-41-1.4	?	42-0
kernel	6.11.6-1	6.8.5-301	6.11.6-300	6.11.6-300	6.12.0-0.rc6.20241105git2e1b3cc9d7f7.52
conmon	2.1.12-3	2.1.12-2	2.1.12-3	2.1.12-3	2.1.12-3
containers-common	?	0.60.4-2	0.60.4-4	0.60.4-4	0.60.4-5
crun	1.18.2-1	1.17-1	1.18.2-1	1.18.1-1	1.18.2-1
golang	2:1.23~2	1.22.7-1	1.23.2-2	1.23.2-2	1.23.2-2
systemd	257~rc1-3	255.13-1	256.7-1	256.7-1	256.7-1

...and old (c20241016t144444z-f40f39d13, the VMs that were running fine):

	debian	prior-fedora	fedora	fedora-aws	rawhide
base	13.5	39-1.5	Generic	?	42-0
kernel	6.11.2-1	6.5.6-300	6.8.5-301	6.8.5-301	6.8.5-301
conmon	2.1.12-1	2.1.12-2	2.1.12-2	2.1.12-2	2.1.12-3
containers-common	?	1-99	0.60.4-1	0.60.4-1	0.60.4-1
crun	1.17-1	1.17-1	1.17-1	1.17-1	1.17-1
golang	2:1.23~2	1.22.8-1	1.22.7-1	1.22.7-1	1.23.2-2
systemd	256.7-1	254.18-1	255.13-1	255.13-1	256.7-1

(I can't use the script's --baseline helper because these are different fedoræ)

Interesting observation: when we see this fail in system tests, it's often in the serial tests (the first pass). That points against it being a high-system-load issue.

So far, no luck reproducing on 1mt.

[edsantiago]

I guess this isn't too shocking, but flake is now showing up in APIv2 tests:

         not ok 1195 [27-containersEvents] GET libpod/events?stream=false&since=(T) : select(.status | contains("died")).Action
         #  expected: died
         #    actual: 
         not ok 1196 [27-containersEvents] GET libpod/events?stream=false&since=(T) : select(.status | contains("died")).Actor.Attributes.containerExitCode
         #  expected: 1
         #    actual: