Ubuntu 22.04+ perf issue

[ami-GS]

Describe the bug

secnetperf results from 22.04 and 24.04 are slower than that of 20.04.

• Both physically separated and local VMs have similar problem
• HPS/RPS tests are significantly slow (tested locally)
• Throughput test is unstable. sometimes fast, sometimes not
• Tried sysconfig change (buffersize etc.). slow
• NIC config doesn't affect (GRO LRO are on by default)
• Tried Both openssl and openssl3. slow.
• Tried disabling AppArmor. slow.
• Server Ubuntu22.04+ and Client Ubuntu20.04 is slow
• Server Ubuntu20.04 and Client Ubuntu22.04+ is enough fast

Server side QUIC handshake related part become slower when Ubuntu22.04+?

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

# Server
$ ./artifacts/bin/linux/x64_Release_openssl/secnetperf -exec:maxtput -io:epoll -pollidle:10000
# Client
$ ./artifacts/bin/linux/x64_Debug_openssl/secnetperf -target:172.25.246.35 -exec:maxtput -rconn:1 -share:1 -prate:1 -tcp:0 -trimout -watchdog:25000 -pollidle:10000 -run:12s -io:epoll -conns:100

Expected behavior

both should be similar perf

Actual outcome

1/3 - 1/10 slower HPS perf

Additional details

No response

[ami-GS]

These are server side flamegraph by HPS test. both are Release build and using openssl.

20.04 (1866 HPS)

24.04 (1247 HPS)

[ami-GS]

Tried these

• A: Ubuntu20.04 Docker container on Ubuntu22.04 Hyper-V VM on Windows
• B: Ubuntu24.04 Docker container on Ubuntu22.04 Hyper-V VM on Windows
• C: Ubuntu20.04 Docker container on Ubuntu20.04 Hyper-V VM on Windows
• D: Ubuntu24.04 Docker container on Ubuntu20.04 Hyper-V VM on Windows
  A,B kernel is 6.5.0-1024-azure
  C,D kernel is 5.4.0-192-generic

Performance of
A <-> A and B <-> B are almost same. (480-590 HPS)
C <-> C, D <-> D, C <-> D and D <-> C are almost same. (1370-1550 HPS)

From these experiments, issue should be kernel related

memo
Ubuntu 20.04 VM: 5.4.0-192-generic
Ubuntu 22.04 VM: 6.5.0-1024-azure
Ubuntu 24.04 VM: 6.8.0-40-generic

[ami-GS]

vmstat

on kernel 5.4.0-192-generic (faster)

on kernel 6.5.0-1024-azure (slower)

faster side has slightly fewer interrupt but more context switches.
For me, Interesting part is beginning of the test, slower side consumes less user CPU, more idle CPU and same system CPU.