Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High CPU usage when running Katran in shared mode with bonding interface #235

Open
tantm3 opened this issue Sep 5, 2024 · 11 comments
Open

High CPU usage when running Katran in shared mode with bonding interface #235

tantm3 opened this issue Sep 5, 2024 · 11 comments

Comments

@tantm3
Copy link

tantm3 commented Sep 5, 2024 •
edited
Loading

Hi everyone!

I am currently running Katran as a L3 Director load balancer for our services.
I would like to run Katran with a bonding interface because I believe it's easier to add more network interfaces rather than servers for scaling Katran's workload.
I followed this issue (#13) and make the Katran work normally in shared mode and bonding interface with those command:

# Network config
1: lo: ...
2: ens2f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 xdp/id:1900 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 02:96:77:09:2b:73 brd ff:ff:ff:ff:ff:ff permaddr d4:f5:ef:36:1a:60
    altname enp55s0f0
3: ens2f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 xdp/id:1905 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 02:96:77:09:2b:73 brd ff:ff:ff:ff:ff:ff permaddr d4:f5:ef:36:1a:68
    altname enp55s0f1
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: ipip0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
6: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr 16dd:8fa:927c::
7: ipip60@NONE: <NOARP,UP,LOWER_UP> mtu 1452 qdisc noqueue state UNKNOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr 42d6:82ed:7cf5::
    inet6 fe80::40d6:82ff:feed:7cf5/64 scope link 
       valid_lft forever preferred_lft forever
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:96:77:09:2b:73 brd ff:ff:ff:ff:ff:ff
    inet 10.50.73.53/24 brd 10.50.73.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::96:77ff:fe09:2b73/64 scope link 
       valid_lft forever preferred_lft forever
## for xdp root I edited the install_xdproot.sh script.
## And I run just one command to add Katran loadbalancer xdp program
sudo ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/bpfprog/bpf/balancer.bpf.o -default_mac 58:e4:34:56:46:e0  -forwarding_cores=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19 -numa_nodes=0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1 -healthchecker_prog ./deps/bpfprog/bpf/healthchecking_ipip.o -intf=ens2f0 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=100000 -map_path /sys/fs/bpf/jmp_ens2 -prog_pos=2
## Katran VIP + REAL config
2024/09/08 04:23:39 vips len 1
VIP:        49.213.85.151 Port:     80 Protocol: tcp
Vip's flags: 
 ->49.213.85.171     weight: 1 flags: 
exiting
  • I do mapping irq with cpu:
i40e-ens2f0-TxRx-0(45) is affinitive with  00,00000001 (from CPU1 .....)
i40e-ens2f0-TxRx-1(46) is affinitive with  00,00000002
i40e-ens2f0-TxRx-2(47) is affinitive with  00,00000004
i40e-ens2f0-TxRx-3(48) is affinitive with  00,00000008
i40e-ens2f0-TxRx-4(49) is affinitive with  00,00000010
i40e-ens2f0-TxRx-5(50) is affinitive with  00,00000020
i40e-ens2f0-TxRx-6(51) is affinitive with  00,00000040
i40e-ens2f0-TxRx-7(52) is affinitive with  00,00000080
i40e-ens2f0-TxRx-8(53) is affinitive with  00,00000100
i40e-ens2f0-TxRx-9(54) is affinitive with  00,00000200
i40e-ens2f1-TxRx-0(95) is affinitive with  00,00000400
i40e-ens2f1-TxRx-1(96) is affinitive with  00,00000800
i40e-ens2f1-TxRx-2(97) is affinitive with  00,00001000
i40e-ens2f1-TxRx-3(98) is affinitive with  00,00002000
i40e-ens2f1-TxRx-4(99) is affinitive with  00,00004000
i40e-ens2f1-TxRx-5(100) is affinitive with  00,00008000
i40e-ens2f1-TxRx-6(101) is affinitive with  00,00010000
i40e-ens2f1-TxRx-7(102) is affinitive with  00,00020000
i40e-ens2f1-TxRx-8(103) is affinitive with  00,00040000
i40e-ens2f1-TxRx-9(104) is affinitive with  00,00080000 (to CPU 20)
  • The problem arose when I saw the Katran's statics, it's shown 100% lru miss rate
##katran_goclient -s -lru
summary: 6380747 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 6668858 pkts/sec. lru hit: -0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 6657124 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: -0.00%) fallback lru hit: 0 pkts/sec

  • And all 20 CPUs are consumed by ksoftirqd
    image

  • Here is the screenshot that shown the output of perf report:
    image

I am not sure this performance issue is relating to Katran or not. So, I post this question here to find some clues.

Feel free to ask me to provide more information!
@tantm3 tantm3 changed the title How to tunning performance Katran shared mode with bonding interface? High CPU usage when running Katran in shared mode with bonding interface Sep 8, 2024
@tantm3
Copy link
Author

tantm3 commented Sep 9, 2024

I updated one more test case from my research.
I want to know whether bonding interface cause the overload of CPU usage, so, I removed the bonding interface and ran Katran shared mode directly in two physical interfaces.

  • Here is my network config
1: lo: ....
2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:2118 qdisc mq state UP group default qlen 1000
    link/ether d4:f5:ef:ac:ac:f0 brd ff:ff:ff:ff:ff:ff
    altname enp18s0f0
    inet 10.50.73.55/24 brd 10.50.73.255 scope global ens1f0
       valid_lft forever preferred_lft forever
    inet6 fe80::d6f5:efff:feac:acf0/64 scope link 
       valid_lft forever preferred_lft forever
3: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:2123 qdisc mq state UP group default qlen 1000
    link/ether d4:f5:ef:ac:ac:f8 brd ff:ff:ff:ff:ff:ff
    altname enp18s0f1
    inet 10.50.73.52/24 brd 10.50.73.255 scope global ens1f1
       valid_lft forever preferred_lft forever
    inet6 fe80::d6f5:efff:feac:acf8/64 scope link 
       valid_lft forever preferred_lft forever
....
8: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
9: ipip0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
10: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr f27f:220f:4e91::
11: ipip60@NONE: <NOARP,UP,LOWER_UP> mtu 1452 qdisc noqueue state UNKNOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr baba:33a1:a4e6::
    inet6 fe80::b8ba:33ff:fea1:a4e6/64 scope link 
       valid_lft forever preferred_lft forever
  • Command that I used to run Katran
sudo ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/bpfprog/bpf/balancer.bpf.o -default_mac 58:e4:34:56:46:e0 -forwarding_cores=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19 -numa_nodes=0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1 -healthchecker_prog ./deps/bpfprog/bpf/healthchecking_ipip.o -intf=ens1f0 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=1000000 -map_path /sys/fs/bpf/jmp_ens1 -prog_pos=2
  • Here is CPU mapping information
i40e-ens1f0-TxRx-0(44) is affinitive with  00,00000001
i40e-ens1f0-TxRx-1(45) is affinitive with  00,00000002
i40e-ens1f0-TxRx-2(46) is affinitive with  00,00000004
i40e-ens1f0-TxRx-3(47) is affinitive with  00,00000008
i40e-ens1f0-TxRx-4(48) is affinitive with  00,00000010
i40e-ens1f0-TxRx-5(49) is affinitive with  00,00000020
i40e-ens1f0-TxRx-6(50) is affinitive with  00,00000040
i40e-ens1f0-TxRx-7(51) is affinitive with  00,00000080
i40e-ens1f0-TxRx-8(52) is affinitive with  00,00000100
i40e-ens1f0-TxRx-9(53) is affinitive with  00,00000200
i40e-ens1f1-TxRx-0(103) is affinitive with  00,00000400
i40e-ens1f1-TxRx-1(104) is affinitive with  00,00000800
i40e-ens1f1-TxRx-2(105) is affinitive with  00,00001000
i40e-ens1f1-TxRx-3(106) is affinitive with  00,00002000
i40e-ens1f1-TxRx-4(107) is affinitive with  00,00004000
i40e-ens1f1-TxRx-5(108) is affinitive with  00,00008000
i40e-ens1f1-TxRx-6(109) is affinitive with  00,00010000
i40e-ens1f1-TxRx-7(110) is affinitive with  00,00020000
i40e-ens1f1-TxRx-8(111) is affinitive with  00,00040000
i40e-ens1f1-TxRx-9(112) is affinitive with  00,00080000
  • Katran's stats:
    image
  • CPU Usage is also full
    image
  • Here is the output from perf command
  • watching with perf top
    image
  • Recording and perf report
    image
    image

There is a little performance improvement when running with physical interfaces (according to the output from Katran), but the CPU usage is still full.

@tantm3
Copy link
Author

tantm3 commented Sep 23, 2024

Hi @avasylev @tehnerd,

Could you guys share some thoughts on my setup?
I still struggle with this.

@tehnerd
Copy link
Contributor

tehnerd commented Sep 24, 2024

sudo sysctl -a | grep bpf ?

@tantm3
Copy link
Author

tantm3 commented Sep 25, 2024

Hi @tehnerd,

Here is the output:
image

@tehnerd
Copy link
Contributor

tehnerd commented Sep 26, 2024

Hmm. Strange.
Please collect
1 perf record -a -F 23 -- sleep 10
2 same perf as before (in your previous pictures; I guess you were using -g as well). When looking into report move to balancers bpf program and use 'a' shortcut. That would show assembly code so we would understand where exactly in bpf program we consume cpu

@tehnerd
Copy link
Contributor

tehnerd commented Sep 26, 2024

Also how the traffic pattern looks like ? Are they real tcp streams or just random packets

@tantm3
Copy link
Author

tantm3 commented Sep 26, 2024

Please collect 1 perf record -a -F 23 -- sleep 10 2 same perf as before (in your previous pictures; I guess you were using -g as well)

  • I have a little trouble getting assembly code from perf report.
  • With bpf program, it shows an error
image
  • With other processes, I used a shortcut and it returned the output like this one
image

  • Anyway, I share all the output that I collected from those commands.

  • perf record -a -F 23 -- sleep 10

image
  • Assembly code when I jump (press enter) deeper-dive in katran's bpf program
image
  • perf record -ag -- sleep 20
image
  • perf top --sort comm,dso
image

Again, it shows an error when I press a shortcut to the bpf program
image

@tantm3
Copy link
Author

tantm3 commented Sep 26, 2024 •
edited
Loading

Also how the traffic pattern looks like ? Are they real tcp streams or just random packets

  • I used Pktgen to generate traffic, here is the configuration:
    image

  • In short, I want to stimulate the syn flood packet and send it to Katran. I used xdpdump tool to capture the packet and the packets look like this:

06:24:05.724378 IP 49.213.85.169.14397 > 49.213.85.152.80: Flags [S], seq 74616:74622, win 8192, length 6: HTTP
06:24:05.724396 IP 49.213.85.169.14997 > 49.213.85.152.80: Flags [S], seq 74616:74622, win 8192, length 6: HTTP
06:24:05.724401 IP 49.213.85.169.14984 > 49.213.85.152.80: Flags [S], seq 74616:74622, win 8192, length 6: HTTP
  • Here is the Katran configuration
katran_goclient -l -server localhost:8080
2024/09/26 06:26:30 vips len 2
VIP:        49.213.85.153 Port:     80 Protocol: tcp
Vip's flags:
 ->49.213.85.171     weight: 1 flags:
VIP:        49.213.85.152 Port:     80 Protocol: tcp
Vip's flags:
 ->49.213.85.171     weight: 1 flags:
exiting

UPDATE:

driver: i40e
version: 5.15.0-122-generic
firmware-version: 10.53.7
expansion-rom-version:
bus-info: 0000:37:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
  • And I saw a lot of package drops at the rx flow
image
  • I think this is the important information that may show Katran is not causing high CPU usage.

@tehnerd
Copy link
Contributor

tehnerd commented Sep 26, 2024

Feels like bpf program is not jitted. Could you please run bpftool prog list and bpftool map list

@tehnerd
Copy link
Contributor

tehnerd commented Sep 26, 2024

Also in perf report (which was taken with -ag)
Please show the output filtered with bpf keyword (/ and than type bpf)

@tantm3
Copy link
Author

tantm3 commented Sep 27, 2024 •
edited
Loading

Yes sure, here is the output of those commands
bpftool prog list

image

bpftool map list

image
image

perf report with the output filtered with bpf keyword

image

UPDATE
Here is the interface tag, it seems like the bpf program is jitted, at least from the outside view, hope this information is helpful.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tehnerd @tantm3