dpdk - Pod1 中的 UDPSender 使用 DPDK rte_eth_tx_burst API，在另一个 Pod recv_from() 套接字系统调用无法接收
问题描述
为了清楚起见，在 2021 年 6 月 14 日重写问题
设置细节：
- 运行 Centos 7.6 的 Linux 机器充当 Kubernetes 集群中的节点。Docker 是容器类型。更多机器细节：
[root@node19823 ~]# uname -a
Linux node19823 3.10.0-693.2.2.rt56.623.el7.x86_64 #1 SMP PREEMPT RT Thu Sep 14 16:53:49 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@node19823 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.5", GitCommit:"e338cf2c6d297aa603b50ad3a301f761b4173aa6", GitTreeState:"clean", BuildDate:"2020-12-09T11:18:51Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"[root@node19823 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.5", GitCommit:"e338cf2c6d297aa603b50ad3a301f761b4173aa6", GitTreeState:"clean", BuildDate:"2020-12-09T11:10:32Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
[root@node19823 ~]#
[root@node19823 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
Details of my Application
- 10 VFs are created on a mlx_core5 100G PF.
- DPDK version is 19-11
- Two pods Pod1 and Pod2 run on this machine.
- Pod1 and uses VF1. Pod2 uses another VF2 of the same 100G PF. VFs are assigned to Pods using Kubernetes SRIO-DevicePlugin and SRIOV-CNI plugin.
- Pod1 and Pod2 are supposed to exchange full-duplex UDP traffic.
- Pod1 uses DPDK-PMD-over-VF1 for both send & receive UDP packets. VF1 is setup with 1 rxQ and 1 txQ for this purpose.
- Pod2 uses DPDK-PMD-over-VF2 for send-alone. VF2 is set up with 1 rxQ and 1 txQ. For receiving UDP traffic, Pod2 uses a simple UDP-socket boudn to same IP address as of VF2.
Below are the traffic combinations tried
- pod2-dpdk-pmd-tx-over-vf2 --> pod1-Dpdk-pmd-rx-over-vf1 ==> SUCCESS.
- pod1-dpdk-pmd-tx-over-vf1 --> pod2-udp-socket-rx-bound-to-vf2 ==>
FAILURE.
- pod1-udp-socket-tx-bound-to-vf1 --> pod2-udp-socket-rx-bound-to-vf2 ==> SUCCESS.
Looking for help in understanding the reason for FAILURE. I have verified that
- A. Ethernet/IP/UDP headers filled by pod1 dpdk-sender (pod1-dpdk-pmd-tx-over-vf1) is correct . I have forwarded the packets constructed by this app to wireshark and wireshark did not show any errors.
- B. Even tcpdump inside Pod2 does not show the packets sent by pod1. When dstMac address is correct, I expected the packet to at least show up on the dst-machine (pod) - it is ok if it failed in the higher layers of pod's tcpip stack. But why does the packet not appear in wireshark?
- Am I missing any settings (PMD APIs or ethtool commands?) in offloading all rx packets to linux tcpip stack in vf2 (as said above, i want to send using DPDK-PMD but receive over udp-socket).
Does dpdk-sender work seamlessly with nondpdk-receiver? It should in my opinion as sender/receiver don't always have control on each-others' design.
As I said above, there are no problems if both Sender and receiver are in socket-system call mode.
/proc/dev/net in Pod1 (VF1 device name inside pod1 = netsxu )
[root@**cs-dpdk-sender*-1-64c7d64877-5ml7p bin]# cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
**netsxu**: 1431238 4207 0 0 0 0 0 4207 32796 190 0 0 0 0 0 0
netf1u: 949856 2808 0 0 0 0 0 2808 36222 204 0 0 0 0 0 0
eth0: 3017151 14452 0 0 0 0 0 0 20239378037 15505655 0 0 0 0 0 0
lo: 7618450 3500 0 0 0 0 0 0 7618450 3500 0 0 0 0 0 0
nete1c: 1120380 5599 0 0 0 0 0 2775 211850 3039 0 0 0 0 0 0
neto1c: 1485613 4337 0 0 0 0 0 4265 36142 233 0 0 0 0 0 0
/proc/dev/net in Pod2 (Vf2 inside pod2 = netsxu, udp-socket bound to netsxu)
[root@**cs-udp-rx**-1-8466587c76-mfc8r bin]# cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
**netsxu**: 950624 2820 0 0 0 0 0 2820 34674 197 0 0 0 0 0 0
netsim: 1968514 5816 0 0 0 0 0 5815 35166 200 0 0 0 0 0 0
eth0: 5049 34 0 0 0 0 0 0 2936 30 0 0 0 0 0 0
lo: 931 19 0 0 0 0 0 0 931 19 0 0 0 0 0 0
[root@cs-gnb-1-dltg-1-8466587c76-mfc8r bin]#