Linux KVM SRIOV欺骗数据包丢帧

您好Jamesdblea，感谢您在英特尔有线以太网社区发帖。
您目前使用的是什么以太网适配器或控制器？
请发布输出oflspci |
grep以太网;
ethtool -i; andethtool -k。
如果您有任何其他问题，请不要犹豫。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for posting in Intel Wired Ethernet Communities.  What Ethernet adapter or controller are you currently using?  Please post the output of lspci | grep Ethernet; ethtool -i ; and ethtool -k.  If you have any other questions please do not hesitate to ask.

Best regards,
Daniel D
Intel Customer Support

None


以下为原文

Here is the last two examples I have.
 

# lspci | grep Ethernet
81:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
81:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
 
# lspci -v -s 81:00.0
81:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2
 
# ethtool -i ethx
driver: ixgbe
version: 5.3.7
firmware-version: 0x61c10001
expansion-rom-version:
bus-info: 0000:81:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -k ethx
Features for ethx:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on

 
 

# lspci | grep Ethernet
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


# lspci -v -s 05:00.0
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2

# ethtool -i ethx
driver: ixgbe
version: 5.3.7
firmware-version: 0x61c10001
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -k ethx
Features for ethx:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: off
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]

 

你好Jamesdblea，谢谢你的输出。
我们会调查此问题并尽快为您更新。
如果您有任何其他问题，请告诉我们。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for the outputs.  We will investigate the issue and update you as soon as possible.  Please let us know if you have any other questions.

Best regards,
Daniel D
Intel Customer Support

您好Jamesdblea，感谢您在调查此问题时的耐心等待。
Libvirt在使用前为VF分配有效的MAC地址。
由于i40e驱动程序中的安全功能，分配给VM的VF不允许从VM中更改VF MAC地址。
重复的MAC将导致通信丢失。
使用以下命令从主机手动分配MAC地址：ip link set vf 0 mac aa：bb：cc：dd：ee：ff有关详细信息，请参阅SR-IOV配置指南的第15页。
请验证VF上的MAC地址是否重复。
如果您需要进一步的帮助，请提供创建VF所采取的步骤。
如果您有任何疑问，请告诉我们。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for your patience while we investigate this issue.  Libvirt assigns a valid MAC address to the VF prior to use.  Due to a security feature in the i40e driver the VF assigned to the VM is not allowed to change the VF MAC address from within the VM.  A duplicate MAC will cause loss of communication.  Manually assign a MAC address from the host using the following command:
ip link set vf 0 mac aa:bb:cc:dd:ee:ff

See page 15 of the SR-IOV configuration guide for more information.  Please verify that the MAC address is not duplicated on the VF.  Provide the steps you take to create the VFs if you need further assistance.  If you have any questions please let us know.

Best regards,
Daniel D
Intel Customer Support

地狱丹尼尔，
这是从guest虚拟机发送的数据包的输出。
我还做了一个tcpdump来匹配任何与当前MAC不匹配的数据包。
有问题的客人是VF ​​7。
家长：
#ip link show dev ethx
6：ethx：mtu 1500 qdisc mq master br0 state UP模式DEFAULT组默认qlen 1000 
link / ether 90：e2：ba：0f：cb：90 brd ff：ff：ff：ff：ff：ff 
vf 0 MAC 52：54：00：f7：95：ed，欺骗检查，链接状态自动，信任，query_rss关闭 
vf 1 MAC 52：54：00：30：1b：67，欺骗检查，链接状态自动，信任，query_rss关闭 
vf 2 MAC 52：54：00：a7：7a：45，欺骗检查，链接状态自动，信任，query_rss关闭 
vf 3 MAC 52：54：00：4a：d3：cb，欺骗检查，链接状态自动，信任，查询_关闭 
vf 4 MAC 52：54：00：c8：9e：57，欺骗检查，链接状态自动，信任，query_rss关闭 
vf 5 MAC 52：54：00：17：44：c3，欺骗检查，链接状态自动，信任，query_rss关闭 
vf 6 MAC 52：54：00：45：b3：61，欺骗检查，链接状态自动，信任，query_rss关闭 
vf 7 MAC 52：54：00：fa：f7：dc，欺骗检查，链接状态自动，信任，query_rss关闭
儿童：
#ip link show dev eth0
2：eth0：mtu 1500 qdisc mq master bond0 state UP模式DEFAULT组默认qlen 1000 
link / ether 52：54：00：fa：f7：dc brd ff：ff：ff：ff：ff：ff
#ip link show dev bond0
3：bond0：mtu 1500 qdisc noqueue state UP模式DEFAULT组默认qlen 1000 
link / ether 52：54：00：fa：f7：dc brd ff：ff：ff：ff：ff：ff
#tcpdump -i eth0 -Q out not ether host 52：54：00：fa：f7：dc
tcpdump：详细输出被抑制，使用-v或-vv进行完整协议解码
监听eth0，链接类型EN10MB（以太网），捕获大小262144字节
捕获0个数据包
过滤器收到5837个数据包
内核丢弃4331个数据包
#tcpdump -i bond0 -Q out not ether host 52：54：00：fa：f7：dc
tcpdump：详细输出被抑制，使用-v或-vv进行完整协议解码
监听bond0，链接型EN10MB（以太网），捕获大小262144字节
捕获0个数据包
过滤器收到2480个数据包
内核丢弃了595个数据包



以下为原文

Hell Daniel,
 
Here is the output for packets being sent from the guest. I've also done a tcpdump to match any packets exiting that don't match the current MAC. Guest with the issue is VF 7.
 
Parent:

# ip link show dev ethx
6: ethx: mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:0f:cb:90 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 52:54:00:f7:95:ed, spoof checking on, link-state auto, trust on, query_rss off
    vf 1 MAC 52:54:00:30:1b:67, spoof checking on, link-state auto, trust on, query_rss off
    vf 2 MAC 52:54:00:a7:7a:45, spoof checking on, link-state auto, trust on, query_rss off
    vf 3 MAC 52:54:00:4a:d3:cb, spoof checking on, link-state auto, trust on, query_rss off
    vf 4 MAC 52:54:00:c8:9e:57, spoof checking on, link-state auto, trust on, query_rss off
    vf 5 MAC 52:54:00:17:44:c3, spoof checking on, link-state auto, trust on, query_rss off
    vf 6 MAC 52:54:00:45:b3:61, spoof checking on, link-state auto, trust on, query_rss off
    vf 7 MAC 52:54:00:fa:f7:dc, spoof checking on, link-state auto, trust on, query_rss off

 
Child:

# ip link show dev eth0
2: eth0: mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:fa:f7:dc brd ff:ff:ff:ff:ff:ff
# ip link show dev bond0
3: bond0: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:fa:f7:dc brd ff:ff:ff:ff:ff:ff
# tcpdump -i eth0 -Q out not ether host 52:54:00:fa:f7:dc
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
 
0 packets captured
5837 packets received by filter
4331 packets dropped by kernel
# tcpdump -i bond0 -Q out not ether host 52:54:00:fa:f7:dc
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
 
0 packets captured
2480 packets received by filter
595 packets dropped by kernel
 

你好Jamesdblea，谢谢你的输出。
我们会进一步调查此问题，并尽快回复您。
如果您有任何其他问题，请随时与我们联系。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for the outputs.  We will investigate the issue further, and get back to you soon.  If you have any other questions please do not hesitate to contact us.

Best regards,
Daniel D
Intel Customer Support

您好Jamesdblea，感谢您等我们调查此问题。
您是否尝试将孩子更改为VF7以外的VF并查看问题是否仍然发生在另一个VF上？
您能否为我们提供您使用的一些配置？
特别是QEMU和KVM设置，以及以太网接口的配置。
我们还注意到bond0已设置，但是它用于什么？
如果您有任何问题，请告诉我们。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for waiting while we investigate this issue.  Have you tried to change the child to a VF other than VF7 and see if the issue still occurs on another VF?  Would you be able to provide us some of the configurations you used.  Specifically QEMU and KVM settings, and configurations for the Ethernet interfaces.  We also notice bond0 is set, but what is it being used for?  Let us know if you have any questions.

Best regards,
Daniel D
Intel Customer Support

丹尼尔你好，
这次使用的实际VF是7，下次它将是另一个，接下来是另一个。
我正在解决这个问题，这个问题突然出现在几十台服务器上。
VF＃没有相关性。
Bond在少数情况下使用，其中客人是带有保税网络的父母，SRIOV是从两个网络到客人。
这种情况只有一个网络连接;
债券没有两个支持。
无论是否有债券，这种错误都会发生。
以下是我们用于直通的一些配置，如果您需要更具体的一些，请告诉我。
客人通过libvirt：         
QEMU流程：
qemu 31168 246 12.7 8947996 8404636？
SL10 Oct04 4024：19 / usr / libexec / qemu-kvm -name GUESTNAME -S -machine pc-i440fx-rhel7.0.0，accel = kvm，u*** = off，dump-guest-core = off -cpu IvyBridge，+ ds，
+ ACPI，+ SS，+ HT，+ TM，+ PBE，+ dtes64，+监视器，+ ds_cpl，+ VMX，+ SMX，+ EST，+ TM2，+ xtpr，+ PDCM，+ PCID，+ DCA，+ osxsave
，+ arat，+ xsaveopt，+ pdpe1gb，-spec-ctrl -m 8192 -realtime mlock = off -smp 4，sockets = 4，cores = 1，threads = 1 -uuid fc9c3820-a1ab-4cab-8671-717d4e2560c0 -display
none -n​​o-user-config -nodefaults -chardev socket，id = charmonitor，path = / var / lib / libvirt / qemu / domain-29-GUESTNAME / monitor.sock，server，nowait -mon chardev = charmonitor，id = monitor
，mode = control -rtc base = utc，driftfix = slew -global kvm-pit.lost_tick_policy = delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3 = 1 -global PIIX4_PM.disable_s4 = 1 -boot strict = on -
设备ich9-u***-ehci1，id = u***，bus = pci.0，addr = 0x3.0x7 -device ich9-u***-uhci1，masterbus = u***.0，firstport = 0，bus = pci.0，multifunction = on，
addr = 0x3 -device ich9-u***-uhci2，masterbus = u***.0，firstport = 2，bus = pci.0，addr = 0x3.0x1 -device ich9-u***-uhci3，masterbus = u***.0
，firstport = 4，bus = pci.0，addr = 0x3.0x2 -device ahci，id = sata0，bus = pci.0，addr = 0x4 -drive file = / imgs / GUESTNAME / guest.disk0，format = raw，
if = none，id = drive-sata0-0-0，discard = unmap -device ide-hd，bus = sata0.0，drive = drive-sata0-0-0，id = sata0-0-0，bootindex = 1
-chardev pty，id = charserial0 -device isa-serial，chardev = charserial0，id = serial0 -device vfio-pci，host = 05：11.2，id = hostdev0，bus = pci.0，addr = 0x6 -device virtio-balloon
-pci，id = balloon0，bus = pci.0，addr = 0x5 -msg timestamp = on



以下为原文

Hello Daniel,
 
The actual VF used this time was 7, next time it'll be another, next another. I'm troubleshooting this issue popping up on more than several dozen servers. No correlation on VF#.
 
Bond is used in a few cases where guests are on parents with a bonded network and SRIOV from two nics to the guest. This case there is only one network connection; bond isn't backed up by two. Again this error can happen with or without bond.
 
Here is some configs we use for passthrough, if you need some more specific, please let me know.
 
Guest passthrough libvirt:

   
     
     
     
       

     
     
     

   

QEMU process:

qemu     31168  246 12.7 8947996 8404636 ?     SLl  Oct04 4024:19 /usr/libexec/qemu-kvm -name GUESTNAME -S -machine pc-i440fx-rhel7.0.0,accel=kvm,u***=off,dump-guest-core=off -cpu IvyBridge,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+osxsave,+arat,+xsaveopt,+pdpe1gb,-spec-ctrl -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid fc9c3820-a1ab-4cab-8671-717d4e2560c0 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-29-GUESTNAME/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-u***-ehci1,id=u***,bus=pci.0,addr=0x3.0x7 -device ich9-u***-uhci1,masterbus=u***.0,firstport=0,bus=pci.0,multifunction=on,addr=0x3 -device ich9-u***-uhci2,masterbus=u***.0,firstport=2,bus=pci.0,addr=0x3.0x1 -device ich9-u***-uhci3,masterbus=u***.0,firstport=4,bus=pci.0,addr=0x3.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x4 -drive file=/imgs/GUESTNAME/guest.disk0,format=raw,if=none,id=drive-sata0-0-0,discard=unmap -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device vfio-pci,host=05:11.2,id=hostdev0,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

你好Jamesdblea，谢谢你的回复。
我们将考虑Libvirt和QEMU配置。
会很快给你更新。
如果您有任何问题，请告诉我们。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for the reply.  We will take the Libvirt and QEMU configurations into consideration.  Will update you soon.  Let us know if you have any questions.

Best regards,
Daniel D
Intel Customer Support  

你好Jamesdblea，谢谢你的耐心等待。
在运行CentOS 7.5（父）和Ubuntu 15（子）的计算机上使用82599 2端口以太网控制器，我们无法复制该问题。
我们按照SR-IOV配置指南中的步骤操作，并在端口0上启用了2x VF。没有帧被丢弃到另一个主机或父节点到子节点。
您的配置中是否存在偏离指南的内容？
如果您有任何问题，请告诉我们。最好的问候，Daniel DIntel客户支持



以下为原文

Hello Jamesdblea,

Thank you for your patience.  Using an 82599 2 port Ethernet controller on a machine running CentOS 7.5 (Parent) and Ubuntu 15 (Child) we could not duplicate the issue.  We followed the steps in SR-IOV configuration Guide  and enabled 2x VF on port 0.  No frames were dropped child to another host or parent to child.  Is there something in your configuration that deviates from the guide?  Let us know if you have any questions.

Best regards,
Daniel D
Intel Customer Support