After upgrading from CU168 to CU170 I experience massive problems with heavily increased RTT due to packets not reaching the remote on first shot. One can see statistics like this on the remote host: RX packets 59551 bytes 10061459 (9.5 MiB) RX errors 45175 dropped 0 overruns 0 frame 0 TX packets 56693 bytes 11750353 (11.2 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 On the IPFire’ side all looks fine: RX packets 78639 bytes 13943835 (13.2 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 97137 bytes 50830661 (48.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 The ethernet-chips on all affected IPFire devices are: Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) Subsystem: Intel Corporation I211 Gigabit Network Connection Kernel driver in use: igb Kernel modules: igb First I thought that it might be related to bug #12750 since one of the affected remote hosts used such a device, but even after chaning to a Realtek-based adapter, the problem persists. Furthermore, it figured out that the hardware on remote side doesn't seem to matter, as pinging a Netgear Switch also shows exactly the same shaky RTT and packet loss. More informations and observations can be found here: https://community.ipfire.org/t/paket-loss-on-remote-system-after-upgrade-to-core-170/8690
Bootlog entries of the "igb" driver do not differ, just the order of how the system finds the devices seems different, though. No error found there. I noticed that the module IPFire uses actually is missing the version number: [ 4.332006] igb: Copyright (c) 2007-2014 Intel Corporation. While on an Ubuntu system it looks like: [ 15.081317] kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k [ 15.228785] kernel: igb: Copyright (c) 2007-2014 Intel Corporation. Why this? Even "modinfo" does not display the version of the driver: modinfo on IPfire: filename: /lib/modules/5.15.59-ipfire/kernel/drivers/net/ethernet/intel/igb/igb.ko.xz license: GPL v2 description: Intel(R) Gigabit Ethernet Network Driver author: Intel Corporation, <e1000-devel@lists.sourceforge.net> srcversion: 096D8F12DA47D6BCE79F918 modinfo on Ubuntu: filename: /lib/modules/5.4.0-125-generic/kernel/drivers/net/ethernet/intel/igb/igb.ko version: 5.6.0-k license: GPL v2 description: Intel(R) Gigabit Ethernet Network Driver author: Intel Corporation, <e1000-devel@lists.sourceforge.net> srcversion: B1937764485BFCD6F542175 So, what version of igb do Core 168 and 170 actually use? Are they different? Any other logs that may help?
Created attachment 1101 [details] Screenshot of htop showing ksoftirqd threads
Also, there seem to be high CPU load taken by ksoftirqd threads now after update to core 170 (see attachment). From my understanding, high cpu usage of these workers mean that there are IRQ in a waiting queue. I did not see that in core 168 before. Maybe it's related to the problem here. /proc/interrupts show that the network devices are nailed to specific cpu cores. No IRQ balancing here. Is that by design? CPU0 CPU1 CPU2 CPU3 0: 41 0 0 0 IR-IO-APIC 2-edge timer 1: 0 0 4 0 IR-IO-APIC 1-edge i8042 8: 0 0 0 8420 IR-IO-APIC 8-fasteoi rtc0 9: 0 1 0 0 IR-IO-APIC 9-fasteoi acpi 12: 0 6 0 0 IR-IO-APIC 12-edge i8042 26: 152 0 0 0 IR-IO-APIC 26-fasteoi intel_ish_ipc 120: 0 0 0 0 DMAR-MSI 0-edge dmar0 121: 0 0 0 0 DMAR-MSI 1-edge dmar1 133: 0 0 0 370558 IR-PCI-MSI 294912-edge ahci[0000:00:12.0] 134: 0 0 0 0 IR-PCI-MSI 344064-edge xhci_hcd 135: 0 1 0 0 IR-PCI-MSI 524288-edge red0 136: 0 0 96881921 0 IR-PCI-MSI 524289-edge red0-rx-0 137: 0 0 0 86107013 IR-PCI-MSI 524290-edge red0-rx-1 138: 113105097 0 0 0 IR-PCI-MSI 524291-edge red0-tx-0 139: 0 143072576 0 0 IR-PCI-MSI 524292-edge red0-tx-1 140: 0 17 0 0 IR-PCI-MSI 229376-edge 141: 0 0 1 0 IR-PCI-MSI 1048576-edge green0 142: 0 0 0 1425110816 IR-PCI-MSI 1048577-edge green0-rx-0 143: 87309978 0 0 0 IR-PCI-MSI 1048578-edge green0-rx-1 144: 0 334674720 0 0 IR-PCI-MSI 1048579-edge green0-tx-0 145: 0 0 1455484607 0 IR-PCI-MSI 1048580-edge green0-tx-1 146: 1 0 0 0 IR-PCI-MSI 1572864-edge orange0 147: 0 178428105 0 0 IR-PCI-MSI 1572865-edge orange0-rx-0 148: 0 0 1387667154 0 IR-PCI-MSI 1572866-edge orange0-rx-1 149: 0 0 0 1541885858 IR-PCI-MSI 1572867-edge orange0-tx-0 150: 59202681 0 0 0 IR-PCI-MSI 1572868-edge orange0-tx-1 151: 0 0 0 1 IR-PCI-MSI 3145728-edge blue0 152: 9252405 0 0 0 IR-PCI-MSI 3145729-edge blue0-rx-0 153: 0 14709737 0 0 IR-PCI-MSI 3145730-edge blue0-rx-1 154: 0 0 14514237 0 IR-PCI-MSI 3145731-edge blue0-tx-0 155: 0 0 0 16171849 IR-PCI-MSI 3145732-edge blue0-tx-1
Finally I found a difference between a system that is affected by the bug, and a similar system where the bug does not occur: System with problems: 08:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) Subsystem: Intel Corporation I211 Gigabit Network Connection Flags: bus master, fast devsel, latency 0, IRQ 20, IOMMU group 6 Memory at 81100000 (32-bit, non-prefetchable) [size=128K] I/O ports at 9000 [size=32] Memory at 81120000 (32-bit, non-prefetchable) [size=16K] (...) System without problems: 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) Subsystem: Intel Corporation I211 Gigabit Network Connection Flags: bus master, fast devsel, latency 0, IRQ 21 Memory at 81100000 (32-bit, non-prefetchable) [size=128K] I/O ports at b000 [size=32] Memory at 81120000 (32-bit, non-prefetchable) [size=16K] (...) As you can see, on the first system IOMMU is enabled, probably via BIOS. This seems to be disabled on the 2nd system. I read that enabled IOMMU can lead to problems with certain drivers in Linux, so for a test I disabled IOMMU by changing line linux /vmlinuz-5.15.59-ipfire root=UUID=e8cf0bb4-4816-451b-ab5b-9ae2881097f0 ro panic=10 rd.auto to linux /vmlinuz-5.15.59-ipfire root=UUID=e8cf0bb4-4816-451b-ab5b-9ae2881097f0 ro panic=10 rd.auto iommu=off in /boot/grub/grub.cfg and rebooted. Indeed the packet lag and retransmission problems disappeared, but since the system now only can access 3.73GB RAM, this can only be a temporary work-around for this issue. It still would be great if we could find out why an enabled IOMMU was no problem with former releases and eventually find and fix the root cause.
I suspect it is a bug in the kernel. I recently found some kernel mailing lists about iommu when I was searching about iommu with regard to sata drives. I can't find those kernel mails again but they mentioned that iommu had been turned on by default in one of the kernel updates because they believed it was generally usable but then found a lot of complaints coming from intel based hardware. CU171 is likely being issued for Testing by end of this week or so and it has a new kernel release in that. Maybe that kernel version might have a fix. Would be worth testing out. Will try and do further searching to see if I can find the info again.
(In reply to Adolf Belka from comment #5) > I suspect it is a bug in the kernel. This definitely sounds like it. > CU171 is likely being issued for Testing by end of this week or so and it > has a new kernel release in that. Maybe that kernel version might have a > fix. Would be worth testing out. Yes, absolutely. That would be the next step. I do not want to disable IOMMU support across the distribution, because of one broken set of hardware if we can avoid it. It might also be a firmware bug, so updating that would worth trying, too.
Unfortunately even after enabling VT-d in BIOS, my staging device (which is the same brand and almost the same motherboard) does not show the bug. No matter what I set in BIOS, the pings are running with low latency and without any loss. I'm a bit clueless right now. So, I could install core 171 on my stage device, but since I can't reproduce the bug there, I can not test if the kernel update actually solves the problem. Also, I do not feel comfortable with installing the test branch on the productive systems. Sorry for that! At least the work-around with the kernel options works for me.
Another forum user having the same issue has reported that updating to CU171 unstable yesterday solved the issue for him. https://community.ipfire.org/t/packet-loss-on-remote-system-after-upgrade-to-core-170/8690 So when CU171 goes to full release then that should hopefully also fix the issue for @ChrisK
(In reply to ChrisK from comment #7) > Unfortunately even after enabling VT-d in BIOS, my staging device (which is > the same brand and almost the same motherboard) does not show the bug. No > matter what I set in BIOS, the pings are running with low latency and > without any loss. > I'm a bit clueless right now. That is quite interesting. Does the problem happen with BIOS settings put on default? It should be possible to disable IOMMU in the BIOS somewhere.
If I disable "VT-d" in BIOS, IOMMU seems no longer supplied to the OS. At least Linux does not recognize it and the igb-driver does not display "IOMMU" in lspci. Quite as if I disable it by putting "iommu=off" in the boot options. There is no dedicated setting for IOMMU in BIOS, but I read that many BIOS brands have a settings called "VT-d" that (at least for intel chips) also steers the state of IOMMU. For AMD-cpus there shall be a dedicated settings in BIOS regarding this article: https://us.informatiweb.net/tutorials/it/bios/enable-iommu-or-vt-d-in-your-bios.html#amd-iommu I plan to do a shot maintenance downtime for the productive IPFires next week. I'll take the chance and check the BIOS for these settings. I'm quite sure that "VT-d" is enabled there, maybe that was a default for these older boards? I'll let you know about my findings then.
I just updated one of the affected system to Core 171 stable and sorry to say, but it did not resolve the problem for me. As soon as I re-enable IOMMU again, the problems are back: PING 172.16.227.200 (172.16.227.200) 56(84) bytes of data. 64 bytes from 172.16.227.200: icmp_seq=1 ttl=64 time=618 ms 64 bytes from 172.16.227.200: icmp_seq=2 ttl=64 time=320 ms 64 bytes from 172.16.227.200: icmp_seq=3 ttl=64 time=830 ms 64 bytes from 172.16.227.200: icmp_seq=4 ttl=64 time=85.8 ms 64 bytes from 172.16.227.200: icmp_seq=5 ttl=64 time=681 ms 64 bytes from 172.16.227.200: icmp_seq=6 ttl=64 time=64.8 ms uname -r 5.15.71-ipfire Seems like I'll have to switch off VT-d in BIOS anyway.
What is the status of this bug now with CU179. There have been several kernel updates and also some firmware ones in the intervening period. Did disabling iommu in the bios work or not. I am not sure that there is very much that IPFire can do for this issue, as it looks to be a kernel or firmware bug.
As the issue seems to be more of a kernel or firmware bug and there has been no further update for 60 days on whether it is still occurring in the latest Core Updates 179 to 183 this bug is being closed. If it is believed to be an IPFire bug and is still valid for the current released Core Update then the bug can be reopened together with the supporting information.