Hello, I'm testing Core-Update 183 Development Build: master/b4880b75 with kernel 6.6.13-ipfire. Early this morning the kernel reported an 'assertion failure' and dropped the ORANGE and BLUE interfaces only. GREEN and RED remained up, despite all four adaptors being the same Intel-based NIC. My hardware profile is: https://fireinfo.ipfire.org/profile/12a8c1b37699b320895097608705c5cc374254a9 Jan 31 03:05:13 gateway kernel: ------------[ cut here ]------------ Jan 31 03:05:13 gateway kernel: RTNL: assertion failed at net/core/dev.c (2984) Jan 31 03:05:13 gateway kernel: WARNING: CPU: 0 PID: 17232 at net/core/dev.c:2984 netif_set_real_num_rx_queues+0x81/0xa0 Jan 31 03:05:13 gateway kernel: Modules linked in: tun snd_pcm_oss snd_mixer_oss it87 hwmon_vid xt_REDIRECT act_mirred act_con nmark em_ipt act_gact cls_basic ifb sch_ingress xt_layer7 xt_dccp xt_dscp cls_u32 sch_htb xt_NFQUEUE nfnetlink_queue xt_MASQUE RADE cfg80211 8021q garp cpufreq_conservative cpufreq_ondemand amd_pstate_ut xt_set ip_set_hash_net xt_multiport ip_set xt_has hlimit xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi i915 sch_cake intel_rapl_common intel_soc_dts_thermal i ntel_soc_dts_iosf intel_powerclamp coretemp kvm_intel drm_buddy ttm drm_display_helper drm_kms_helper snd_hda_intel kvm at24 i TCO_wdt snd_intel_dspcfg iTCO_vendor_support regmap_i2c snd_hda_codec snd_hda_core igb snd_hwdep irqbypass snd_pcm ptp psmouse snd_timer pps_core snd pcspkr dca tpm_crb lpc_ich i2c_algo_bit i2c_i801 soundcore tpm_tis mfd_core i2c_smbus tpm_tis_core tpm rfkill_gpio rfkill Jan 31 03:05:13 gateway kernel: lp parport_pc parport Jan 31 03:05:13 gateway kernel: efivarfs xfs crct10dif_pclmul crc32_pclmul polyval_generic ghash_clmulni_intel i2c_hid_acpi s ha512_ssse3 i2c_hid drm sha256_ssse3 sha1_ssse3 serio_raw i2c_core video wmi dm_mirror dm_region_hash dm_log dm_mod Jan 31 03:05:13 gateway kernel: CPU: 0 PID: 17232 Comm: kworker/0:1 Tainted: G W 6.6.13-ipfire #1 Jan 31 03:05:13 gateway kernel: Hardware name: Default string Default string/Aptio CRB, BIOS 5.6.5 08/01/2021 Jan 31 03:05:13 gateway kernel: Workqueue: pm pm_runtime_work Jan 31 03:05:13 gateway kernel: RIP: 0010:netif_set_real_num_rx_queues+0x81/0xa0 Jan 31 03:05:13 gateway kernel: Code: eb d4 80 3d 63 f9 d2 00 00 75 e1 ba a8 0b 00 00 48 c7 c6 ba bd e1 b9 48 c7 c7 28 6c ea b 9 c6 05 47 f9 d2 00 01 e8 bf 8b 80 ff <0f> 0b eb be b8 ea ff ff ff 31 d2 31 f6 31 ff c3 cc cc cc cc b8 ea Jan 31 03:05:13 gateway kernel: RSP: 0018:ffffc90008597ce8 EFLAGS: 00010246 Jan 31 03:05:13 gateway kernel: RAX: 0000000000000000 RBX: ffff888103a88000 RCX: 0000000000000000 Jan 31 03:05:13 gateway kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Jan 31 03:05:13 gateway kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000 Jan 31 03:05:13 gateway kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 Jan 31 03:05:13 gateway kernel: R13: ffff888103a88000 R14: ffff8881001975e8 R15: ffff888103a88000 Jan 31 03:05:13 gateway kernel: FS: 0000000000000000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000 Jan 31 03:05:13 gateway kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 31 03:05:13 gateway kernel: CR2: 00007f971eb84004 CR3: 0000000129030000 CR4: 00000000001026f0 Jan 31 03:05:13 gateway kernel: <TASK> Jan 31 03:05:13 gateway kernel: ? netif_set_real_num_tx_queues+0x1f3/0x210 Jan 31 03:05:13 gateway kernel: ? __warn+0x81/0x130 Jan 31 03:05:13 gateway kernel: ? netif_set_real_num_tx_queues+0x1f3/0x210 Jan 31 03:05:13 gateway kernel: ? report_bug+0x1a2/0x1d0 Jan 31 03:05:13 gateway kernel: ? handle_bug+0x49/0xa0 Jan 31 03:05:13 gateway kernel: ? exc_invalid_op+0x17/0x80 Jan 31 03:05:13 gateway kernel: ? asm_exc_invalid_op+0x1a/0x20 Jan 31 03:05:13 gateway kernel: ? netif_set_real_num_tx_queues+0x1f3/0x210 Jan 31 03:05:13 gateway kernel: ? netif_set_real_num_tx_queues+0x1f3/0x210 Jan 31 03:05:13 gateway kernel: __igb_open+0x407/0x6c0 [igb] Jan 31 03:05:13 gateway kernel: __igb_resume+0x23c/0x280 [igb] Jan 31 03:05:13 gateway kernel: ? __pfx_pci_pm_runtime_resume+0x10/0x10 Jan 31 03:05:13 gateway kernel: __rpm_callback+0x44/0x170 Jan 31 03:05:13 gateway kernel: rpm_callback+0x35/0x70 Jan 31 03:05:13 gateway kernel: ? __pfx_pci_pm_runtime_resume+0x10/0x10 Jan 31 03:05:13 gateway kernel: rpm_resume+0x576/0x7b0 Jan 31 03:05:13 gateway kernel: ? __pm_runtime_resume+0x58/0x80 Jan 31 03:05:13 gateway kernel: pm_runtime_work+0x70/0xb0 Jan 31 03:05:13 gateway kernel: process_one_work+0x174/0x340 Jan 31 03:05:13 gateway kernel: worker_thread+0x28c/0x3b0 Jan 31 03:05:13 gateway kernel: ? preempt_count_add+0x7e/0xb0 Jan 31 03:05:13 gateway kernel: ? _raw_spin_lock_irqsave+0x27/0x70 Jan 31 03:05:13 gateway kernel: ? __pfx_worker_thread+0x10/0x10 Jan 31 03:05:13 gateway kernel: kthread+0xf7/0x130 Jan 31 03:05:13 gateway kernel: ? __pfx_kthread+0x10/0x10 Jan 31 03:05:13 gateway kernel: ret_from_fork+0x34/0x50 Jan 31 03:05:13 gateway kernel: ? __pfx_kthread+0x10/0x10 Jan 31 03:05:13 gateway kernel: ret_from_fork_asm+0x1b/0x30 Jan 31 03:05:13 gateway kernel: </TASK> Jan 31 03:05:13 gateway kernel: ---[ end trace 0000000000000000 ]--- Jan 31 03:05:13 gateway kernel: ------------[ cut here ]------------ Jan 31 03:05:13 gateway kernel: RTNL: assertion failed at net/core/dev.c (2984)
I've searched but been unable to find any other evidence or anything else which could have triggered the problem.
Sorry, immediately preceding the syslog output below was this bit: Jan 31 03:05:06 gateway kernel: igb 0000:02:00.0 orange0: igb: orange0 NIC Link is Down Jan 31 03:05:06 gateway kernel: igb 0000:03:00.0 blue0: igb: blue0 NIC Link is Down Jan 31 03:05:09 gateway ntpd[6997]: Deleting interface #4 orange0, 172.16.18.1#123, interface stats: received=0, sent=0, dropp ed=0, active_time=111672 secs Jan 31 03:05:09 gateway ntpd[6997]: Deleting interface #5 blue0, 172.16.17.1#123, interface stats: received=0, sent=0, dropped =0, active_time=111672 secs Jan 31 03:05:13 gateway kernel: ------------[ cut here ]------------
Oh this doesn't look good. @Arne, can we have an update to 6.6.14, please? I tried to go through the changes and find anything that might have triggered this, but apart from lots of changes around the networking subsystem and ethtool I cannot put my finger on it. But it's worth a try: > https://git.ipfire.org/?p=thirdparty/kernel/stable.git;a=shortlog;h=refs/tags/v6.6.14
This may not be due to the new kernel. Searching back through old logs I see it has happened (two events logged each time) about every 2 to 3 months. See below. Do you suspect a kernel driver error? Any idea why it only affects some NICs, not all at once? I'm not sure how I can catch more evidence with such a long interval between problems. # ionice -c3 nice zgrep 'assertion failed' /var/log/messages.*.gz /var/log/messages.6.gz:Dec 20 03:04:32 gateway kernel: RTNL: assertion failed at net/core/dev.c (2918) /var/log/messages.6.gz:Dec 20 03:04:32 gateway kernel: RTNL: assertion failed at net/core/dev.c (2966) /var/log/messages.11.gz:Nov 16 19:58:52 gateway kernel: RTNL: assertion failed at net/core/dev.c (2893) /var/log/messages.11.gz:Nov 16 19:58:52 gateway kernel: RTNL: assertion failed at net/core/dev.c (2941) /var/log/messages.28.gz:Jul 22 03:04:27 gateway kernel: RTNL: assertion failed at net/core/dev.c (2893) /var/log/messages.28.gz:Jul 22 03:04:27 gateway kernel: RTNL: assertion failed at net/core/dev.c (2941) /var/log/messages.37.gz:May 18 20:53:20 gateway kernel: RTNL: assertion failed at net/core/dev.c (2891) /var/log/messages.37.gz:May 18 20:53:20 gateway kernel: RTNL: assertion failed at net/core/dev.c (2939) /var/log/messages.47.gz:Mar 9 03:04:27 gateway kernel: RTNL: assertion failed at net/core/dev.c (2891) /var/log/messages.47.gz:Mar 9 03:04:27 gateway kernel: RTNL: assertion failed at net/core/dev.c (2939)
Is this still a problem on recent kernels?
Yes, unfortunately I still have the error, but it is very rare. Thanks for checking! The most recent instance (on September 10th) was unusual. The previous one was in my /var/log/messages.27.gz file saying "March 14") so it was probably from 2024. Anyway, my IPFire router has all non-red interfaces connected to a Ubiquti Unifi switch. On the 10th they rolled out a firmware update which was unstable and many users reported problems. They later rolled back that firmware. The firmware update (and possibly the rollback) occured while I was asleep. When I woke in the morning and tried to diagnose why the WiFi was down I had to forcibly power down and power on the Unifi switch. After this other links came up, but IPFire didn't. So I pressed the power button for a graceful shutdown and then powered back on my IPFire router PC to resolve the problem. I needed to work from home within minutes so sadly didn't do any root cause analysis.
(In reply to dnl from comment #6) > I needed to work from home within minutes so sadly didn't do > any root cause analysis. That is alright. I assume you are on the latest release which would have a kernel 6.12.x?
Right now my system is running kernel 6.12.41-ipfire. However I did a quick search through 'messages' looking for pakfire installs. It seems that on September 10th it was the previous stable IPFire release. Does that help?
Hi @dnl - I presume this is still an issue? I was having a look at the assert message and it seems that the igb driver was forgotten when its sibling igc got the bugfix for this issue. It's basically a problem of not holding the RTNL lock (a network specific kernel lock) when the system resumes a NIC after having sent it to runtime suspend (happening normally a few seconds after a link drop). This is basically what the assert is screaming about. (you could confirm that runtime PM was enabled by checking the content of /sys/bus/pci/devices/${PCI_ADDR}/power/control with PCI_ADDR being where your NIC is located) I'd like to send a fix for this issue upstream. How can I credit you in the Reported-by tag? I'll also send a patch to the mailing list in order to include the fix in IPFire.
Hello Antonio, Thank you very much for picking up this bug! Yes, unfortunately it still happens. I've used an AI to check this answer, but these are my own words. Since November 2025 (as far back as my logs go) it has occurred three times: Date Links down RTNL assertion Links up Kernel 2026-04-08 04:08:22 04:09:33 04:09:36 6.18.7-ipfire 2026-08-19 04:08:31 04:09:37 04:09:40 6.18.32-ipfire 2026-09-10 04:08:54 04:10:00 04:10:03 6.18.32-ipfire Every time, orange0, blue0 and green0p0 lose link together. All three connect to the same Unifi switch on different VLANs. red0 stays up. About 66 seconds later the assertion fires and the links come back up 3 seconds after that. A short link drop on 2026-10-01 (3 seconds) did not trigger it for unknown reasons. On 6.18 the AI I'm using finds the assertion is in netdev_lock.h rather than net/core/dev.c, but the path is the same as you describe (from 2026-09-10): RTNL: assertion failed at ./include/net/netdev_lock.h (72) WARNING: CPU: 0 PID: 28519 at ./include/net/netdev_lock.h:72 netdev_ops_assert_locked.part.0+0x1f/0x30 CPU: 0 UID: 0 PID: 28519 Comm: kworker/0:1 Not tainted 6.18.32-ipfire #1 PREEMPT(voluntary) Workqueue: pm pm_runtime_work Call Trace: netif_set_real_num_tx_queues+0x2a6/0x2b0 __igb_open+0x1c0/0x710 [igb] __igb_resume+0x141/0x230 [igb] __rpm_callback+0x48/0x1f0 rpm_callback+0x34/0x80 rpm_resume+0x4b3/0x6d0 pm_runtime_work+0x70/0xb0 process_one_work+0x18c/0x350 worker_thread+0x196/0x310 kthread+0xfc/0x250 Yes, runtime PM is enabled on all four igb ports: red0 0000:01:00.0 power/control=auto orange0 0000:02:00.0 power/control=auto blue0 0000:03:00.0 power/control=auto green0p0 0000:04:00.0 power/control=auto The system now runs Core Update 205 with 6.18.60-ipfire (I always patch within a day or two of a new release). There has been no event since 2026-09-10, but no link drop of that kind either. Yes, I can test a patched kernel. I'd prefer to remain (semi)anonymous so am happy not to be credited with this. Besides I only reported it, you're doing the hard work! Thank you, dnl
(In reply to dnl from comment #10) > Hello Antonio, > > Thank you very much for picking up this bug! My pleasure - I like to dig into desperate kernel cases :-D > > Yes, unfortunately it still happens. I've used an AI to check this answer, > but these are my own words. Since November 2025 (as far back as my logs go) > it has occurred three times: > > Date Links down RTNL assertion Links up Kernel > 2026-04-08 04:08:22 04:09:33 04:09:36 6.18.7-ipfire > 2026-08-19 04:08:31 04:09:37 04:09:40 6.18.32-ipfire > 2026-09-10 04:08:54 04:10:00 04:10:03 6.18.32-ipfire > > Every time, orange0, blue0 and green0p0 lose link together. All three > connect to the same Unifi switch on different VLANs. red0 stays up. About 66 > seconds later the assertion fires and the links come back up 3 seconds after > that. A short link drop on 2026-10-01 (3 seconds) did not trigger it for > unknown reasons. Glad it still does, because it means we have a way to test the fix! > > On 6.18 the AI I'm using finds the assertion is in netdev_lock.h rather than > net/core/dev.c, but the path is the same as you describe (from 2026-09-10): > > RTNL: assertion failed at ./include/net/netdev_lock.h (72) > WARNING: CPU: 0 PID: 28519 at ./include/net/netdev_lock.h:72 > netdev_ops_assert_locked.part.0+0x1f/0x30 > CPU: 0 UID: 0 PID: 28519 Comm: kworker/0:1 Not tainted 6.18.32-ipfire #1 > PREEMPT(voluntary) > Workqueue: pm pm_runtime_work > Call Trace: > netif_set_real_num_tx_queues+0x2a6/0x2b0 > __igb_open+0x1c0/0x710 [igb] > __igb_resume+0x141/0x230 [igb] > __rpm_callback+0x48/0x1f0 > rpm_callback+0x34/0x80 > rpm_resume+0x4b3/0x6d0 > pm_runtime_work+0x70/0xb0 > process_one_work+0x18c/0x350 > worker_thread+0x196/0x310 > kthread+0xfc/0x250 > Yep, someRTNL code was moved, but the driver still behaves the same. > Yes, runtime PM is enabled on all four igb ports: > > red0 0000:01:00.0 power/control=auto > orange0 0000:02:00.0 power/control=auto > blue0 0000:03:00.0 power/control=auto > green0p0 0000:04:00.0 power/control=auto thanks for checking! > > The system now runs Core Update 205 with 6.18.60-ipfire (I always patch > within a day or two of a new release). There has been no event since > 2026-09-10, but no link drop of that kind either. > > Yes, I can test a patched kernel. Ok! I'll drop a patch to the list soonish and post its link here, so you can have a go and let us know. > > I'd prefer to remain (semi)anonymous so am happy not to be credited with > this. Besides I only reported it, you're doing the hard work! No problem at all - I just like to give each party the right to be credited appropriately, but if you don't want, it's fine with me.
Hi dnl, the patch was posted to the mailing list: https://lists.ipfire.org/development/20261113105708.3818370-1-antonio@mandelbit.com/T/#u Let me know if you are able to pick it up and build a new kernel/image for testing.