Created attachment 620 [details] Log extract showing OOM Killer in action I am running a script to update the IDS rules regularly; this includes checking the updates before telling Snort to re-read the rules and also persisting the enable/disable state of rules over the update. Unsurprisingly this takes a lot of memory. In Core 120 and earlier the would lead to swapping of memory, however this is no longer happening under core 122. Instead the Out-Of-Memory Killer is being invoked. Temporarily increasing the amount of swap space to 1.5GB has made no difference. I've attached a log extract showing the OOM Killer in action. It appears that something in the update - probably the kernel change - has disabled memory swapping. -- Background information The script can be found here: https://github.com/timfprogs/ipfidsupdate/tree/version3 It calls: wget to download the rule updates tar to make a backup of the old rules oinkmaster to update the rules snort -T to test that the updated rules are correct It then sends SIGHUP to the running instances of Snort to tell them to re-read the rules. It's during the call to snort -T that the OOM killer is invoked; I believe that this is likely to be the point of peak memory usage.
The swap beahviour shouldn't have changed, but we got a new kernel. You could try to play around with the vm.swappiness sysctl parameter which is set to 0. That used for prevent the kernel from swapping too early. Maybe this disables it entirely now.
While Wikipedia articles need to be taken with a pinch of salt, there does appear to have been a change in the meaning of swappiness=0, although I would have expected the change to be in the old kernel as well. I'll try setting it to 1 and see what happens. https://en.wikipedia.org/wiki/Swappiness
With swappiness=1 the rule update last night went through without any processes being killed, and with swap space being used. Looking at the references from the Wikipedia article, this one https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel looks to be best - especially since it includes details of the kernel change. If I'm reading it correctly, with swappiness=0 the kernel will swap memory, but can't handle the case where a sudden demand is made for more than the available amount of free physical memory - no matter how much swap space is available. The mystery is why this fairly old change should only have come into effect now, but perhaps with the old kernel my memory usage was just the right side of a boundary and the different memory requirements have pushed me across the boundary. Given that there are a number of tasks which can run suddenly on IPFire, perhaps the default value for swappiness should be 1?
Yes, that sounds like a reasonable change. Would you submit a patch for this?
OK. I've attached a patch.
Created attachment 630 [details] Proposed patch
Great, this looks good. Would you submit it to the mailing list, too? https://wiki.ipfire.org/devel/submit-patches
When I do that it rejects the email 'You are not allowed to post to this mailing list'
You will have to subscribe first and then you will be able to post to the list. Go to https://lists.ipfire.org for that.
This has been shipped with c125. Thanks for working on this!