Bug 11839

Summary: Swap not working on core 122
Product: IPFire Reporter: Tim <ipfb>
Component: ---Assignee: Michael Tremer <michael.tremer>
Status: CLOSED FIXED QA Contact:
Severity: Major Usability    
Priority: - Unknown - CC: michael.tremer
Version: 2   
Hardware: x86_64   
OS: All   
Attachments: Log extract showing OOM Killer in action
Proposed patch

Description Tim 2018-08-30 21:09:57 UTC
Created attachment 620 [details]
Log extract showing OOM Killer in action

I am running a script to update the IDS rules regularly; this includes checking the updates before telling Snort to re-read the rules and also persisting the enable/disable state of rules over the update.  Unsurprisingly this takes a lot of memory.

In Core 120 and earlier the would lead to swapping of memory, however this is no longer happening under core 122.  Instead the Out-Of-Memory Killer is being invoked.  Temporarily increasing the amount of swap space to 1.5GB has made no difference.  I've attached a log extract showing the OOM Killer in action.

It appears that something in the update - probably the kernel change - has disabled memory swapping.

--

Background information

The script can be found here: https://github.com/timfprogs/ipfidsupdate/tree/version3

It calls:

wget to download the rule updates
tar to make a backup of the old rules
oinkmaster to update the rules
snort -T to test that the updated rules are correct
It then sends SIGHUP to the running instances of Snort to tell them to re-read the rules.  It's during the call to snort -T that the OOM killer is invoked; I believe that this is likely to be the point of peak memory usage.
Comment 1 Michael Tremer 2018-08-31 12:18:56 UTC
The swap beahviour shouldn't have changed, but we got a new kernel.

You could try to play around with the vm.swappiness sysctl parameter which is set to 0. That used for prevent the kernel from swapping too early. Maybe this disables it entirely now.
Comment 2 Tim 2018-08-31 18:49:37 UTC
While Wikipedia articles need to be taken with a pinch of salt, there does appear to have been a change in the meaning of swappiness=0, although I would have expected the change to be in the old kernel as well.  I'll try setting it to 1 and see what happens.

https://en.wikipedia.org/wiki/Swappiness
Comment 3 Tim 2018-09-01 15:25:41 UTC
With swappiness=1 the rule update last night went through without any processes being killed, and with swap space being used.

Looking at the references from the Wikipedia article, this one

https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel

  looks to be best - especially since it includes details of the kernel change.  If I'm reading it correctly, with swappiness=0 the kernel will swap memory, but can't handle the case where a sudden demand is made for more than the available amount of free physical memory - no matter how much swap space is available.

The mystery is why this fairly old change should only have come into effect now, but perhaps with the old kernel my memory usage was just the right side of a boundary and the different memory requirements have pushed me across the boundary.

Given that there are a number of tasks which can run suddenly on IPFire, perhaps the default value for swappiness should be 1?
Comment 4 Michael Tremer 2018-09-10 00:30:01 UTC
Yes, that sounds like a reasonable change.

Would you submit a patch for this?
Comment 5 Tim 2018-09-21 17:42:54 UTC
OK.  I've attached a patch.
Comment 6 Tim 2018-09-21 17:43:31 UTC
Created attachment 630 [details]
Proposed patch
Comment 7 Michael Tremer 2018-09-23 15:23:24 UTC
Great, this looks good.

Would you submit it to the mailing list, too?

https://wiki.ipfire.org/devel/submit-patches
Comment 8 Tim 2018-09-23 18:12:03 UTC
When I do that it rejects the email 'You are not allowed to post to this mailing list'
Comment 9 Michael Tremer 2018-09-23 21:22:07 UTC
You will have to subscribe first and then you will be able to post to the list. Go to https://lists.ipfire.org for that.
Comment 10 Michael Tremer 2018-12-19 20:34:26 UTC
This has been shipped with c125. Thanks for working on this!