Bug 11839

Summary:	Swap not working on core 122
Product:	IPFire	Reporter:	Tim <ipfb>
Component:	---	Assignee:	Michael Tremer <michael.tremer>
Status:	CLOSED FIXED	QA Contact:
Severity:	Major Usability
Priority:	- Unknown -	CC:	michael.tremer
Version:	2
Hardware:	x86_64
OS:	All
Attachments:	Log extract showing OOM Killer in action Proposed patch

Description Tim 2018-08-30 21:09:57 UTC

Created attachment 620 [details]
Log extract showing OOM Killer in action

I am running a script to update the IDS rules regularly; this includes checking the updates before telling Snort to re-read the rules and also persisting the enable/disable state of rules over the update.  Unsurprisingly this takes a lot of memory.

In Core 120 and earlier the would lead to swapping of memory, however this is no longer happening under core 122.  Instead the Out-Of-Memory Killer is being invoked.  Temporarily increasing the amount of swap space to 1.5GB has made no difference.  I've attached a log extract showing the OOM Killer in action.

It appears that something in the update - probably the kernel change - has disabled memory swapping.

--

Background information

The script can be found here: https://github.com/timfprogs/ipfidsupdate/tree/version3

It calls:

wget to download the rule updates
tar to make a backup of the old rules
oinkmaster to update the rules
snort -T to test that the updated rules are correct
It then sends SIGHUP to the running instances of Snort to tell them to re-read the rules.  It's during the call to snort -T that the OOM killer is invoked; I believe that this is likely to be the point of peak memory usage.

Comment 1 Michael Tremer 2018-08-31 12:18:56 UTC

The swap beahviour shouldn't have changed, but we got a new kernel.

You could try to play around with the vm.swappiness sysctl parameter which is set to 0. That used for prevent the kernel from swapping too early. Maybe this disables it entirely now.

Comment 2 Tim 2018-08-31 18:49:37 UTC

While Wikipedia articles need to be taken with a pinch of salt, there does appear to have been a change in the meaning of swappiness=0, although I would have expected the change to be in the old kernel as well.  I'll try setting it to 1 and see what happens.

https://en.wikipedia.org/wiki/Swappiness

Comment 3 Tim 2018-09-01 15:25:41 UTC

With swappiness=1 the rule update last night went through without any processes being killed, and with swap space being used.

Looking at the references from the Wikipedia article, this one

https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel

  looks to be best - especially since it includes details of the kernel change.  If I'm reading it correctly, with swappiness=0 the kernel will swap memory, but can't handle the case where a sudden demand is made for more than the available amount of free physical memory - no matter how much swap space is available.

The mystery is why this fairly old change should only have come into effect now, but perhaps with the old kernel my memory usage was just the right side of a boundary and the different memory requirements have pushed me across the boundary.

Given that there are a number of tasks which can run suddenly on IPFire, perhaps the default value for swappiness should be 1?

Comment 4 Michael Tremer 2018-09-10 00:30:01 UTC

Yes, that sounds like a reasonable change.

Would you submit a patch for this?

Comment 5 Tim 2018-09-21 17:42:54 UTC

OK.  I've attached a patch.

Comment 6 Tim 2018-09-21 17:43:31 UTC

Created attachment 630 [details]
Proposed patch

Comment 7 Michael Tremer 2018-09-23 15:23:24 UTC

Great, this looks good.

Would you submit it to the mailing list, too?

https://wiki.ipfire.org/devel/submit-patches

Comment 8 Tim 2018-09-23 18:12:03 UTC

When I do that it rejects the email 'You are not allowed to post to this mailing list'

Comment 9 Michael Tremer 2018-09-23 21:22:07 UTC

You will have to subscribe first and then you will be able to post to the list. Go to https://lists.ipfire.org for that.

Comment 10 Michael Tremer 2018-12-19 20:34:26 UTC

This has been shipped with c125. Thanks for working on this!