Created attachment 1004 [details] talos_subscribed_badly-enbabled_sids Running with running c164 master/b69659af/x86_64, I noticed unexpected IPS rules triggering. Grepping through the ruleset files -- Talos Subcribed -- I found some 559 extra rule sids enabled compared with my c163 operational system. It seems these unexpectedly enabled sids do not have a metadata policy in their definition in the rule file -- not balanced-ips, not secuirty-ips, and not max-detect-ips. I believe somehow with the new feature allowing multiple providers, some sids are getting incorrectly enabled. The attachment has info on the specific rule files and sids.
Created attachment 1005 [details] New rulefiles present in c164 -- not in c163 downloads Three of the rule files do not exist on the c163 operational system -- *deleted.rules, *decoder.rules, and *preprocessor.rules. Those files, however, only account for about 60 of the 559 newly enabled sids.
Just tried again with "IPFire 2.27 (x86_64) - core164 Development Build: master/e895c2de" Did a fresh install and then added Talos Subscribed provider. Checked for the suspect sids; and yes they are still present. This time a few more than on the previous run -- from 559 before to 666 now. Grepping for suspect sids in /var/lib/suricata ... egrep "^drop .*sid:|^alert .*sid:" *.rules | egrep -v "policy.*-ips" | wc -l 666
Perhaps of interest, many (not all) of the suspect sids have metadata -- but metadata without a policy *-ips specified.
Seemingly possible related discussion here: https://community.ipfire.org/t/no-dns-and-webgui-after-recommended-new-firewall-settings/7394/22
Created attachment 1006 [details] fast.log with event storm It would seem the problem is only with both: Talos VRT rules for registered user Talos VRT rules with subscription Other rule set do not seem to be a problem See attached fast.log
(In reply to Charles Brown from comment #5) > Created attachment 1006 [details] > fast.log with event storm > > It would seem the problem is only with both: > > Talos VRT rules for registered user > Talos VRT rules with subscription > > Other rule set do not seem to be a problem > See attached fast.log The problem is present in c164 with "either" of the above. (Not necessary to have both ... sorry for any confusion)
Created attachment 1007 [details] Significant Diff in rule files from c163 to c164/5 I did some grepping through the files in /var/lib/suricata on my c163 operational system and on my c164/5 test system. As you can see from the attached, the number of enabled rules is very different between the two systems. Both are using Talos VRT Subscripted rules.
(In reply to Charles Brown from comment #7) > Created attachment 1007 [details] > Significant Diff in rule files from c163 to c164/5 > > I did some grepping through the files in /var/lib/suricata on my c163 > operational system and on my c164/5 test system. As you can see from the > attached, the number of enabled rules is very different between the two > systems. Both are using Talos VRT Subscripted rules. Earlier I thought there were only a few hundred extraneous rules with the new IDS version but as can be seen in this attachment there are actually several thousand newly enabled rules using the new IDS -- with Talos VRT Subscripted rules.
Hello Charles, a big thanks for pointing this out and providing such detailed information about the issue. I finally was able to nail it down and sent a fix to the mailing list: https://patchwork.ipfire.org/project/ipfire/patch/20220313192725.3955-1-stefan.schantl@ipfire.org/ With this the files on my testing system with Talos Registered Ruleset moved from 58664 Rules down to 55122. Testing with the old mechanism from core 163 provided only 42957 rules. The difference between the "fixed" and the "old" mechanism is located in the "deleted.rules" file which contains the difference of about 13k rules. Searching the net for more details of about that deleted.rules file gave me the following result: https://seclists.org/snort/2011/q3/262 So it seems this file is not required and also should not be used. So the question is keep the rules of this file or drop it like the old approach, and how should we handle this with other providers? Best regards, -Stefan
According to the Talos rulesets from Sourcefire, the Profepoint rulesets (emergingthreads) also contains such a deleted.rules file. Reading through the following doc gaves: https://tools.emergingthreats.net/docs/ETPro%20Rule%20Categories.pdf 12. Deleted–This category is for signatures removed from the rule set. Note that typically rules are retained in a deactivated state within their respective rule files (starting with a #) but some rules that are duplicates, moved from Pro to Open (and thus need a new SID for the Open rule) or are too problematic to retain are moved to the Deleted category.
Created attachment 1018 [details] CU-164 IPS Nightly Update Troubles Hello Stefan, Your ids-functions.pl patch definitely makes things better. However my system is still having some c164 issues. The attached is an edited snippet from syslog for nightly update at 01:25 am ... then a scheduled reboot at 02:15 am As you can see in the log Oink spews a gazziion Warnings (literally hundreds of thousands -- putting tens of megs into syslog. Then when Oink finishes, it does not reload the rules to Suricata. Note: Suricata doesn't get reloaded until the scheduled reboot 45 minutes later. Also, oink shows error downloading but then continues on. Probably related, I get the permanent "spinning wait" indication on the "Intrusion Prevention System" page whenever doing Customize ruleset or doing Force Ruleset update -- saying wait for completion (or something like that). Checking for for Oink process with 'ps -ef | grep oink' I wait for it to exit and then look for the lock file. The lock file is not present but the spinning wait indication continues.
I sent another patch to the mailing list, which finally skips the deleted.rules files and so the new extract and update mechanism provides the same amount of rules than the old meachanism. https://patchwork.ipfire.org/project/ipfire/patch/20220315172557.2451-1-stefan.schantl@ipfire.org/ I'll have a look on all remaining IDS related issues.
I did the 'C165: Fix ownership of suricata classification.config file' and applied your latest 'ids-functions.pl: Skip deleted.rules files' patch. Indeed, things are much better -- no spinning forever on ids.cgi. Still seeing: Mar 15 13:46:53 ipfire oinkmaster[10036]: <ERROR> Downloading ruleset for provider: subscripted. Mar 15 13:48:57 ipfire oinkmaster[10167]: <ERROR> Downloading ruleset for provider: emerging. and this: Mar 15 13:51:09 ipfire suricata: [ERRCODE: SC_ERR_FOPEN(44)] - Error opening file: "/usr/share/suricata/threshold.config": No such file or directory Looking much better, -Charles
Related comments / issues referenced at: https://community.ipfire.org/t/core-164-ips-ruleset-update-in-progress-with-no-end-upgrade-from-163-to-164/7447/9
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=e41bb76cc34e17e165cecfbfcd8f974faed23bb7
https://blog.ipfire.org/post/ipfire-2-27-core-update-165-released