Talos Subscribed rules on c163 operational system [root@ipfire bin]# ./getrulecount.sh security-ips rules => 17534 balanced-ips rules => 9008 max-detect-ips rules => 34948 other ips rules => 10593 ----------------------------------- Total Rule Count => 43141 Same rule set running on my c164/165 test system I get ... [root@ipfire bin]# ./getrulecount.sh security-ips rules => 4282 balanced-ips rules => 1102 max-detect-ips rules => 9291 other ips rules => 15967 ----------------------------------- Total Rule Count => 23958 Note the short set is from fresh a install of c165 without doing any backup restore
(In reply to Charles Brown from comment #0) > Talos Subscribed rules on c163 operational system > [root@ipfire bin]# ./getrulecount.sh > security-ips rules => 17534 > balanced-ips rules => 9008 > max-detect-ips rules => 34948 > other ips rules => 10593 > ----------------------------------- > Total Rule Count => 43141 > > Same rule set running on my c164/165 test system I get ... > [root@ipfire bin]# ./getrulecount.sh > security-ips rules => 4282 > balanced-ips rules => 1102 > max-detect-ips rules => 9291 > other ips rules => 15967 > ----------------------------------- > Total Rule Count => 23958 > > Note the short set is from fresh a install of c165 without doing any backup > restore When I got to /var/lib/suricata on my c163 operational system and do: du -sh . I get 24M on the c164/c165 test system, it only shows 21M
Created attachment 1000 [details] C163 Operational System Rule File Sizes
Created attachment 1001 [details] c164/C165 Test System Rule File Sizes
Created attachment 1002 [details] ls -lSr *.rules from c163 operational system
Created attachment 1003 [details] ls -lSr *.rules from c164/c165 test system
Hello Charles, good find - it seems that the rules tarballs from soucefire (snort.org) contains rulefiles with the same name in different folders. The old extract logic in c163 merged or appended these files into one bigger one. The new logic simply overwrites them, because I did not noticed this during development. Best regards and a big thanks, -Stefan
Just for my understanding: Is this bug affecting the release of Core Update 164?
(In reply to Peter Müller from comment #7) > Just for my understanding: Is this bug affecting the release of Core Update > 164? I would think this is 'blocking' c164. Perhaps you should confirm with Stefan Schantl if this issue is in both c164 and c165.
Hello Charles, I've sent a patch to hopefully fix this issue to the development mailing list. Could you please test and verify if you have the same amount of rules again? Thanks in advance, -Stefan
Missed the link to the patch: https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1-stefan.schantl@ipfire.org/
(In reply to Stefan Schantl from comment #10) > Missed the link to the patch: > > https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1- > stefan.schantl@ipfire.org/ Well, that definitely provided more rules -- many more, in fact. du -sh on /var/lib/suricata reports 40M now vs 24M on c163. More details coming in attachments shortly
(In reply to Charles Brown from comment #11) > (In reply to Stefan Schantl from comment #10) > > Missed the link to the patch: > > > > https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1- > > stefan.schantl@ipfire.org/ > > Well, that definitely provided more rules -- many more, in fact. > du -sh on /var/lib/suricata reports 40M now vs 24M on c163. > More details coming in attachments shortly Per earlier post, my c163 ops system shows this for Talos Subscribed Talos Subscribed rules on c163 operational system [root@ipfire bin]# ./getrulecount.sh security-ips rules => 17534 balanced-ips rules => 9008 max-detect-ips rules => 34948 other ips rules => 10593 ----------------------------------- Total Rule Count => 43141 However, now after applying the patch, I get security-ips rules => 19906 balanced-ips rules => 9292 max-detect-ips rules => 37361 other ips rules => 23182 ----------------------------------- Total Rule Count => 58515
It seems I get 3 extra rule files with the c164*/c165 test system -- at least 3 were present before applying your patch. Here are the file names subscripted-decoder.rules subscripted-deleted.rules subscripted-local.rules subscripted-preprocessor.rules
(In reply to Charles Brown from comment #13) > It seems I get 3 extra rule files with the c164*/c165 test system -- at > least 3 were present before applying your patch. Here are the file names > > subscripted-decoder.rules > subscripted-deleted.rules > subscripted-local.rules > subscripted-preprocessor.rules perhaps I can't count -- these are all new to me since c163
Aha, subscripted-deleted.rules would seem to be the culprit leading to the inflated rule count when I do my grepping. It is the one really big file of the 4 (count 'em 4) new files. That one alone accounts 12,128 rules.
Together, those 4 new files account for 12,324 additional rules that are not present in the same ruleset for c163
And there are still an additional 3050 rules in new test system (with your patch) that I haven't resolved with my simple caveman grepping
Hello Charles, thanks for testing and your feedback - nice to hear that the patch fixes the issue and works even better than expected. @Michael, please backport this also to c164.
FWIW, those 'four' new rule files that were not present in c163 have damn little of interest. One has a single 'security-ips' policy rule, Two have some 'max-detect-ips' policy rules, Three have rules with no policy. /var/lib/suricata/subscripted-decoder.rules security-ips rules => 0 balanced-ips rules => 0 max-detect-ips rules => 104 other ips rules => 37 ----------------------------------- Total Rule Count => 141 /var/lib/suricata/subscripted-deleted.rules security-ips rules => 0 balanced-ips rules => 0 max-detect-ips rules => 0 other ips rules => 12128 ----------------------------------- Total Rule Count => 12128 /var/lib/suricata/subscripted-local.rules security-ips rules => 0 balanced-ips rules => 0 max-detect-ips rules => 0 other ips rules => 0 ----------------------------------- Total Rule Count => 0 /var/lib/suricata/subscripted-preprocessor.rules security-ips rules => 1 balanced-ips rules => 0 max-detect-ips rules => 38 other ips rules => 17 ----------------------------------- Total Rule Count => 55
(In reply to Stefan Schantl from comment #18) > Hello Charles, > > thanks for testing and your feedback - nice to hear that the patch fixes the > issue and works even better than expected. > > @Michael, please backport this also to c164. Hi Stefan, I hope my knit-picking hasn't caused you too much of a kerfuffle for you :-) Cheers -cab
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=2b12a010d7ea97996a3b7accaab0316cb71de515 @Michael: Kindly cherry-pick this one into "master" as well.
Per my testing with /master/2022-03-07 18:53:09 +0000-b69659af, this issue is fixed
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=9106bfca42a86f9720c4e2f5d0d166832cac6454
https://blog.ipfire.org/post/ipfire-2-27-core-update-165-released