| Summary: | C164/C165 Testing Shows Fewer IPS Rules Downloaded Compared to C163 Operational System | ||
|---|---|---|---|
| Product: | IPFire | Reporter: | Charles Brown <cab_77573> |
| Component: | --- | Assignee: | Stefan Schantl <stefan.schantl> |
| Status: | CLOSED FIXED | QA Contact: | Michael Tremer <michael.tremer> |
| Severity: | - Unknown - | ||
| Priority: | - Unknown - | CC: | peter.mueller, stefan.schantl |
| Version: | 2 | ||
| Hardware: | unspecified | ||
| OS: | Unspecified | ||
| Attachments: |
C163 Operational System Rule File Sizes
c164/C165 Test System Rule File Sizes ls -lSr *.rules from c163 operational system ls -lSr *.rules from c164/c165 test system |
||
(In reply to Charles Brown from comment #0) > Talos Subscribed rules on c163 operational system > [root@ipfire bin]# ./getrulecount.sh > security-ips rules => 17534 > balanced-ips rules => 9008 > max-detect-ips rules => 34948 > other ips rules => 10593 > ----------------------------------- > Total Rule Count => 43141 > > Same rule set running on my c164/165 test system I get ... > [root@ipfire bin]# ./getrulecount.sh > security-ips rules => 4282 > balanced-ips rules => 1102 > max-detect-ips rules => 9291 > other ips rules => 15967 > ----------------------------------- > Total Rule Count => 23958 > > Note the short set is from fresh a install of c165 without doing any backup > restore When I got to /var/lib/suricata on my c163 operational system and do: du -sh . I get 24M on the c164/c165 test system, it only shows 21M Created attachment 1000 [details]
C163 Operational System Rule File Sizes
Created attachment 1001 [details]
c164/C165 Test System Rule File Sizes
Created attachment 1002 [details]
ls -lSr *.rules from c163 operational system
Created attachment 1003 [details]
ls -lSr *.rules from c164/c165 test system
Hello Charles, good find - it seems that the rules tarballs from soucefire (snort.org) contains rulefiles with the same name in different folders. The old extract logic in c163 merged or appended these files into one bigger one. The new logic simply overwrites them, because I did not noticed this during development. Best regards and a big thanks, -Stefan Just for my understanding: Is this bug affecting the release of Core Update 164? (In reply to Peter Müller from comment #7) > Just for my understanding: Is this bug affecting the release of Core Update > 164? I would think this is 'blocking' c164. Perhaps you should confirm with Stefan Schantl if this issue is in both c164 and c165. Hello Charles, I've sent a patch to hopefully fix this issue to the development mailing list. Could you please test and verify if you have the same amount of rules again? Thanks in advance, -Stefan Missed the link to the patch: https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1-stefan.schantl@ipfire.org/ (In reply to Stefan Schantl from comment #10) > Missed the link to the patch: > > https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1- > stefan.schantl@ipfire.org/ Well, that definitely provided more rules -- many more, in fact. du -sh on /var/lib/suricata reports 40M now vs 24M on c163. More details coming in attachments shortly (In reply to Charles Brown from comment #11) > (In reply to Stefan Schantl from comment #10) > > Missed the link to the patch: > > > > https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1- > > stefan.schantl@ipfire.org/ > > Well, that definitely provided more rules -- many more, in fact. > du -sh on /var/lib/suricata reports 40M now vs 24M on c163. > More details coming in attachments shortly Per earlier post, my c163 ops system shows this for Talos Subscribed Talos Subscribed rules on c163 operational system [root@ipfire bin]# ./getrulecount.sh security-ips rules => 17534 balanced-ips rules => 9008 max-detect-ips rules => 34948 other ips rules => 10593 ----------------------------------- Total Rule Count => 43141 However, now after applying the patch, I get security-ips rules => 19906 balanced-ips rules => 9292 max-detect-ips rules => 37361 other ips rules => 23182 ----------------------------------- Total Rule Count => 58515 It seems I get 3 extra rule files with the c164*/c165 test system -- at least 3 were present before applying your patch. Here are the file names subscripted-decoder.rules subscripted-deleted.rules subscripted-local.rules subscripted-preprocessor.rules (In reply to Charles Brown from comment #13) > It seems I get 3 extra rule files with the c164*/c165 test system -- at > least 3 were present before applying your patch. Here are the file names > > subscripted-decoder.rules > subscripted-deleted.rules > subscripted-local.rules > subscripted-preprocessor.rules perhaps I can't count -- these are all new to me since c163 Aha, subscripted-deleted.rules would seem to be the culprit leading to the inflated rule count when I do my grepping. It is the one really big file of the 4 (count 'em 4) new files. That one alone accounts 12,128 rules. Together, those 4 new files account for 12,324 additional rules that are not present in the same ruleset for c163 And there are still an additional 3050 rules in new test system (with your patch) that I haven't resolved with my simple caveman grepping Hello Charles, thanks for testing and your feedback - nice to hear that the patch fixes the issue and works even better than expected. @Michael, please backport this also to c164. FWIW, those 'four' new rule files that were not present in c163 have damn little of interest. One has a single 'security-ips' policy rule, Two have some 'max-detect-ips' policy rules, Three have rules with no policy.
/var/lib/suricata/subscripted-decoder.rules
security-ips rules => 0
balanced-ips rules => 0
max-detect-ips rules => 104
other ips rules => 37
-----------------------------------
Total Rule Count => 141
/var/lib/suricata/subscripted-deleted.rules
security-ips rules => 0
balanced-ips rules => 0
max-detect-ips rules => 0
other ips rules => 12128
-----------------------------------
Total Rule Count => 12128
/var/lib/suricata/subscripted-local.rules
security-ips rules => 0
balanced-ips rules => 0
max-detect-ips rules => 0
other ips rules => 0
-----------------------------------
Total Rule Count => 0
/var/lib/suricata/subscripted-preprocessor.rules
security-ips rules => 1
balanced-ips rules => 0
max-detect-ips rules => 38
other ips rules => 17
-----------------------------------
Total Rule Count => 55
(In reply to Stefan Schantl from comment #18) > Hello Charles, > > thanks for testing and your feedback - nice to hear that the patch fixes the > issue and works even better than expected. > > @Michael, please backport this also to c164. Hi Stefan, I hope my knit-picking hasn't caused you too much of a kerfuffle for you :-) Cheers -cab https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=2b12a010d7ea97996a3b7accaab0316cb71de515 @Michael: Kindly cherry-pick this one into "master" as well. Per my testing with /master/2022-03-07 18:53:09 +0000-b69659af, this issue is fixed |
Talos Subscribed rules on c163 operational system [root@ipfire bin]# ./getrulecount.sh security-ips rules => 17534 balanced-ips rules => 9008 max-detect-ips rules => 34948 other ips rules => 10593 ----------------------------------- Total Rule Count => 43141 Same rule set running on my c164/165 test system I get ... [root@ipfire bin]# ./getrulecount.sh security-ips rules => 4282 balanced-ips rules => 1102 max-detect-ips rules => 9291 other ips rules => 15967 ----------------------------------- Total Rule Count => 23958 Note the short set is from fresh a install of c165 without doing any backup restore