Bug 12792

Summary: C164/C165 Testing Shows Fewer IPS Rules Downloaded Compared to C163 Operational System
Product: IPFire Reporter: Charles Brown <cab_77573>
Component: ---Assignee: Stefan Schantl <stefan.schantl>
Status: CLOSED FIXED QA Contact: Michael Tremer <michael.tremer>
Severity: - Unknown -    
Priority: - Unknown - CC: peter.mueller, stefan.schantl
Version: 2   
Hardware: unspecified   
OS: Unspecified   
Attachments: C163 Operational System Rule File Sizes
c164/C165 Test System Rule File Sizes
ls -lSr *.rules from c163 operational system
ls -lSr *.rules from c164/c165 test system

Description Charles Brown 2022-03-05 13:07:35 UTC
Talos Subscribed rules on c163 operational system
[root@ipfire bin]# ./getrulecount.sh
  security-ips rules => 17534
  balanced-ips rules => 9008
max-detect-ips rules => 34948
     other ips rules => 10593
-----------------------------------
    Total Rule Count => 43141

Same rule set running on my c164/165 test system I get ...
[root@ipfire bin]# ./getrulecount.sh
  security-ips rules => 4282
  balanced-ips rules => 1102
max-detect-ips rules => 9291
     other ips rules => 15967
-----------------------------------
    Total Rule Count => 23958

Note the short set is from fresh a install of c165 without doing any backup restore
Comment 1 Charles Brown 2022-03-05 13:15:23 UTC
(In reply to Charles Brown from comment #0)
> Talos Subscribed rules on c163 operational system
> [root@ipfire bin]# ./getrulecount.sh
>   security-ips rules => 17534
>   balanced-ips rules => 9008
> max-detect-ips rules => 34948
>      other ips rules => 10593
> -----------------------------------
>     Total Rule Count => 43141
> 
> Same rule set running on my c164/165 test system I get ...
> [root@ipfire bin]# ./getrulecount.sh
>   security-ips rules => 4282
>   balanced-ips rules => 1102
> max-detect-ips rules => 9291
>      other ips rules => 15967
> -----------------------------------
>     Total Rule Count => 23958
> 
> Note the short set is from fresh a install of c165 without doing any backup
> restore

When I got to /var/lib/suricata on my c163 operational system and do:
du -sh .
I get 24M
on the c164/c165 test system, it only shows 21M
Comment 2 Charles Brown 2022-03-05 13:30:21 UTC
Created attachment 1000 [details]
C163 Operational System Rule File Sizes
Comment 3 Charles Brown 2022-03-05 13:31:34 UTC
Created attachment 1001 [details]
c164/C165 Test System Rule File Sizes
Comment 4 Charles Brown 2022-03-05 13:49:23 UTC
Created attachment 1002 [details]
ls -lSr *.rules from c163 operational system
Comment 5 Charles Brown 2022-03-05 13:52:18 UTC
Created attachment 1003 [details]
ls -lSr *.rules from c164/c165 test system
Comment 6 Stefan Schantl 2022-03-05 14:34:00 UTC
Hello Charles,

good find - it seems that the rules tarballs from soucefire (snort.org) contains rulefiles with the same name in different folders.

The old extract logic in c163 merged or appended these files into one bigger one.
The new logic simply overwrites them, because I did not noticed this during development.

Best regards and a big thanks,

-Stefan
Comment 7 Peter Müller 2022-03-05 14:40:45 UTC
Just for my understanding: Is this bug affecting the release of Core Update 164?
Comment 8 Charles Brown 2022-03-05 15:27:12 UTC
(In reply to Peter Müller from comment #7)
> Just for my understanding: Is this bug affecting the release of Core Update
> 164?

I would think this is 'blocking' c164.  Perhaps you should confirm with  Stefan Schantl if this issue is in both c164 and c165.
Comment 9 Stefan Schantl 2022-03-05 15:32:24 UTC
Hello Charles,

I've sent a patch to hopefully fix this issue to the development mailing list.

Could you please test and verify if you have the same amount of rules again?

Thanks in advance,

-Stefan
Comment 10 Stefan Schantl 2022-03-05 15:32:59 UTC
Missed the link to the patch:

https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1-stefan.schantl@ipfire.org/
Comment 11 Charles Brown 2022-03-05 15:55:46 UTC
(In reply to Stefan Schantl from comment #10)
> Missed the link to the patch:
> 
> https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1-
> stefan.schantl@ipfire.org/

Well, that definitely provided more rules -- many more, in fact.
du  -sh on /var/lib/suricata  reports 40M now vs 24M on c163.
More details coming in attachments shortly
Comment 12 Charles Brown 2022-03-05 15:59:15 UTC
(In reply to Charles Brown from comment #11)
> (In reply to Stefan Schantl from comment #10)
> > Missed the link to the patch:
> > 
> > https://patchwork.ipfire.org/project/ipfire/patch/20220305152717.5879-1-
> > stefan.schantl@ipfire.org/
> 
> Well, that definitely provided more rules -- many more, in fact.
> du  -sh on /var/lib/suricata  reports 40M now vs 24M on c163.
> More details coming in attachments shortly

Per earlier post, my c163 ops system shows this for Talos Subscribed
Talos Subscribed rules on c163 operational system
[root@ipfire bin]# ./getrulecount.sh
  security-ips rules => 17534
  balanced-ips rules => 9008
max-detect-ips rules => 34948
     other ips rules => 10593
-----------------------------------
    Total Rule Count => 43141

However, now after applying the patch, I get

  security-ips rules => 19906
  balanced-ips rules => 9292
max-detect-ips rules => 37361
     other ips rules => 23182
-----------------------------------
    Total Rule Count => 58515
Comment 13 Charles Brown 2022-03-05 16:25:30 UTC
It seems I get 3 extra rule files with the c164*/c165 test system -- at least 3 were present before applying your patch.  Here are the file names

subscripted-decoder.rules
subscripted-deleted.rules
subscripted-local.rules
subscripted-preprocessor.rules
Comment 14 Charles Brown 2022-03-05 16:26:52 UTC
(In reply to Charles Brown from comment #13)
> It seems I get 3 extra rule files with the c164*/c165 test system -- at
> least 3 were present before applying your patch.  Here are the file names
> 
> subscripted-decoder.rules
> subscripted-deleted.rules
> subscripted-local.rules
> subscripted-preprocessor.rules

perhaps I can't count -- these are all new to me since c163
Comment 15 Charles Brown 2022-03-05 16:34:12 UTC
Aha, subscripted-deleted.rules would seem to be the culprit leading to the inflated rule count when I do my grepping. It is the one really big file of the 4 (count 'em 4) new files. That one alone accounts 12,128 rules.
Comment 16 Charles Brown 2022-03-05 17:07:31 UTC
Together, those 4 new files account for 12,324 additional rules that are not present in the same ruleset for c163
Comment 17 Charles Brown 2022-03-05 17:15:32 UTC
And there are still an additional 3050 rules in new test system (with your patch) that I haven't resolved with my simple caveman grepping
Comment 18 Stefan Schantl 2022-03-05 17:55:26 UTC
Hello Charles,

thanks for testing and your feedback - nice to hear that the patch fixes the issue and works even better than expected.

@Michael, please backport this also to c164.
Comment 19 Charles Brown 2022-03-05 19:08:19 UTC
FWIW, those 'four' new rule files that were not present in c163 have damn little of interest. One has a single 'security-ips' policy rule, Two have some 'max-detect-ips' policy rules, Three have rules with no policy.
 
/var/lib/suricata/subscripted-decoder.rules
  security-ips rules => 0
  balanced-ips rules => 0
max-detect-ips rules => 104
     other ips rules => 37
-----------------------------------
    Total Rule Count => 141

/var/lib/suricata/subscripted-deleted.rules
  security-ips rules => 0
  balanced-ips rules => 0
max-detect-ips rules => 0
     other ips rules => 12128
-----------------------------------
    Total Rule Count => 12128

/var/lib/suricata/subscripted-local.rules
  security-ips rules => 0
  balanced-ips rules => 0
max-detect-ips rules => 0
     other ips rules => 0
-----------------------------------
    Total Rule Count => 0

/var/lib/suricata/subscripted-preprocessor.rules
  security-ips rules => 1
  balanced-ips rules => 0
max-detect-ips rules => 38
     other ips rules => 17
-----------------------------------
    Total Rule Count => 55
Comment 20 Charles Brown 2022-03-05 19:13:21 UTC
(In reply to Stefan Schantl from comment #18)
> Hello Charles,
> 
> thanks for testing and your feedback - nice to hear that the patch fixes the
> issue and works even better than expected.
> 
> @Michael, please backport this also to c164.

Hi Stefan,

I hope my knit-picking hasn't caused you too much of a kerfuffle for you :-)

Cheers
-cab
Comment 21 Peter Müller 2022-03-05 19:30:11 UTC
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=2b12a010d7ea97996a3b7accaab0316cb71de515

@Michael: Kindly cherry-pick this one into "master" as well.
Comment 22 Charles Brown 2022-03-08 13:11:25 UTC
Per my testing with /master/2022-03-07 18:53:09 +0000-b69659af, this issue is fixed