Bug 12788

Changed version number from 3 to 2 as the bug is related to IPFire 2.x

I did a full install of Core 164 using the iso at the following link.
https://nightly.ipfire.org/core164/2022-02-26%2014:19:45%20+0000-6e2c8f48/x86_64/

After restoring my backup and trying to look at the rules set for the emerging threats that I had listed I could not see any rules listed, the same as Rejjy.

I then installed Abuse.ch and could then see the rule listed when I pressed Customise rule set. I selected it and applied it and then tried to view it again and as with Rejjy I had a blank section showing nothing.

Doing a core update with pakfire from 163 to 164 Testing gave no problems at all.

What I couldn't determine is if the 164 iso in the above link is up to date as the version that is upgraded by pakfire.

Hello thanks for reporting this issue,

Unfortunately I'm unable to reproduce it.

What it did:

1.) Downloaded the linked core 164 iso which Adolf posted.
2.) Installed in a VM.
3.) Bring the system online and login into the WUI.
4.) Add "Emergingthreads.net Community Ruleset" as provider.
5.) Select customize the ruleset and enable some files.
6.) Press "Update" to take the changes effect.
7.) After the page allowed interaction again, I clicked on customize the ruleset.

Everything is displayed correctly, the enabled ruleset files are enabled etc.

I did a second test and opened a category and enabled / disabled various rules inside it and pressed "Update" again.

Once again, after the page is unlocked again, I clicked to customize the ruleset again, and the page still is displayed correctly. All enabled ruleset categories are enabled and the enabled/disabled rules inside the category are fine.

If possible, please post the output of the "/var/log/httpd/error_log" to get more details.

Thanks in advance,

-Stefan

Hi Stefan,

Between your steps 3 & 4 I restored a backup from my Core Update 163 vm whci had the Emerging threats ruleset working. From your sequence that is the only difference that I can see between what we each did.

I am opening my vm systems back up and will look at the error_log and see what it says.

I will also try going from the core update 164 iso and directly install the emerging threats ruleset and see if that then works for me.

When the screen shows empty for the customize fileset then there is no error message in the httpd/error_log.

However when I delete a provider I get the following two error lines.

oisf rules
2/3/2022 -- 18:57:24 - <Error> - [ERRCODE: SC_ERR_FOPEN(44)] - Failed to open configuration include file /var/ipfire/suricata/suricata-oisf_trafficid-used-rulefiles.yaml: No such file or directory
2/3/2022 -- 18:57:24 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to include configuration file /var/ipfire/suricata/suricata-used-providers.yaml

emerging threats rules
2/3/2022 -- 18:57:42 - <Error> - [ERRCODE: SC_ERR_FOPEN(44)] - Failed to open configuration include file /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml: No such file or directory
2/3/2022 -- 18:57:42 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to include configuration file /var/ipfire/suricata/suricata-used-providers.yaml

sslbl blacklist
2/3/2022 -- 18:57:58 - <Error> - [ERRCODE: SC_ERR_FOPEN(44)] - Failed to open configuration include file /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml: No such file or directory
2/3/2022 -- 18:57:58 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to include configuration file /var/ipfire/suricata/suricata-used-providers.yaml

I had a look at the files mentioned in the error messages after I had installed three providers.

suricata-used-providers.yaml contains the three providers as include lines

suricata-emerging-used-rulefiles.yaml was empty apart from the standard content. The same for suricata-sslbl_blacklist-used-rulefiles.yaml

The third one that I added last, suricata-oisf_trafficid-used-rulefiles.yaml contained a line for the rules selected.

------------------------------
suricata-emerging-used-rulefiles.yaml

%YAML 1.1
---

#Autogenerated file. Any custom changes will be overwritten!
------------------------------
suricata-sslbl_blacklist-used-rulefiles.yaml

%YAML 1.1
---

#Autogenerated file. Any custom changes will be overwritten!
------------------------------
suricata-oisf_trafficid-used-rulefiles.yaml

%YAML 1.1
---

#Autogenerated file. Any custom changes will be overwritten!
 - oisf_trafficid-ruleset.rules
------------------------------

I installed from the same Core164 iso and went directly to adding emerging threats, abuse.ch and oisf rulesets.

I was able to add all three and continue to see the selection  and change things by pressing the customize ruleset button. So everything working as expected.

Then I restored my core update 163 backup and IPS only had the emerging threats ruleset selected but pressing customize ruleset gave a blank screen. Added abuse.ch and oisf and was able to see just these rulesets. Selected them and then tried to use customize rulesets again but again got a blank screen.

So the problem appears to be related to the restore from a backup from the single provider IPS structure.

Created attachment 996 [details]
HTTPD Error_log file as requested

Attached is my HTTPD Error_log file as requested.

This is from the httpd/error_log:

[Wed Mar 02 13:22:51.398738 2022] [core:error] [pid 2870:tid 136107555538496] (70007)The timeout specified has expired: [client 10.0.2.11:63504] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: https://fw-at4b.at4b.net:444/

A big thanks for testing and providing such a huge amount of feedback.

In Adolf's case it seems to only be a problem when dealing with backups and not a general one. This may could be the reason why I was not able to reproduce this issue.

In Rejjy's case I only can read from the log that some files are missing, which defenitely will be created by the WUI when a provider will be added.

Please post some more details and steps what you are doing to fall into this issue. Also please post a directory listing (ls -la) of your "/var/ipfire/suricata/" directory after you got the white page.

Additionaly, please modify your "/srv/web/ipfire/cgi-bin/ids.cgi" file and un-comment (remove the tailing "#") the lines 26 and 27

#use warnings;
#use CGI::Carp 'fatalsToBrowser'

in script. This will display the error instead of just providing a white page.

Freds's error log sadly does not provide any usefull to solve this issue. If you are also affected please also post steps what your are doing, the directory listing and additionally the output after the WUI file modification.

Thanks in advance,

-Stefan

I am not able to reproduce this either. Since this is now blocking the release of the Core Update, I would appreciate if everybody who can would help us finding and resolving the issue as fast as possible.

If we are not able to do this by tomorrow, I would prefer to revert the IPS changes and move them into the next Core Update so that we don't have any pressure and more time to find the best solution.

I reproduced issue -- the blank area where the rule files should be per the following steps ...
.
1) Clean install core164/2022-02-26 14:19:45 +0000-6e2c8f48/x86_64/ipfire-2.27.x86_64-full-core164.iso
2) Edited "/srv/web/ipfire/cgi-bin/ids.cgi" uncommenting
  #use warnings;
  #use CGI::Carp 'fatalsToBrowser'  
3) Restored backup from CU-163 with Talos Subcribed ruleset used
4) Rebooted
5) Went to Intrusion Prevention System page and clicked on 'Customize ruleset'
 -- the page was empty where the rule files should have been displayed.  
 -- No error / debug was shown on the display

Here's the list of files in "/var/ipfire/suricata/"
-rw-r--r-- 1 root   root       0 Mar  3 09:56 ignored
-rw-r--r-- 1 root   root   20916 Apr 17  2020 oinkmaster.conf
-rw-r--r-- 1 root   root      77 Jul 11  2020 oinkmaster-enabled-sids.conf.bkup
-rw-r--r-- 1 nobody nobody   306 Feb 28 10:18 oinkmaster-modify-sids.conf
-rw-r--r-- 1 nobody nobody    71 Mar  3 09:56 oinkmaster-provider-includes.conf
-rw-r--r-- 1 nobody nobody    32 Mar  3 09:56 oinkmaster-subscripted-modified-sids.conf
-rw-r--r-- 1 nobody nobody    71 Mar  3 09:56 providers-settings
-rw-r--r-- 1 root   root    5451 Feb 26 14:09 ruleset-sources
-rw-r--r-- 1 nobody nobody    97 Mar  3 09:56 settings
-rw-rw-r-- 1 nobody nobody   829 Mar  3 09:56 suricata-default-rules.yaml
-rw-r--r-- 1 nobody nobody   219 Mar  3 09:58 suricata-dns-servers.yaml
-rw-r--r-- 1 nobody nobody   137 Mar  3 09:58 suricata-homenet.yaml
-rw-r--r-- 1 nobody nobody   102 Feb 28 10:18 suricata-http-ports.yaml
-rw-r--r-- 1 nobody nobody    76 Mar  3 09:56 suricata-subscripted-used-rulefiles.yaml
-rw-r--r-- 1 nobody nobody   147 Mar  3 09:56 suricata-used-providers.yaml

Additional info from last comment / test 

Just noticed, the  "suricata-subscripted-used-rulefiles.yam' did not contain any rule files.  However I have many selected on my CU 163 box where the backup was made.

Hello Charles,

a big thanks for join the testing and giving such a detailed report.

As I can read you also restored a backup and ran into this issue, so I would assume, that it only appears when a backup got restored.

To clarify the empty page:

Does this mean the page got rendered correctly (Header menu, buttons, footer) and simply does not contain any kind of rules, or did you entirely get a blank (white) page?

I'll try to reproduce this again and restore a backup.

Thanks in advance,

-Stefan

The page looks fine other than the missing rulefiles.

Also, I took a hack at copying my ruleset selections from the backup file into suricata-subscripted-used-rulefiles.yam and rebooted.  That also resulted in the blank section of the screen where I would expect the rule file selections

Just noticed there are no rule files in /var/lib/suricata

Can you please post a directory listing of your "/var/tmp/" directory.

I think the tarball for the rules is missing or not renamed correctly.

Created attachment 997 [details]
update-ids-ruleset errors

Here's the listing of /var/temp ...
total 109220
-rw------- 1 nobody nobody 94339819 Mar  3 11:06 idsrules-subscripted.tar.gz
-rw------- 1 nobody nobody 17498112 Mar  3 11:00 _m2rSZHEkf.tmp
-rw-r--r-- 1 root   root          0 Mar  3 11:07 var-temp.log

Also, I manually rant update-ids-ruleset at the console and had a string of errors

Could not set uid/gid on '/tmp/ids_tmp/rules/subscripted-finger.rules' at /var/ipfire/ids-functions.pl line 576.
Could not set uid/gid on '/tmp/ids_tmp/rules/subscripted-file-executable.rules' at /var/ipfire/ids-functions.pl line 576.
Could not set uid/gid on '/tmp/ids_tmp/rules/subscripted-server-samba.rules' at /var/ipfire/ids-functions.pl line 576.
Could not set uid/gid on '/tmp/ids_tmp/rules/subscripted-server-oracle.rules' at /var/ipfire/ids-functions.pl line 576.
Could not set uid/gid on '/tmp/ids_tmp/rules/subscripted-pua-other.rules' at /var/ipfire/ids-functions.pl line 576.

... see attached file

So, now I deleted my one and only provider selection and re-added it.
That seems to have brought things into a workable state -- the rulefiles are now showing.  It seems that after restore of backup from 163, things are in a broken state until the previous IPS settings are removed and then manually re-entered.

Ugh, I spoke too soon.  After 'successfully' selecting a couple of rulefiles, I thought it went okay.  When I re-selected 'Customize ruleset' the screen section for rule files was again 'Blank' ... and /var/lib/suricata was devoid of rule files

So what happens ... If I delete my one and only provider, reboot, and then add back my Talos Subscribed guy, It looks like things are working.  I then select a rule file with 'Customize rule set' and immediately the rule files in /var/lib/suricata disappear.  A subsequent click on 'Customize rule set' then gives the 'Blank' rule files condition

The "Could not set uid/gid on *" errors are an aesthetic issue and can be ignored at the moment. I recently have sent a patch to the ML to fix them.

Okay, the one key consistency here is: after doing restore from backup file, the rule files in /var/lib/suricata keep getting blown away, hence the blank rule file list on 'Customize ruleset' page. Even after getting properly downloaded at placed in /var/lib/suricata 'once' when deleting / re-adding my rule provider, then selecting a rule file and applying it seems to cause the files in /var/lib/suricata to get blown away.

Hello Charles,

I setup a c163 in an virtual environment, configured the IDS, created a backup and restored it on a freshly installed c164 virtual machine.

I exactly got the same issue as you reported and was able to nail it down:

The main configuration file of oinkmaster (a helper tool to alter the ruleset) will be replaced by the old one from c163 which is now incompatible and refuses the tool to run.

As a result of this, the ruleset cannot be customized anymore / the directory which should contain them is empty.

The good news is, after I have manually edited this file to the new and correct values everything worked well again.

So we have to find a way to not extract this file, while restoring a backup.

Sounds like a little pre and post processing in the 'restore' implementation

(In reply to Charles Brown from comment #25)
> Sounds like a little pre and post processing in the 'restore' implementation

Saving off the 'good' oinkmaster main configuration file before the backup restore extraction happens and then restoring it (if necessary) after the backup restore does its extraction thing

(In reply to Charles Brown from comment #26)
> (In reply to Charles Brown from comment #25)
> > Sounds like a little pre and post processing in the 'restore' implementation
> 
> Saving off the 'good' oinkmaster main configuration file before the backup
> restore extraction happens and then restoring it (if necessary) after the
> backup restore does its extraction thing

> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=026935a1375551f833997a95f63898112527a0f8

I just pushed this into next. Please check the latest nightly build as soon as it is done compiling (in about 4 hours) and let me know if restoring a backup on this doesn't break the IPS any more.

If that is confirmed, I will backport the changes into c164 and prepare for a release.

Thank you all for helping investigate this.

(In reply to Michael Tremer from comment #27)
> (In reply to Charles Brown from comment #26)
> > (In reply to Charles Brown from comment #25)
> > > Sounds like a little pre and post processing in the 'restore' implementation
> > 
> > Saving off the 'good' oinkmaster main configuration file before the backup
> > restore extraction happens and then restoring it (if necessary) after the
> > backup restore does its extraction thing
> 
> > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=026935a1375551f833997a95f63898112527a0f8
> 
> I just pushed this into next. Please check the latest nightly build as soon
> as it is done compiling (in about 4 hours) and let me know if restoring a
> backup on this doesn't break the IPS any more.
> 
> If that is confirmed, I will backport the changes into c164 and prepare for
> a release.
> 
> Thank you all for helping investigate this.

If this patch fixes the backup restore issue, there are still two other patches from Stefan Schantl that would need to go into c164 ...
https://patchwork.ipfire.org/project/ipfire/patch/20220303044943.3678-1-stefan.schantl@ipfire.org/
https://patchwork.ipfire.org/project/ipfire/patch/20220303185559.3551-1-stefan.schantl@ipfire.org/

I have tested out the CU165 next latest iso version with the backup exclude patch and I can confirm that this now works.

I installed the CU165 version, added some providers. Then I restored my CU163 version and then I only had the Emerging Threats provider (because CU163 only had one provider) but I could press customize ruleset and see the whole ruleset and it had the rules selected that I had used on CU163. I could then add additional providers and each time I was able to see the combined list of rules.

Working well and as expected now.

Adolf, where can i find the updated cu 165 .iso so i can test ?

Never mind. I managed to dig my way around and found it.

Two interesting 'features' with next/c165: 
1) the rulefile selections from my c163 backup did not make the trip. However this is sort of understandable with the significant changes supporting multiple providers; 
2) the Add Whitelisted Hosts feature seems completely broken -- when attempting to add a host, the only thing you get is a completely 'blank' screen -- no headers, nothing.

Could one of you others testing with c165 see if the Whitelisted Hosts thing is working for you?

(In reply to Charles Brown from comment #32)
> 1) the rulefile selections from my c163 backup did not make the trip.

In my test I had emerging threats in CU165 with two rules set and after my backup restore I had emerging threats with all of the rules that IO has selected on my CU163 setup. So for me the rulefile selections did make the trip.

Looks like there is still some other subtlety around.

(In reply to Charles Brown from comment #32)
> 2) the Add Whitelisted Hosts feature seems completely broken -- when
> attempting to add a host, the only thing you get is a completely 'blank'
> screen -- no headers, nothing.
> 
> Could one of you others testing with c165 see if the Whitelisted Hosts thing
> is working for you?

I just tried it with CU164 without restore (so working for showing rulesets) CU165 with the fix and CU164 after a restore (so not working for showing rulesets).

In all three cases the whitelisting worked.

I added a host IP address and a remark and pressed add and got a line in the whitelisting section. In all cases added two hosts and both ended up listed.

I confirm Aldolf's findings in regards to the cu 165.
(Core Update 165 Development Build: next/41915357).
I preformed a full install from the .iso, played around with the ruleset (note I don't recall seeing a default provider, but that may be a good thing), added providers, modified ruleset etc and all worked well without the restore.

I did a restore from a Dec2021 backup.  All worked as expected !!!

I also did the curl http://testmynids.org/uid/index.html test and it did report back in the log, so IPS is working.

May I suggest a slight cosmetic change to the WUI of the IPS.
The 'Automatic Rule Update' dropdown for 'disabled' should be removed as the auto updates for each provider has it's own checkbox now, so it is redundant i think.

The 'Automatic Rule Update' should then be renamed as 'Automatic Rule Update Interval' , and near it some text indicating that a forced manual update function is located in the 'Edit' function of the specific provider.  
As it stands, it is rather confusing which is why had brought it up in the community forum [Manual Update missing on core 164-TEST], feb28-2022
But that is just my opinion :) 


Either way, good job guys !   Looking good.

I will also try out the whitelist feature

Just did another fresh install of c165. I don't know what went wrong on the first pass but this go round, things do indeed look like they work as expected.
Time for another iteration ...

With reference to Charles comment 32

1-) I too lost my original Emergingthreats.net Community Rules from the backup (they came back unchecked).  Not a show-stopper in my opinion.

2-)
So i tested the whitelist using a internet address.
It did proccess the IP, and it did let traffic go by, but only partially.

 -- Without the white list ...
Date:	03/04 17:02:08 	Name:	ET HUNTING Suspicious TLS SNI Request for Possible COVID-19 Domain M1
Priority:	2 	Type:	Potentially Bad Traffic
IP info: 	192.168.68.181:45092 -> 108.159.227.82:443
References:	none found	SID: 	2029707

-- After white listing 108.159.227.82
Date:	03/04 17:04:27 	Name:	ET HUNTING Suspicious Domain Request for Possible COVID-19 Domain M1
Priority:	2 	Type:	Potentially Bad Traffic
IP info: 	192.168.68.181:39165 -> 192.168.68.1:53
References:	none found	SID: 	2029709

The 108.159.227.82 was the IP addr which was originally blocked and logged. i whitelisted it, and it no longer reported in the log on the second attempt (i did not have to reboot).

However, the second attempt to access the same site seems to have been blocked by local DNS this time. Likely separate issue.

Bottom line is that i'm not seeing the blank WUI that Charles was seeing.

Created attachment 998 [details]
File list from /var/ipfire/suricat -- showing ownershp and permissions

Okay, on this iteration the rule files settings did indeed get restored as desired.  But the Whitelisted Hosts problem returned.  That is, attempts to add a whitelist entry resulted in getting nothing but a completely blank page.  I did a little digging and found this in httpd log: 

“error_log:Unable to write to file /var/ipfire/suricata/ignored at /var/ipfire/general-functions.pl line 902. 

See the attachment showing list of files in “var/ipfire/suricata with ownership and permissions

(In reply to Charles Brown from comment #39)
> Created attachment 998 [details]
> File list from /var/ipfire/suricat -- showing ownershp and permissions
> 
> Okay, on this iteration the rule files settings did indeed get restored as
> desired.  But the Whitelisted Hosts problem returned.  That is, attempts to
> add a whitelist entry resulted in getting nothing but a completely blank
> page.  I did a little digging and found this in httpd log: 
> 
> “error_log:Unable to write to file /var/ipfire/suricata/ignored at
> /var/ipfire/general-functions.pl line 902. 
> 
> See the attachment showing list of files in “var/ipfire/suricata with
> ownership and permissions

Aha, perhaps I had stopped suricata at the console to kill a stream of junk alerts from my upstream router.  Then restarting it at the console wrecked the file ownership and permissions for /var/ipfire/suricata/ignored so it couldn't be updated from the WUI.

Not related to this IPS issue, but related to recovery issue from older backup to cu 165.

Noticed that in the 'Files Option', the two newly added features 
'Log dropped spoofed packets and marsians'
and
'Drop packets from and to hostile networks (listed at Spamhaus DROP, etc.)'

are neither ON or OFF, after doing a backup restore from older core (159 or 161).

This makes sense as this is new to 164 and 165, and likely overwritten during restore.  However, should there maybe be a check to force a default if no value is given ?  I suspect this could produce issues later if some code is looking for a value.

Should I report this as a separate bug ???

(In reply to Charles Brown from comment #40)
> (In reply to Charles Brown from comment #39)
> > Created attachment 998 [details]
> > File list from /var/ipfire/suricat -- showing ownershp and permissions
> > 
> > Okay, on this iteration the rule files settings did indeed get restored as
> > desired.  But the Whitelisted Hosts problem returned.  That is, attempts to
> > add a whitelist entry resulted in getting nothing but a completely blank
> > page.  I did a little digging and found this in httpd log: 
> > 
> > “error_log:Unable to write to file /var/ipfire/suricata/ignored at
> > /var/ipfire/general-functions.pl line 902. 
> > 
> > See the attachment showing list of files in “var/ipfire/suricata with
> > ownership and permissions
> 
> Aha, perhaps I had stopped suricata at the console to kill a stream of junk
> alerts from my upstream router.  Then restarting it at the console wrecked
> the file ownership and permissions for /var/ipfire/suricata/ignored so it
> couldn't be updated from the WUI.

Yep, I just deleted the 'ignored' file and then I was able to go add a whitelisted hosts entry from the WUI

Created attachment 999 [details]
File list from /var/ipfire/suricat -- showing ownershp and permissions

Aside from me breaking things by stopping and starting suricata at the console, it looks like it works well.

(In reply to Rejjy_S from comment #41)
> Not related to this IPS issue, but related to recovery issue from older
> backup to cu 165.
> 
> Noticed that in the 'Files Option', the two newly added features 
> 'Log dropped spoofed packets and marsians'
> and
> 'Drop packets from and to hostile networks (listed at Spamhaus DROP, etc.)'
> 
> are neither ON or OFF, after doing a backup restore from older core (159 or
> 161).
> 
> This makes sense as this is new to 164 and 165, and likely overwritten
> during restore.  However, should there maybe be a check to force a default
> if no value is given ?  I suspect this could produce issues later if some
> code is looking for a value.
> 
> Should I report this as a separate bug ???


Bug # 12791 submitting

I just noticed the number of rules present on my c1654/c165 system is much smaller than the same ruleset showing on my c163 system.

c163 system rules using Talos Subscribed has 43,141 rules
[root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert " /var/lib/suricata/*.rules | wc -l
43141

c164/c165 system rules using Talos Subscribed has only 23,954 rules
[root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert " /var/lib/suricata/*.rules | wc -l
23954

That would be 19,187 fewer rules for Talos Suscribed rule set on the c164/c165 system.

(In reply to Charles Brown from comment #46)
> I just noticed the number of rules present on my c1654/c165 system is much
> smaller than the same ruleset showing on my c163 system.
> 
> c163 system rules using Talos Subscribed has 43,141 rules
> [root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert "
> /var/lib/suricata/*.rules | wc -l
> 43141
> 
> c164/c165 system rules using Talos Subscribed has only 23,954 rules
> [root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert "
> /var/lib/suricata/*.rules | wc -l
> 23954
> 
> That would be 19,187 fewer rules for Talos Suscribed rule set on the
> c164/c165 system.

Talos Subscribed rules on c163 operational system
[root@ipfire bin]# ./getrulecount.sh
  security-ips rules => 17534
  balanced-ips rules => 9008
max-detect-ips rules => 34948
     other ips rules => 10593
-----------------------------------
    Total Rule Count => 43141

Same rule set running on my c164/165 test system I get ...
[root@ipfire bin]# ./getrulecount.sh
  security-ips rules => 4282
  balanced-ips rules => 1102
max-detect-ips rules => 9291
     other ips rules => 15967
-----------------------------------
    Total Rule Count => 23958

(In reply to Charles Brown from comment #47)
> (In reply to Charles Brown from comment #46)
> > I just noticed the number of rules present on my c1654/c165 system is much
> > smaller than the same ruleset showing on my c163 system.
> > 
> > c163 system rules using Talos Subscribed has 43,141 rules
> > [root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert "
> > /var/lib/suricata/*.rules | wc -l
> > 43141
> > 
> > c164/c165 system rules using Talos Subscribed has only 23,954 rules
> > [root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert "
> > /var/lib/suricata/*.rules | wc -l
> > 23954
> > 
> > That would be 19,187 fewer rules for Talos Suscribed rule set on the
> > c164/c165 system.
> 
> Talos Subscribed rules on c163 operational system
> [root@ipfire bin]# ./getrulecount.sh
>   security-ips rules => 17534
>   balanced-ips rules => 9008
> max-detect-ips rules => 34948
>      other ips rules => 10593
> -----------------------------------
>     Total Rule Count => 43141
> 
> Same rule set running on my c164/165 test system I get ...
> [root@ipfire bin]# ./getrulecount.sh
>   security-ips rules => 4282
>   balanced-ips rules => 1102
> max-detect-ips rules => 9291
>      other ips rules => 15967
> -----------------------------------
>     Total Rule Count => 23958


This issue is not being driven by the restore of the oinkmaster.conf root cause so I would suggest raising it as a separate bug.

(In reply to Adolf Belka from comment #48)
> (In reply to Charles Brown from comment #47)
> > (In reply to Charles Brown from comment #46)
> > > I just noticed the number of rules present on my c1654/c165 system is much
> > > smaller than the same ruleset showing on my c163 system.
> > > 
> > > c163 system rules using Talos Subscribed has 43,141 rules
> > > [root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert "
> > > /var/lib/suricata/*.rules | wc -l
> > > 43141
> > > 
> > > c164/c165 system rules using Talos Subscribed has only 23,954 rules
> > > [root@ipfire ~]# egrep "^drop |^alert |^#drop |^#alert "
> > > /var/lib/suricata/*.rules | wc -l
> > > 23954
> > > 
> > > That would be 19,187 fewer rules for Talos Suscribed rule set on the
> > > c164/c165 system.
> > 
> > Talos Subscribed rules on c163 operational system
> > [root@ipfire bin]# ./getrulecount.sh
> >   security-ips rules => 17534
> >   balanced-ips rules => 9008
> > max-detect-ips rules => 34948
> >      other ips rules => 10593
> > -----------------------------------
> >     Total Rule Count => 43141
> > 
> > Same rule set running on my c164/165 test system I get ...
> > [root@ipfire bin]# ./getrulecount.sh
> >   security-ips rules => 4282
> >   balanced-ips rules => 1102
> > max-detect-ips rules => 9291
> >      other ips rules => 15967
> > -----------------------------------
> >     Total Rule Count => 23958
> 
> 
> This issue is not being driven by the restore of the oinkmaster.conf root
> cause so I would suggest raising it as a separate bug.

https://bugzilla.ipfire.org/show_bug.cgi?id=12792 submitted

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=a956712e75643a4581da8246cef4135a31660746
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=8353e28ad271c0c8d604dc147b4ce0ff399c1530

Bumping to MODIFIED.

@Michael: Please cherry-pick these two and https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=40034794498e0952e1407c82b9a3f526a8fe8305 into "master" so the changes will become effective for Core Update 164.

Patch to fix the permission issue after restoring a backup has been sent to the mailing list:

https://patchwork.ipfire.org/project/ipfire/patch/20220305175310.3036-1-stefan.schantl@ipfire.org/

@Michael, please merge to c164.

(In reply to Stefan Schantl from comment #51)
> @Michael, please merge to c164.

I hope I got them all. The build is already running for both next and master.

Please confirm if this bug is being fixed in the image tomorrow. If so, I would like to start the final build for c164 and *finally* release it this week.

I've just finished testing 164 master/b69659af, full install from .iso
There were no providers defaulted, so I think this is good.

I did a restore from 161 backup and all rules that I had set for 
"Emergingthreads.net Community Ruleset" were all ok.
Forced update of the ruleset .. still ok.

Added provider Abuse.ch SSLBL Blacklist Rules. 
Still all ok when doing a 'Customize Ruleset'

So looks like all in order for this specific bug.

However, appears that the patch that Stefan submitted on my bug 12791 didn't make it to this build, as the restore set these 3 options back to a blank no-default value.

I have just completed a test of the same version as Rejjy_S specified.

Everything also worked for me as I would expect it to. I was able to successfully restore a CU163 backup into the CU164 version. All the enabled rules can be seen and modified as required.

This confirms the fix for this bug for me.

Per my testing with /master/2022-03-07 18:53:09 +0000-b69659af, this issue is fixed

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=da3611b2767298e3f300b12b6ae03958a193c871
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=85b1d83b2a6fe2beb8169f3e810e915c4ad54036

Bumping this to VERIFIED (see: https://wiki.ipfire.org/devel/bugzilla/workflow) b/c of positive testing feedback. Excellent - thanks very much for that! :-)