As stated in https://community.ipfire.org/t/invalid-header-block-at-offset/11321 there is an revolving error message in /var/log/error.log when updating the emerginthreats.net ruleset. Found in /var/log/httpd/error.log: [core:notice] [pid 3647:tid 130990554702016] AH00094: Command line: '/usr/sbin/httpd' Invalid header block at offset unknown at /var/ipfire/ids-functions.pl line 550. Invalid header block at offset unknown at /var/ipfire/ids-functions.pl line 550. ... ... ... 1_12_09, updated_at 2024_03_18;): checksum error at /var/ipfire/ids-functions.pl line 550. This happens only when ET ruleset was updated and a new update is existing for the respective ruleset. Not when just activating the existing or new rulesets. Looking in older (Core-update 183) error.logs there is no error of this kind to found. However there doesn’t seem to be any error with these rules. They are firing as expected.
@Stefan: Could you please have a look at what is going wrong here?
*** Bug 13638 has been marked as a duplicate of this bug. ***
I have narrowed this down to the Archive::Tar perl module which seems to be unhappy with the format of the ET tarball. On Debian, however, the problem does not arise.
Hello Waynie, Michael and Adolf, I'm very sorry because I'm unable to reproduce any of this error messages. I looked into my http error log and there isn't any of those messages. Also when I execute the "update-ids-ruleset" script on the console none of this lines got displayed or will be logged. Once a day, the emerging ruleset automatically got update on my firewall without any messages. I don't have any plan why there could be such a difference... -Stefan
Created attachment 1492 [details] Test Script I can reproduce this with this script. Just download the tarball and run it and you should see these messages on IPFire but not on Debian. It is either a bug in the Archive::Tar module, or the module is reading garbage from the decompression.
When downloading the ruleset with wget and using your script I'm finally also able to reproduce this error messages.
Checking CPAN, our currently used and shipped version of Archive::Tar is very outdated. Should I do the update and send a patch or is someone already working on it?
(In reply to Stefan Schantl from comment #7) > Checking CPAN, our currently used and shipped version of Archive::Tar is > very outdated. > > Should I do the update and send a patch or is someone already working on it? Please update the package. I am not working on this.
Good Evening, Bad News, I've updated the module to the latest available Version, moved the files to my IPFire test machine and run the test script again - same result, the messages about the invalid header block still appear. When reading the docs (https://perldoc.perl.org/Archive::Tar.txt) the following is documented about this messages: ### according to the posix spec, the last 12 bytes of the header are ### null bytes, to pad it to a 512 byte block. That means if these ### bytes are NOT null bytes, it's a corrupt header. See: ### www.koders.com/c/fidCE473AD3D9F835D690259D60AD5654591D91D5BA.aspx ### line 111 { my $nulls = join '', "\0" x 12; unless( $nulls eq substr( $chunk, 500, 12 ) ) { $self->_error( qq[Invalid header block at offset $offset] ); next LOOP; } }
I have debugged this somewhat further and found out that the problem seems to be a file with zero length. After that, the module does not jump to the correct place for the next header and therefore logs these messages. That happens for as long as the module is reading the payload of the next file and finds the next file. We therefore fail to extract one file which is "rules/threatview_CS_c2.rules". I cannot find anything in the perl bug tracker, the functioning version in Debian seems to be the same file that we are using. I don't think this will be difficult to fix, I am just not sure where to look for this problem. An alternative could be that we simply call the tar command, but I don't want to commit to any larger rewrites of our code just yet.
I've rebased the perl code to use a simple perl binding for libarchive to extract the tar archive. This works pretty well, fixes the issue, speeds up the extract process, allows more archive types.... https://patchwork.ipfire.org/project/ipfire/patch/20240330113530.1710702-5-stefan.schantl@ipfire.org/
(In reply to Stefan Schantl from comment #11) > I've rebased the perl code to use a simple perl binding for libarchive to > extract the tar archive. > > This works pretty well, fixes the issue, speeds up the extract process, > allows more archive types.... > > https://patchwork.ipfire.org/project/ipfire/patch/20240330113530.1710702-5- > stefan.schantl@ipfire.org/ Thank you. I merged this. Let's give this some good testing to be sure that everything works as expected.
Has been merged into CU185 Testing
Tested out on CU185 Testing with latest merges. Had Emerging Threats ruleset with an update date of 3rd March. Ran the update-ids-rulesets command and the emerging threats ruleset was updated to 5th April and there were no error messages given out at all. Checked the httpd/error_log file and there were no Invalid header block at offset.... messages in the log anymore. The above verifies that the bug has been fixed.
https://www.ipfire.org/blog/ipfire-2-29-core-update-185-released