After upgrading to Core Update 172 from 171, squid-asnbl-helper crashes squid after a few minutes with the following error: Traceback (most recent call last): File "/usr/bin/asnbl-helper.py", line 488, in <module> resolvedasn = resolve_asn(singleip, ASNDB) File "/usr/bin/asnbl-helper.py", line 184, in resolve_asn result = asndb.lookup(str(ipaddr)) Which results in squid parent to detect a too fast restart of the children(s) and to shutdown. squid[-]: Squid Parent: squid-1 process - exited with status 1 squid[-]: Squid Parent: (squid-1) process - started squid[-]: Squid Parent: squid-1 process - exited with status 1 squid[-]: Squid Parent: (squid-1) process - started squid[-]: Squid Parent: squid-1 process - exited with status 1 squid[-]: Squid Parent: (squid-1) process - started squid[-]: Squid Parent: squid-1 process - exited with status 1 squid[-]: Squid Parent: (squid-1) process - started squid[-]: Squid Parent: squid-1 process - exited with status 1 squid[-]: Squid Parent: (squid-1) process - started squid[-]: Squid Parent: squid-1 process - exited with status 1 squid[-]: Squid Parent: squid-1 process - will not be restarted for 3600 seconds due t o repeated, frequent failures squid[-]: Exiting due to repeated, frequent failures Also when fast flux is activated, squid-asnbl-helper is complaining about a missing ASN-Blocklist source: squid-asnbl-helper[-] WARN: No ASNBL configured. This is acceptable as long as this script is configured to do anything, you just have been warned... squid-asnbl-helper[-] INFO: ASN database operational - excellent. Waiting for input... The option for ACTIVE_ASNBLS is empty in /var/ipfire/proxy/asnbl-helper.conf maybe as Spamhaus has closed it's ASN-Blocklist? As a workaround, the whole anomaly detection needs to be deselected for a normal squid behaviour.
Confirmed. => https://lists.ipfire.org/pipermail/development/2022-December/015115.html Thanks for reporting.
This is odd, I haven't seen this error on my machines. Will investigate - perhaps a spillover effect from the libloc update?
(In reply to Peter Müller from comment #2) > This is odd, I haven't seen this error on my machines. Will investigate - > perhaps a spillover effect from the libloc update? Don't know much about it, but when running a location db update I get the following error message: # location --debug update HTTP GET Request to location.ipfire.org URL: https://location.ipfire.org/databases/1/location.db.xz Headers: If-modified-since: Sun, 15 Jan 2023 05:24:18 GMT User-agent: location/0.9.15 HTTP Response: 200 Headers: date: Sun, 15 Jan 2023 07:54:05 GMT last-modified: Sun, 15 Jan 2023 05:26:54 GMT etag: "513f2c-5f246b6f4867e" accept-ranges: bytes content-length: 5324588 x-content-type-options: nosniff x-frame-options: deny referrer-policy: strict-origin x-xss-protection: 1; mode=block content-type: application/x-xz strict-transport-security: max-age=31536000; includeSubDomains; preload connection: close Opening downloaded database at /var/lib/location/tmp3p87n16h Downloaded new database from Sun, 15 Jan 2023 05:24:18 GMT libloc: loc_database_verify: Error verifying the first signature: error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long
Confirmed. Exactly.
https://blog.ipfire.org/post/ipfire-2-27-core-update-173-released
Sorry to report, but this problem isn't fixed completely in Core 173. Shortly after an automatic update of the blacklist, following error shutting down squid completely again: Mar 01 01:27:14 squid-asnbl-helper[32004] WARN: Destination 'IP_ADDRESS_OF_PROXY' resolves to IP addresses 'IP_ADDRESS_OF_PROXY' without corresponding ASN, probably selectively announced Mar 01 01:27:14 squid-asnbl-helper[32004] INFO: Denying access to destination 'IP_ADDRESS_OF_PROXY' due to suspected selective announcements Traceback (most recent call last): File "/usr/bin/asnbl-helper.py", line 493, in <module> resolvedasn = resolve_asn(singleip, ASNDB) File "/usr/bin/asnbl-helper.py", line 193, in resolve_asn if not result.asn: UnboundLocalError: local variable 'result' referenced before assignment 2023/03/01 01:34:23 kid1| WARNING: external_acl_type #Hlpr3 exited ... Mar 01 01:34:24 squid-asnbl-helper[32478] WARN: No ASNBL configured. This is acceptable as long as this script is configured to do anything, you just have been warned... Traceback (most recent call last): File "/usr/bin/asnbl-helper.py", line 493, in <module> resolvedasn = resolve_asn(singleip, ASNDB) File "/usr/bin/asnbl-helper.py", line 193, in resolve_asn if not result.asn: UnboundLocalError: local variable 'result' referenced before assignment ... 2023/03/01 01:34:24 kid1| FATAL: The external_acl_type helpers are crashing too rapidly, need help! So I think, this bug should be reopened until a full fix is available.
After some more debugging and log file analysis, the crash has no dependency to the blacklist update. The helper crashes multiple times every hour the last 24 hours, sometimes so often that it shutdown squid completely. After the deactivation of the fast flux detection, squid is running stable without a problem. And I already checked that /usr/bin/asnbl-helper.py is identical to V0.2.4. So the latest available patch was correctly installed with the update.
Thank you for reporting back. I'm getting the feeling that we are dealing with multiple bugs in squid-asnbl here, and some of these are triggered only under certain circumstances. Thus far, I was unsuccessful trying to spot or reproduce this behaviour on any IPFire installations I have access to. Having updated to Core Update 173, does libloc work properly again? Precisely, what does the output of "location --debug update" read?
Jm2C: ... root@ipfire: /var/log/squid # location --debug update Already on the latest version ... On Core 173: no seen problems here. 'squid' is running (v5.8).
Created attachment 1140 [details] Make squid-asnbl compatible with python3 This patch makes the functions 'resolve_asn' and 'asndb_response_tests' fully compatible with python3 and should resolve their 'UnboundLocalError' errors.
(In reply to Matthias Fischer from comment #9) > On Core 173: no seen problems here. 'squid' is running (v5.8). What's the expected squid version for CU173? Is it v5.7 or v5.8? # squid --version Squid Cache: Version 5.7 Service Name: squid ... Location db reports also: "Already on the latest version". Nethertheless, even when it was hard to deal with this bug, I found the problem and tested the patch for about 32 hours right now to make squid-asnbl-helper fully compatible to python3. Also the patch deals with how V3 sometimes falsely "recognizes" local result variables as global. Benefit about the changes will be a narrowed function definition, but I don't think there will be a measurable lower memory consumption or speed improvement. For all who want more details what this patch do and why this error is so hard to reproduce: First I discovered that the helper only sometimes throwing the error and quits even for the same values and queries. Also the timespan until the error happens was quite different for every restart of squid (minutes to hours). And it does not depend on the traffic on the proxy, even one connection could cause a crash while ten or hundrets won't. After a few days of testing different solutions and done a lot of debugging, redesigning the function did not fully solve the problem. Such standard things like checking the result variable for NULL (or it's equivalent "is None" in python) before evaluating it's subfunction produces the exact same error message. But with that knowledge it more and more turns out that python3 sometimes 'detects' the local return variable if it was a misused global. So for a full fix, the return variable also has to be initialized that python3 won't detect it's usage as an 'UnboundLocalError' to succesfully fix this bug.
(In reply to Nicolas Pöhlmann from comment #11) > > What's the expected squid version for CU173? Is it v5.7 or v5.8? > > # squid --version > Squid Cache: Version 5.7 > Service Name: squid > ... > CU173 is running with 5.7 5.8 has been submitted by Matthias into what will become CU174.
Created attachment 1141 [details] Make squid-asnbl compatible with python3 (V2) Slightly updated the patch. V2 removed the global keyword as it's not needed anymore. The patch was tested and running 10 days without any problem.
I will pick this up and submit a patch into the dev mailing list as just having it in Bugzilla won't change anything. I would be good if you could also flag this up as an issue to the git site for the squid-asnbl package so they can fix it at source.
Submission patch has been created and submitted to the dev mailing list and to patchwork. https://lists.ipfire.org/pipermail/development/2023-March/015692.html https://patchwork.ipfire.org/project/ipfire/patch/20230322182852.3449030-1-adolf.belka@ipfire.org/