Bug 13236 - Anomaly detection denies connection to Amazon, Intel etc.
Summary: Anomaly detection denies connection to Amazon, Intel etc.
Status: CLOSED FIXED
Alias: None
Product: IPFire
Classification: Unclassified
Component: --- (show other bugs)
Version: 2
Hardware: unspecified Unspecified
: Will only affect a few users Major Usability
Assignee: Peter Müller
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-08-04 02:32 UTC by Nicolas Pöhlmann
Modified: 2024-03-25 10:32 UTC (History)
3 users (show)

See Also:


Attachments
squid logs showing the errors accessing kagi.com (1.69 KB, text/plain)
2023-08-11 15:13 UTC, cfusco
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicolas Pöhlmann 2023-08-04 02:32:51 UTC
After today's update of the location database, if the option "Deny access to destinations hosted on selectively announced networks" is enabled, at least the following domains are blocked by squid:

*.amazon.com
*.amazon.de
community.intel.com
static.twitchcdn.net
and other smaller domains I tested.

After deselecting the option, these domains work as expected. Others like youtube.com are working without changing it. So it seems that with the latest update a lot of AWS-Servers are blocked.

# location --version
location 0.9.16

# location version
Thu, 03 Aug 2023 05:22:02 GMT

# location update
Already on the latest version

# update-location-database
The database has been updated recently

The firewall is running on CU176.
Comment 1 Nicolas Pöhlmann 2023-08-07 06:53:43 UTC
Just FYI: I've tested 2 firewalls running CU175, none of them were affected before the update to the latest CU177 even with an updated location database. Afterwards, they were both affected and the option needs to be disabled to have a fully functional proxy.

And I digged deeper into the logs and these error messages started right after the update of the location database (Thu, 03 Aug 2023 05:22:02 GMT).

Aug 07 08:13:03 squid-asnbl-helper[3454] WARN: Destination 'www.amazon.de' resolves to IP addresses '2600:9000:222e:dc00:e:13a1:b913:5f61' without corresponding ASN, probably selectively announced
Aug 07 08:13:03 squid-asnbl-helper[3454] WARN: Destination 'www.amazon.de' resolves to IP addresses '2600:9000:222e:8200:e:13a1:b913:5f61' without corresponding ASN, probably selectively announced
... (repeated another 6 times) ...
Aug 07 08:13:03 squid-asnbl-helper[3454] INFO: Denying access to destination 'www.amazon.de' due to suspected selective announcements

As it seems the helper script gets multiple IPV6 destinations for the domain instead of only supported IPV4's. Or did I missed that IPFire should now support IPV6?

And somethimes it also gets a mixed IPV4 and IPV6 return before it denies the access:

Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:200:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '13.32.27.64' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:e600:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '13.32.27.15' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:5800:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '13.32.27.5' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:9200:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:0:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:d800:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '13.32.27.83' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:6200:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Aug 07 06:57:17 squid-asnbl-helper[31110] WARN: Destination 'html5.gamedistribution.com' resolves to IP addresses '2600:9000:211e:d600:5:4275:8dc0:93a1' without corresponding ASN, probably selectively announced
Comment 2 Nicolas Pöhlmann 2023-08-07 07:09:54 UTC
Just FYI, here are manual lookups for an affected IPV6 (www.amazon.de and community.intel.com).

# location --debug lookup '2600:9000:222e:dc00:e:13a1:b913:5f61'
2600:9000:222e:dc00:e:13a1:b913:5f61:
  Network                 : 2600:9000::/34
  Country                 : United States of America
  Anycast                 : yes

# location --debug lookup '2600:9000:223f:3600:1e:3b87:9a00:93a1'
2600:9000:223f:3600:1e:3b87:9a00:93a1:
  Network                 : 2600:9000::/34
  Country                 : United States of America
  Anycast                 : yes

Or a lookup for an affected IPV4 (html5.gamedistribution.com).

# location --debug lookup '13.32.27.15'
13.32.27.15:
  Network                 : 13.32.0.0/15
  Country                 : United States of America
  Anycast                 : yes

Currently, it seems that only Amazon hosted domains are affected when this option is active.
Comment 3 Adolf Belka 2023-08-11 12:51:02 UTC
I am having the problem also with www.kernel.org. If Deny Selective Announcements option is disabled then the website opens with no problems otherwise it comes back with 

The proxy server is refusing connections.
An error occurred during a connection to www.kernel.org

I am running CU177.
Comment 4 Adolf Belka 2023-08-11 12:54:50 UTC
The message in my log is

Aug 11 14:50:56 squid-asnbl-helper[7822] WARN: Destination 'www.kernel.org' resolves to IP addresses '2604:1380:4601:e00::1' without corresponding ASN, probably selectively announced
Comment 5 Adolf Belka 2023-08-11 13:27:36 UTC
I setup a vm machine and downloaded and installed the databases from 2023-08-03 back to 2023-07-30 and the problem stayed in effect until I got to 2023-07-30 when the www.kernel.org website opened with the Deny Selective Announcements option enabled.

So the problem came in with database version 2023-07-31
Comment 6 cfusco 2023-08-11 15:13:57 UTC
Created attachment 1251 [details]
squid logs showing the errors accessing kagi.com

I have the same problem with the search engine https://kagi.com

Please find in attachment the relevant part of /var/log/squid/cache.log

# location --version
location 0.9.16
# location version
Sun, 06 Aug 2023 05:23:29 GMT
# location update
Downloaded new database from Fri, 11 Aug 2023 05:26:26 GMT
# update-location-database
The database has been updated recently
Comment 7 cfusco 2023-08-11 15:16:00 UTC
I forgot to mention that I am on CU177
Comment 8 Adolf Belka 2023-08-20 13:15:46 UTC
DeepL Translator also is flagged for Selective Announcements


Aug 20 15:10:34 squid-asnbl-helper[14011] WARN: Destination 'www2.deepl.com' resolves to IP addresses '172.65.212.243' without corresponding ASN, probably selectively announced
Aug 20 15:10:34 squid-asnbl-helper[14011] INFO: Denying access to destination 'www2.deepl.com' due to suspected selective announcements
Comment 9 Adolf Belka 2023-08-21 12:07:00 UTC
wikipedia also gets blocked for selective announcements
Comment 10 Michael Tremer 2023-09-04 18:33:40 UTC
A quick update: Since we rolled back to the database being generated by the previous libloc release, this works again. I will have to investigate.
Comment 11 Adolf Belka 2024-03-17 21:18:49 UTC
This issue is back again.

tplink websites, kernel.org and amazon.co.uk all fail due to selective announcements. I am sure others do also.

There have also been other reports in the forum of sites failing to be accessed.

If selective announcements option is disabled in the web proxy then the above sites open with no problems.

[root@ipfire user]# location --version
location 0.9.17
[root@ipfire user]# location version
Sat, 09 Mar 2024 06:05:24 GMT
[root@ipfirer user]# location update
https://location.ipfire.org/databases/ is serving an outdated database. Trying next mirror...
Could not download a new database
Comment 12 Adolf Belka 2024-03-17 21:53:03 UTC
Here is the squid cache log showing the selective announcements

Mar 17 22:04:41 squid-asnbl-helper[2743] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:aa00:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 17 22:04:41 squid-asnbl-helper[2743] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:3800:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 17 22:04:41 squid-asnbl-helper[2743] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:6400:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 17 22:06:14 squid-asnbl-helper[21281] WARN: Destination 'static.siege-amazon.com' resolves to IP addresses '2600:9000:269a:4a00:1a:f1e1:7cc0:93a1' without corresponding ASN, probably selectively announced
Mar 17 22:06:14 squid-asnbl-helper[21281] WARN: Destination 'static.siege-amazon.com' resolves to IP addresses '2600:9000:269a:800:1a:f1e1:7cc0:93a1' without corresponding ASN, probably selectively announced
Mar 17 22:06:14 squid-asnbl-helper[21281] WARN: Destination 'static.siege-amazon.com' resolves to IP addresses '2600:9000:269a:9000:1a:f1e1:7cc0:93a1' without corresponding ASN, probably selectively announced
Mar 17 22:06:14 squid-asnbl-helper[21281] WARN: Destination 'static.siege-amazon.com' resolves to IP addresses '2600:9000:269a:d200:1a:f1e1:7cc0:93a1' without corresponding ASN, probably selectively announced
Mar 17 22:11:41 squid-asnbl-helper[21786] WARN: Destination 'www.kernel.org' resolves to IP addresses '2604:1380:4601:e00::1' without corresponding ASN, probably selectively announced
Mar 17 22:11:41 squid-asnbl-helper[21786] INFO: Denying access to destination 'www.kernel.org' due to suspected selective announcements
Mar 17 22:27:43 squid-asnbl-helper[21786] WARN: Destination 'hfc-fs.s3-eu-west-1.amazonaws.com' resolves to IP addresses '52.218.100.32' without corresponding ASN, probably selectively announced
Mar 17 22:27:43 squid-asnbl-helper[21786] WARN: Destination 'hfc-fs.s3-eu-west-1.amazonaws.com' resolves to IP addresses '52.218.46.74' without corresponding ASN, probably selectively announced
Mar 17 22:27:43 squid-asnbl-helper[21786] INFO: Denying access to destination 'hfc-fs.s3-eu-west-1.amazonaws.com' due to suspected selective announcements

This is just a few lines from the logs. There are more entries for a range of web site locations.
Comment 13 Michael Tremer 2024-03-21 14:52:54 UTC
Is this all still reproducible with the location database from today or later? Lots of changes have been rolled which should hopefully fix this.
Comment 14 Adolf Belka 2024-03-21 19:34:06 UTC
I updated to the latest database (21st March) but I am still having the problems.

location version
Thu, 21 Mar 2024 04:40:05 GMT


Here is the squid cache.log when trying to access amazon.co.uk and amazon.de although amazon.nl does not show the problem.

Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:ba00:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:a200:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:b200:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:5e00:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:5600:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:4a00:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:5200:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.co.uk' resolves to IP addresses '2600:9000:275b:9c00:15:c9dc:593:6781' without corresponding ASN, probably selectively announced
Mar 21 20:25:10 squid-asnbl-helper[2577] INFO: Denying access to destination 'www.amazon.co.uk' due to suspected selective announcements
Mar 21 20:25:13 squid-asnbl-helper[2577] WARN: Destination 'www.amazon.de' resolves to IP addresses '52.85.240.204' without corresponding ASN, probably selectively announced
Mar 21 20:25:13 squid-asnbl-helper[2577] INFO: Denying access to destination 'www.amazon.de' due to suspected selective announcements

Disabling the selective announcements option in the web proxy allows the above two sites to be accessed.

With the latest database the kernel.org site no longer gets blocked due to selective announcement.
Comment 15 Michael Tremer 2024-03-22 16:12:23 UTC
Thank you for testing this. I believe the problem is now resolved:

> root@michael:~# location lookup 2600:9000:275b:4a00:15:c9dc:593:6781 52.85.240.204
> 2600:9000:275b:4a00:15:c9dc:593:6781:
>   Network                 : 2600:9000:2750::/44
>   Country                 : United States of America
>   Autonomous System       : AS16509 - Amazon.com, Inc.
>   Anycast                 : yes
> 52.85.240.204:
>   Network                 : 52.85.240.0/22
>   Country                 : United States of America
>   Autonomous System       : AS16509 - Amazon.com, Inc.
>   Anycast                 : yes

I have written more about this here: 

> https://lists.ipfire.org/hyperkitty/list/location@lists.ipfire.org/thread/GXVEVONEYWQOIAZM5MBX7YAUCWYNH6ZA/
Comment 16 Adolf Belka 2024-03-22 17:57:14 UTC
I updated the database but had to then clear the web proxy cache before it would work properly.

All the website urls that were causing a problem for me before, are now working with no problems.
Comment 17 Michael Tremer 2024-03-25 10:32:02 UTC
Thanks for the feedback. I think we can close this one straight away then since there is nothing to ship in an update. The new database has already been released.