Bug 13934 - ipfire location db erroneously marks documentation ranges within rfc3849 and rfc5737 belonging to US
Summary: ipfire location db erroneously marks documentation ranges within rfc3849 and ...
Status: CLOSED FIXED
Alias: None
Product: Location Database
Classification: Unclassified
Component: Database (show other bugs)
Version: unspecified
Hardware: all All
: Will affect an average number of users Major Usability
Assignee: Spencer Norton
QA Contact: Michael Tremer
URL: https://gitlab.torproject.org/tpo/cor...
Keywords:
Depends on:
Blocks:
 
Reported: 2026-02-02 15:59 UTC by Fimth Nasubbav
Modified: 2026-10-26 16:43 UTC (History)
1 user (show)

See Also:


Attachments
[PATCH] importer: Filter non-global geofeed and RIR umbrella networks (5.06 KB, patch)
2026-10-18 03:15 UTC, Spencer Norton
Details | Diff
Unit test and live verification suite for Bug #13934 (test_bogon_geofeed_filtering.py) (10.50 KB, text/plain)
2026-10-18 03:20 UTC, Spencer Norton
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Fimth Nasubbav 2026-02-02 15:59:18 UTC
Range 2001:db8::/32 is [specifically reserved for documentation and examples only](https://www.rfc-editor.org/rfc/rfc3849), so IP address within this range should not be considered to belong to any country. However, the current ipfire location db marks 2001:db8::/32 belonging to US, and already cause [problem in tor](https://gitlab.torproject.org/tpo/core/tor/-/issues/41205).
Comment 1 Fimth Nasubbav 2026-02-02 16:42:02 UTC
Besides, the current ipfire location db also marks 192.0.2.0/24, 198.51.100.0/24 , and 203.0.113.0/24, which are [reserved for use in examples in specifications and other documents](https://www.rfc-editor.org/rfc/rfc5737) belonging to US, which could also (and might already) impact tor.
Comment 2 Michael Tremer 2026-02-02 17:04:38 UTC
Hello,

thank you for your ticket. This is indeed true. We have been looking at adding an extra flag for some address ranges that mark them as deserved, but sadly we did not have time to come round to this.

Currently we have a lot on our plate so that we have slightly de-prioritised this project (not IPFire Location, just adding the extra flag), but it will happen at some point in the future.

If you are able to help us with funding this project, that would be great so that we can invest more time into it.
Comment 3 Spencer Norton 2026-10-18 03:15:47 UTC
Created attachment 1774 [details]
[PATCH] importer: Filter non-global geofeed and RIR umbrella networks

Root Cause & Fix Summary
========================

Investigating today's live database (Sat, 17 Oct 2026 14:48:44 GMT) and `src/scripts/location-importer.in` shows why `192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24` (RFC 5737) and `2001:db8::/32` (RFC 3849) — along with `2001:2::/48` (RFC 5180), `2001:10::/28` (RFC 4843), and `2002::/16` (RFC 3056) — are marked `US` (and `2001::/32` is marked `AS6939`):

1. Missing `_check_parsed_network()` in `_fetch_geofeed()` and `handle_update_overrides()`:
   `_parse_inetnum_block()`, `_parse_line()`, `_import_aws_ip_ranges()`, `_import_oci_ip_ranges()`, and `_import_spamhaus_drop()` all filter networks through `self._check_parsed_network(network)`, which rejects `not network.is_global`, `network.is_unspecified`, IPv4 `prefixlen < 7`, and IPv6 `prefixlen < 10`. However, `_fetch_geofeed()` (`location-importer.in:1964`) and `handle_update_overrides()` (`location-importer.in:2062`) did not call `self._check_parsed_network(network)`. As a result, third-party geofeeds with unrestricted (`0.0.0.0/0`, `::/0`) overrides (specifically `https://geofeed.constant.com`) injected those 7 non-global prefixes with `country: US`. Also, adding `network.is_multicast or network.is_reserved` to `_check_parsed_network()` catches multicast/reserved prefixes where Python's `ipaddress.is_global` is `True`.

2. Unfiltered `geofeed_networks` union in `handle_write()`:
   In `handle_write()` (`location-importer.in:428`), `INSERT INTO n (network)` selected all rows from `geofeed_networks` without joining `network_geofeeds` on `networks.network >>= geofeed_networks.network` (unlike the country `UPDATE` query at lines 535-552), allowing out-of-bailiwick geofeed prefixes to enter `n` as empty networks.

3. `APNIC-AP` umbrella RIR allocation blocks in `_parse_inetnum_block()`:
   Once the `US` geofeed entries are filtered out, `203.0.113.0/24` and `2001:db8::/32` would otherwise fall back to APNIC's `/8` and `/12`-`/23` IANA pool umbrella objects (`netname: APNIC-AP`, `country: AU`), which cover `203.0.0.0/8` (`202.0.0.0/7`) and `2001:c00::/23` (`2001:c00::/22`) with APNIC's headquarters country (`AU`). Extending the existing historic/umbrella `netname` filter regex in `_parse_inetnum_block()` (`location-importer.in:1337`) with `^APNIC-AP(-ERX)?$` skips those 35 umbrella pool blocks while preserving all 127,903 IPv4 and 2,665 IPv6 explicit `AU` allocations in `apnic.db` (verified against `delegated-apnic-extended-latest`: 0 / 11,606 AU allocations lost).

4. IPv6 special-purpose ranges in `handle_update_announcements()`:
   `handle_update_announcements()` purged IPv4 documentation/benchmark ranges and `2002::/16`, but did not purge `2001::/23` (RFC 2928 / RFC 4380 Teredo / RFC 5180 Benchmarking), `2001:db8::/32` (RFC 3849), or `3fff::/20` (RFC 9637), allowing `2001::/32` (`AS6939`) to persist from BGP announcements.


Verification (End-to-End PostgreSQL + `location-importer` + `libloc` Lookup)
============================================================================

Tested against PostgreSQL 16 + `libloc` 0.9.19 (`make check`: 24/24 C/Python/shell tests PASS, 12/12 Perl tests PASS):

* Before patch (`master` / live `location.db` 2026-10-17):
  - `192.0.2.1`, `198.51.100.1`, `203.0.113.1`, `2001:db8::1`, `2001:2::1`, `2001:10::1`, `2002::1` -> `country: US`
  - `2001::1` -> `AS6939`

* After patch:
  - `192.0.2.1`, `198.51.100.1`, `203.0.113.1`, `2001:db8::1`, `2001:2::1`, `2001:10::1`, `2002::1`, `2001::1`, `3fff::1`, `192.0.0.1`, `198.18.0.1` -> `None` (not present in `location.db`)
  - Legitimate APNIC allocations (`1.1.1.1 -> AU`, `203.0.112.1 -> AU`, `203.0.114.1 -> CN`, `203.2.218.1 -> AU`, `2001:c00::1 -> KR`, `2001:db0::1 -> AU`, `2001:dc0::1 -> AU`) -> unchanged.

Attached:
1. `0001-importer-Filter-non-global-geofeed-and-RIR-umbrella-.patch` (also sent to `location@lists.ipfire.org`)
2. `test_bogon_geofeed_filtering.py` (unit test and live verification suite)
Comment 4 Spencer Norton 2026-10-18 03:20:14 UTC
Created attachment 1775 [details]
Unit test and live verification suite for Bug #13934 (test_bogon_geofeed_filtering.py)

Attached `test_bogon_geofeed_filtering.py`: offline unit test suite (5 tests covering `_check_parsed_network()`, `APNIC-AP` umbrella filtering, `handle_write()` geofeed bailiwick join, and BGP announcement bogon cleanup) plus `--verify-live` verification mode.
Comment 5 Michael Tremer 2026-10-20 08:58:46 UTC
https://git.ipfire.org/?p=location/libloc.git;a=commit;h=b8dd752c6fb89875b057fec5c832d3cc668f8de4

Author: Michael Tremer <michael.tremer@ipfire.org>

importer: Don't drop announcements of all of 2001::/23

2001::/23 has been set aside for protocol assignments, of which some
are being routed on the Internet, like 2001:4:112::/48 of AS112, which
would have lost its AS number. Others are 2001:3::/32 (AMT),
2001:20::/28 (ORCHIDv2) and 2001:30::/28 (Drone Remote ID).

We now only drop what is not meant to be routed: Teredo (2001::/32),
benchmarking (2001:2::/48) and ORCHID (2001:10::/28).

Fixes: #13934 - ipfire location db erroneously marks documentation ranges within rfc3849 and rfc5737 belonging to US
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Comment 6 Michael Tremer 2026-10-20 08:58:46 UTC
https://git.ipfire.org/?p=location/libloc.git;a=commit;h=6274fb37b8d63c9820720bb613418d8bc8a265ee

Author: Spencer Norton <spencer.norton@norvi.tech>

importer: Filter non-global geofeed and RIR umbrella networks

Documentation and special-purpose ranges from RFC 5737 (192.0.2.0/24,
198.51.100.0/24, 203.0.113.0/24), RFC 3849 (2001:db8::/32), RFC 5180
(2001:2::/48), RFC 4843 (2001:10::/28), and RFC 3056 (2002::/16) are
currently exported with country: US, and RFC 4380 Teredo (2001::/32) is
exported with aut-num: 6939:

1. _fetch_geofeed() and handle_update_overrides() did not pass parsed
   prefixes through _check_parsed_network(), allowing non-global rows in
   unrestricted geofeeds (such as https://geofeed.constant.com) into
   geofeed_networks. Also reject multicast and reserved ranges in
   _check_parsed_network(), as Python's ipaddress.is_global returns True
   for 224.0.0.0/4 and ff00::/8.
2. handle_write() selected all rows from geofeed_networks into the
   candidate network table n without joining network_geofeeds, emitting
   258 out-of-bailiwick prefixes with no attributes into the database.
3. APNIC publishes top-level IANA pool placeholders (netname: APNIC-AP
   and APNIC-AP-ERX) with country: AU (for APNIC's Brisbane office) across
   its /8 and /12-/23 blocks, which otherwise cover 203.0.113.0/24
   (via 202.0.0.0/7) and 2001:db8::/32 (via 2001:c00::/22) as AU once the
   geofeed entries are removed. Skip APNIC-AP and APNIC-AP-ERX in
   _parse_inetnum_block() alongside the existing RIR umbrella netnames.
4. Purge IPv6 special-purpose ranges (2001::/23, 2001:db8::/32, 3fff::/20)
   from BGP announcements in handle_update_announcements() alongside the
   existing IPv4 test/documentation rules.

Fixes: #13934 - ipfire location db erroneously marks documentation ranges within rfc3849 and rfc5737 belonging to US
Signed-off-by: Spencer Norton <spencer.norton@norvi.tech>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Comment 7 Michael Tremer 2026-10-20 08:59:00 UTC
A patch by Spencer Norton has been merged into the master branch of libloc (6274fb3), with one change (b8dd752):

* Networks from Geofeeds and overrides are now being checked like those of the RIRs, so that non-global, multicast and reserved networks are being dropped. The Geofeed of Constant (https://geofeed.constant.com) has 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 2001:2::/48, 2001:10::/28, 2001:db8::/32 and 2002::/16 as US, which is where those came from.
* Networks of a Geofeed outside of the networks that refer to it no longer end up in the database without anything.
* The placeholders APNIC-AP and APNIC-AP-ERX of APNIC (country AU) are being skipped.
* Announcements of Teredo (2001::/32), benchmarking (2001:2::/48), ORCHID (2001:10::/28) and documentation (2001:db8::/32, 3fff::/20) are being dropped. The patch dropped all of 2001::/23, which would have taken AS112 away from 2001:4:112::/48, so that has been narrowed down.

Tested with a local import of the WHOIS data of APNIC and ARIN and their Geofeeds: none of the ranges above are in the database any more, and 2001::/32 no longer has AS6939.

This will be in the database once the importer with these changes has been running.
Comment 8 Michael Tremer 2026-10-20 08:59:01 UTC
http://git.ipfire.org/?p=location/libloc.git;a=commit;h=6274fb37b8d63c9820720bb613418d8bc8a265ee

Author: Spencer Norton <spencer.norton@norvi.tech>

importer: Filter non-global geofeed and RIR umbrella networks

Documentation and special-purpose ranges from RFC 5737 (192.0.2.0/24,
198.51.100.0/24, 203.0.113.0/24), RFC 3849 (2001:db8::/32), RFC 5180
(2001:2::/48), RFC 4843 (2001:10::/28), and RFC 3056 (2002::/16) are
currently exported with country: US, and RFC 4380 Teredo (2001::/32) is
exported with aut-num: 6939:

1. _fetch_geofeed() and handle_update_overrides() did not pass parsed
   prefixes through _check_parsed_network(), allowing non-global rows in
   unrestricted geofeeds (such as https://geofeed.constant.com) into
   geofeed_networks. Also reject multicast and reserved ranges in
   _check_parsed_network(), as Python's ipaddress.is_global returns True
   for 224.0.0.0/4 and ff00::/8.
2. handle_write() selected all rows from geofeed_networks into the
   candidate network table n without joining network_geofeeds, emitting
   258 out-of-bailiwick prefixes with no attributes into the database.
3. APNIC publishes top-level IANA pool placeholders (netname: APNIC-AP
   and APNIC-AP-ERX) with country: AU (for APNIC's Brisbane office) across
   its /8 and /12-/23 blocks, which otherwise cover 203.0.113.0/24
   (via 202.0.0.0/7) and 2001:db8::/32 (via 2001:c00::/22) as AU once the
   geofeed entries are removed. Skip APNIC-AP and APNIC-AP-ERX in
   _parse_inetnum_block() alongside the existing RIR umbrella netnames.
4. Purge IPv6 special-purpose ranges (2001::/23, 2001:db8::/32, 3fff::/20)
   from BGP announcements in handle_update_announcements() alongside the
   existing IPv4 test/documentation rules.

Fixes: #13934 - ipfire location db erroneously marks documentation ranges within rfc3849 and rfc5737 belonging to US
Signed-off-by: Spencer Norton <spencer.norton@norvi.tech>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Comment 9 Michael Tremer 2026-10-20 08:59:01 UTC
http://git.ipfire.org/?p=location/libloc.git;a=commit;h=b8dd752c6fb89875b057fec5c832d3cc668f8de4

Author: Michael Tremer <michael.tremer@ipfire.org>

importer: Don't drop announcements of all of 2001::/23

2001::/23 has been set aside for protocol assignments, of which some
are being routed on the Internet, like 2001:4:112::/48 of AS112, which
would have lost its AS number. Others are 2001:3::/32 (AMT),
2001:20::/28 (ORCHIDv2) and 2001:30::/28 (Drone Remote ID).

We now only drop what is not meant to be routed: Teredo (2001::/32),
benchmarking (2001:2::/48) and ORCHID (2001:10::/28).

Fixes: #13934 - ipfire location db erroneously marks documentation ranges within rfc3849 and rfc5737 belonging to US
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Comment 10 Michael Tremer 2026-10-26 16:43:28 UTC
This has been fixed in libloc 0.9.20, which has just been tagged. Closing this as fixed.