The current version of libloc is trying to do this when retrieving an HTTPS URL from within an environment in which a HTTP(S) proxy must be used: CONNECT ftp.apnic.net:80 HTTP/1.0 This is not correct: First, HTTP/1.1 should be used, not HTTP/1.0. Second, the request must go to destination port 443, not 80, since it is meant to be a HTTPS request, and CONNECT to a non-TLS port is usually not permitted. Third, it would be good if libloc would set a User-Agent header in this phase as well. wget, in contrast, does this: CONNECT www.google.com:443 HTTP/1.1 User-Agent: Wget/1.21.3 Host: www.google.com:443 https://docs.python.org/3/howto/urllib2.html#id7 suggests this may be a limitation in the urllib module ("Currently urllib.request does not support fetching of https locations through a proxy."), however, I have yet to find out whether this still applies to urllib3, which is what we use.
(In reply to Peter Müller from comment #0) > The current version of libloc is trying to do this when retrieving an HTTPS > URL from within an environment in which a HTTP(S) proxy must be used: > > CONNECT ftp.apnic.net:80 HTTP/1.0 > > This is not correct: First, HTTP/1.1 should be used, not HTTP/1.0. Second, > the request must go to destination port 443, not 80, since it is meant to be > a HTTPS request, and CONNECT to a non-TLS port is usually not permitted. > Third, it would be good if libloc would set a User-Agent header in this > phase as well. I don't think it makes a difference whether HTTP/1.0 or HTTP/1.1 is used unless CONNECT was only added in 1.1. > https://git.ipfire.org/?p=location/libloc.git;a=blob;f=src/scripts/location-importer.in;hb=f3637cf853126e8909f309ceab0471d98bcf47c9#l1865 We have set a User-Agent. It might be a separate issue then if urllib does not include it in the proxy connection. > wget, in contrast, does this: > > CONNECT www.google.com:443 HTTP/1.1 > User-Agent: Wget/1.21.3 > Host: www.google.com:443 > > https://docs.python.org/3/howto/urllib2.html#id7 suggests this may be a > limitation in the urllib module ("Currently urllib.request does not support > fetching of https locations through a proxy."), however, I have yet to find > out whether this still applies to urllib3, which is what we use. urllib is somewhat limited indeed. But we are not using urllib3. Instead we use the version of urllib that is part of the standard distribution which used to be called "urllib2" in Python 2. But I don't quite know what the alternatives could be. The most commonly used option these days I think is "requests" which I personally find rather basic (only supports HTTP/1.1) and does not support async, etc. It is basically just a wrapper over urllib3. Then there is pycurl which is more the way that I would like to go because it simply wraps cURL. And cURL is great. It is up to date, fast, easy to extend and supports a lot of things that we would want to have. However, pycurl only has a very low-level interface which would make the code a little bit verbose. In most places, I am using tornado.httpclient which is a brilliant wrapper around pycurl and that suits 100% of my needs wherever I am. It does however not feel too obvious to depend on tornado to implement the importer and downloader in libloc. We also have a couple of other projects that suffer from the same issue. Mainly ddns, but there is fireinfo as well. I would really want all of these to be based on the same client, so that we can rather pool knowledge that we have gained in one project and take it to another. Are there any other options that are worth exploring?
http://git.ipfire.org/?p=people/ms/libloc.git;a=commit;h=d1cb7ab09e51d8a418cc8e52a7757f7a7db930e1 Author: Michael Tremer <michael.tremer@ipfire.org> downloader: Let urllib handle proxies We read http_proxy and https_proxy from the environment and set both of them for every request. Setting the second one turned a request for an HTTPS URL into a plain HTTP request, which then asked the proxy for a tunnel to port 80 of the host: CONNECT ftp.apnic.net:80 HTTP/1.1 Requests for HTTP URLs were even asking for a tunnel to the proxy itself. urllib already reads the proxies from the environment, knows how to fetch something using HTTPS through them, and also honours no_proxy, which we did not. So we no longer configure anything ourselves. Every request now gets an opener of its own, because the default one remembers the proxies that were set when it was first used. The headers that the caller passed were being modified, and its User-Agent was always replaced with ours. There is a test with a proxy that records what it is being asked for. Fixes: #13655 - Proxy support for HTTPS URLs is broken Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
This has been fixed in the master branch of libloc (d1cb7ab). The downloader read http_proxy and https_proxy from the environment itself and set both of them for every request. Setting the second one turned a request for an HTTPS URL into a plain HTTP request, which then asked the proxy for a tunnel to port 80 ("CONNECT ftp.apnic.net:80"), exactly as reported. For HTTP URLs it even asked for a tunnel to the proxy itself. urllib already reads the proxies from the environment and knows how to open a tunnel to port 443, so the downloader no longer configures anything. As a side effect no_proxy is being honoured now, which it was not before. There is a test with a proxy that records what it is being asked for, and a request for an HTTPS URL through a real proxy works. I have not run the importer against the servers of the RIRs through a proxy, so it would be good if you could confirm that this works for you.
This has been fixed in libloc 0.9.20, which has just been tagged. Closing this as fixed.