Bug 12271

Summary: DNS resolution is very slow with DNS-over-TLS
Product: IPFire Reporter: Michael Tremer <michael.tremer>
Component: ---Assignee: Michael Tremer <michael.tremer>
Status: CLOSED FIXED QA Contact:
Severity: Major Usability    
Priority: Will affect all users CC: arne.fitzenreiter, matthias.fischer, peter.mueller, stefan.schantl, ummeegge
Version: 2   
Hardware: unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 12233    

Description Michael Tremer 2020-01-10 16:27:29 UTC 
I am running some experimental DNS code and have observed that DNS is very slow when using TLS.

For simplicity I am using 1.1.1.1 and 8.8.8.8 as an example since they are very close to my test host.

I can ping both with less than 4ms. If I try to resolve something using kdig on the console, I get around 3ms for 1.1.1.1:

> [root@fw01 ~]# kdig @1.1.1.1 +tls-ca=/etc/ssl/certs/ca-bundle.crt +tls-hostname=cloudflare-dns.com ipfire.org
> ;; TLS session (TLS1.2)-(ECDHE-SECP256R1)-(ECDSA-SHA256)-(AES-256-GCM)
> ;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 37344
> ;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1
> 
> ;; EDNS PSEUDOSECTION:
> ;; Version: 0; flags: ; UDP size: 1452 B; ext-rcode: NOERROR
> 
> ;; QUESTION SECTION:
> ;; ipfire.org.                  IN      A
>
> ;; ANSWER SECTION:
> ipfire.org.             1088    IN      A       81.3.27.38
> 
> ;; Received 55 B
> ;; Time 2020-01-10 16:21:24 GMT
> ;; From 1.1.1.1@853(TCP) in 3.0 ms

This already includes the TLS handshake with TLS 1.2. Wow.

When I configure this server as a forwarder in unbound, the fastest I see something resolve is around 110ms. Considering that unbound verifies DNSSEC signatures and kdig does not, this is still very far from acceptable. Websites open noticeably slower.

Google uses TLS 1.3 and the results are quite similar.

Our own resolver at 81.3.27.54 is substantially slower and I am not sure why that is, so I would not consider it for a benchmark. But using that with kdig and in unbound is a huge difference again of about 80ms vs 150 or even up to 200ms.

I tried to keep the cache in unbound as hot as possible so that fetching any DNSSEC keys cannot be the reason here.

I think this is a blocker that is bad enough that we cannot release DNS over TLS, yet.

We are not using TCP Fast Open for any of the tests (presumably because openssl does not support it). I currently do not know how to test whether we are using 0-RTT at least with TLS 1.3. However, that does not seem to make such a difference if it is even enable because Cloudflare performs as fast a simple UDP query. That is where we need to go, obviously.

@Erik: I know you have done some research here. Any idea what we can do?
Comment 1 Erik Kapfer 2020-01-10 19:48:25 UTC 
Hi Michael,

(In reply to Michael Tremer from comment #0)
> @Erik: I know you have done some research here. Any idea what we can do?

Currently no, but i do have an older speed reference with much better results which is from may 2019 and we (Matthias and myself) also recognizes a vast speed difference which was about December if i remember it correctly.

The speed differences can differ in general but from 18 measured servers 14 are partly wide below 100ms . Currently not sure what has been changed on unbound in that period of time but for sure this can be made better.

Here is the list checked also with kdig, reference can be found in here --> https://forum.ipfire.org/viewtopic.php?f=50&t=21954#p120691 :

=====================================================================================================
From Host: rec1.dns.lightningwirelabs.com ---- With IP: 81.3.27.54 ---- Date: Fri May 17 10:56:56 CEST 2019

in 182.1 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-ECDSA-SECP384R1-SHA384-CHACHA20-POLY1305

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: kaitain.restena.lu ---- With IP: 158.64.1.29 ---- Date: Fri May 17 10:56:56 CEST 2019

in 28.9 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA512-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dnsovertls.sinodun.com ---- With IP: 145.100.185.15 ---- Date: Fri May 17 10:56:57 CEST 2019

in 33.3 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA512-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dnsovertls1.sinodun.com ---- With IP: 145.100.185.16 ---- Date: Fri May 17 10:56:57 CEST 2019

in 173.6 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dns.cmrg.net ---- With IP: 199.58.81.218 ---- Date: Fri May 17 10:56:57 CEST 2019

in 124.1 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dns.neutopia.org ---- With IP: 89.234.186.112 ---- Date: Fri May 17 10:56:58 CEST 2019

in 57.4 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dot-jp.blahdns.com ---- With IP: 108.61.201.119 ---- Date: Fri May 17 10:56:58 CEST 2019

in 858.6 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-RSA-PSS-RSAE-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dot-de.blahdns.com ---- With IP: 159.69.198.101 ---- Date: Fri May 17 10:57:00 CEST 2019

in 79.0 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-RSA-PSS-RSAE-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dns2.digitalcourage.de ---- With IP: 46.182.19.48 ---- Date: Fri May 17 10:57:01 CEST 2019

in 42.7 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: cloudflare-dns.com ---- With IP: 1.1.1.1 ---- Date: Fri May 17 10:57:02 CEST 2019

in 57.4 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-ECDSA-SECP256R1-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: security-filter-dns.cleanbrowsing.org ---- With IP: 185.228.168.9 ---- Date: Fri May 17 10:57:03 CEST 2019

in 88.0 ms

The encryption is OK and works with: TLS1.2-ECDHE-X25519-RSA-SHA512-CHACHA20-POLY1305

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dns.adguard.com ---- With IP: 176.103.130.130 ---- Date: Fri May 17 10:57:03 CEST 2019

in 37.8 ms

The encryption is OK and works with: TLS1.2-ECDHE-X25519-RSA-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: getdnsapi.net ---- With IP: 185.49.141.37 ---- Date: Fri May 17 10:57:03 CEST 2019

in 31.1 ms

The encryption is OK and works with: TLS1.2-ECDHE-SECP256R1-RSA-SHA512-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dot.securedns.eu ---- With IP: 146.185.167.43 ---- Date: Fri May 17 10:57:04 CEST 2019

in 89.2 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-RSA-PSS-RSAE-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: security-filter-dns.cleanbrowsing.org ---- With IP: 185.228.169.9 ---- Date: Fri May 17 10:57:05 CEST 2019

in 99.9 ms

The encryption is OK and works with: TLS1.2-ECDHE-X25519-RSA-SHA512-CHACHA20-POLY1305

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dns.adguard.com ---- With IP: 176.103.130.131 ---- Date: Fri May 17 10:57:05 CEST 2019

in 28.4 ms

The encryption is OK and works with: TLS1.2-ECDHE-X25519-RSA-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: dns.quad9.net ---- With IP: 9.9.9.10 ---- Date: Fri May 17 10:57:06 CEST 2019

in 1923.7 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-ECDSA-SECP256R1-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================
From Host: cloudflare-dns.com ---- With IP: 1.0.0.1 ---- Date: Fri May 17 10:57:08 CEST 2019

in 52.4 ms

The encryption is OK and works with: TLS1.3-ECDHE-SECP256R1-ECDSA-SECP256R1-SHA256-AES-256-GCM

The certificate is trusted and OK

The DNSSEC validation works and is OK

=====================================================================================================

Best,

Erik
Comment 2 Michael Tremer 2020-01-13 12:31:47 UTC 
Hmm, I can confirm these. Some are really fast, which is amazing. However, when I only use a fast server (in my test Cloudflare with 3ms latency), unbound still takes hundreds of milliseconds to resolve a DNS query. Either unbound has a bug here or we have something misconfigured that slows down unbound but not kdig.

One interesting thing might be that unbound uses OpenSSL and kdig uses GnuTLS.
Comment 3 Peter Müller 2020-03-07 09:28:54 UTC 
Without taking measurements, DNS over TLS seems to be a little bit faster in upcoming Core Update 142.
Comment 4 Peter Müller 2020-12-10 16:40:24 UTC 
For the records, Unbound 1.13.0 finally added support for reusing established TCP and TLS connections:

https://www.nlnetlabs.nl/projects/unbound/download/#unbound-1-13-0

Since Matthias already sent a patch to update it, I expect DNS over TLS (and TCP) to be much faster in Core Update 154.
Comment 5 Peter Müller 2020-12-10 16:41:08 UTC 
https://patchwork.ipfire.org/patch/3705/
Comment 8 Michael Tremer 2021-03-16 12:22:19 UTC 
Core Update 154 has been released