Bug 11004 - Conntrack Helpers are used for IPSec Tunnels
Summary: Conntrack Helpers are used for IPSec Tunnels
Status: CLOSED CANTFIX
Alias: None
Product: IPFire
Classification: Unclassified
Component: --- (show other bugs)
Version: 2
Hardware: all All
: Will affect an average number of users Major Usability
Assignee: Michael Tremer
QA Contact:
URL:
Keywords:
Depends on:
Blocks: IPSECBUGS
  Show dependency treegraph
 
Reported: 2015-12-21 17:40 UTC by Tom Rymes
Modified: 2021-09-04 10:27 UTC (History)
3 users (show)

See Also:


Attachments
attachment-17713-0.html (240 bytes, text/html)
2018-06-19 20:35 UTC, Michael Tremer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Rymes 2015-12-21 17:40:30 UTC
We have to periodically disable TFTP and FTP connection tracking helpers because they interfere with TFTP and FTP traffic being sent over IPSec tunnels. Because there is no NAT involved (from the devices' point of view) when sending traffic over IPSec, it doesn't make sense to me that these helpers would be invoked for those connections.

Is it possible to limit the use of the helpers to traffic going across the NAT to Red? Are they used for traffic crossing from Green to Blue?
Comment 1 Tom Rymes 2016-02-23 18:38:44 UTC
Are there any updates to this? It has come up in the forum in a couple of threads (many of them mine):

http://forum.ipfire.org/viewtopic.php?t=4590
http://forum.ipfire.org/viewtopic.php?t=13321
http://forum.ipfire.org/viewtopic.php?t=15801
http://forum.ipfire.org/viewtopic.php?t=15484

The suggestion to blacklist the modules in the first thread does not seem to work here (or is overwritten during updates), so we have to remember to disable the modules after every reboot/upgrade (generally the only time we reboot).

The big problem with this solution, though, is that disabling the helper breaks port forwarding for TFTP traffic, but is required for IPSec traffic, meaning that I must choose one or the other. Surely there must be a way to prevent traffic between IPSec-connected subnets from using the helpers, while allowing traffic that actually crosses the NAT (from the packet's point of view) to use the helper?
Comment 2 Tom Rymes 2016-03-02 19:49:30 UTC
Not certain if this is relevant, but it seems to me that we could configure IPTables to disable connection tracking for traffic between hosts on the far side of an IPSec tunnel using rules along the lines of the ones in this link:

http://security.stackexchange.com/questions/63041/is-it-safe-to-disable-connection-tracking-in-iptables

This is, of course, way beyond my skill level, so I cannot implement it myself or even confirm it would work, but all of my inquiries indicate that connection tracking should not be needed for traffic over IPSec tunnels (transport mode connections, and the actual key-exchange, etc may be a different matter.

We continue to have frequent problems with this issue.
Comment 3 Michael Tremer 2016-03-03 01:04:37 UTC
Did you try the changes in Core Update 99/100, yet?
Comment 4 Tom Rymes 2016-03-03 02:32:23 UTC
No, I was unaware that there were any changes. Are these the ability to disable helpers in the WUI? If so, my concern is that the helpers are needed for traffic that comes in from the outside world via a port forward (TFTP and FTP come to mind), but must be disabled for traffic over IPSec.

Am I right to presume that I should download the nightly build to investigate?
Comment 5 Michael Tremer 2016-03-10 15:32:00 UTC
(In reply to Tom Rymes from comment #4)
> No, I was unaware that there were any changes. Are these the ability to
> disable helpers in the WUI? If so, my concern is that the helpers are needed
> for traffic that comes in from the outside world via a port forward (TFTP
> and FTP come to mind), but must be disabled for traffic over IPSec.

These are always disabled for IPsec. They are NAT helpers and there is no NAT for IPsec.
Comment 6 Tom Rymes 2016-03-10 15:41:53 UTC
That is not our experience. Our Cisco phones are not able to TFTP across an IPSec tunnel with them enabled. 

If we execute:

rmmod nf_nat_tftp
rmmod nf_conntrack_tftp

Then the Cisco phones can TFTP across IPSec. We must remove the modules from BOTH IPFire machines (on either end of the tunnel).

We also have issues with TFTP requests directed at the Red IP Address and sent to an internal server using DNAT. I cannot test it right now, but my memory is that it works fine out of the box, but if we unload the helper modules, it stops working.
Comment 7 Michael Tremer 2016-04-23 00:23:04 UTC
You can configure these now with Core Update 100.

Does the issue still exist?
Comment 8 Tom Rymes 2016-04-23 02:59:42 UTC
I will have to re-test, but the ability disable the helper resolves the symptom, but does not resolve the problem. In other words, the problem is that the helper is used for IPSec traffic, even though it does not cross NAT (from the packet's point of view). Disabling the helper resolves this problem, but you now use the benefit of the tracker for other connections that *DO* use NAT. 

In particular, as I mentioned above, we cannot port forward traffic from the internet to a server on Green when the helper is disabled.

I will confirm this is still the case and report back.
Comment 9 Peter Müller 2017-08-30 16:45:20 UTC
I can confirm this issue (or something very much related) with Core Update 112.
Placing VoIP calls through an IPsec tunnel does not work:

* Both SIP and H.323 ALGs are disabled on both IPFire systems (and they were rebooted afterwards).
* Both firewall rules allow SIP and RTP (audio) traffic from one internal IP to another.
* When placing a call, the SIP communication works somehow as the phone at the other end rings, but nobody can hear the other.

On machine #1, these hits appeared in the firewall log:
15:34:34 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=543 TOS=0x08 PREC=0x60 TTL=63 ID=32752 PROTO=UDP SPT=5064 DPT=5060 LEN=523 
15:34:26 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=543 TOS=0x08 PREC=0x60 TTL=63 ID=32750 PROTO=UDP SPT=5064 DPT=5060 LEN=523 
15:34:22 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=543 TOS=0x08 PREC=0x60 TTL=63 ID=32749 PROTO=UDP SPT=5064 DPT=5060 LEN=523 
15:34:14 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=543 TOS=0x08 PREC=0x60 TTL=63 ID=32746 PROTO=UDP SPT=5064 DPT=5060 LEN=523 
15:34:08 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=543 TOS=0x08 PREC=0x60 TTL=63 ID=32743 PROTO=UDP SPT=5064 DPT=5060 LEN=523 
15:33:50 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=200 TOS=0x18 PREC=0xA0 TTL=63 ID=0 DF PROTO=UDP SPT=5004 DPT=5004 LEN=180 
15:33:49 IN=green0 OUT=ppp0 MAC=[REDACTED] SRC=10.XXX.XXX.XXX DST=87.123.XXX.XXX LEN=543 TOS=0x08 PREC=0x60 TTL=63 ID=32733 PROTO=UDP SPT=5064 DPT=5060 LEN=523

On machine #2, it is only one line:
15:33:50 IN=ppp0 OUT=green0 MAC= SRC=87.173.XXX.XXX DST=10.XXX.XXX.XXX LEN=200 TOS=0x10 PREC=0x00 TTL=57 ID=0 DF PROTO=UDP SPT=5004 DPT=5004 LEN=180 MARK=0xc8

Enabling the ALGs seem to have no effect.

Since I have absolutely no clue what to do, are there any suggestions?

@Tom: Did you solve the issue?
Comment 10 Tom Rymes 2017-08-30 16:59:56 UTC
Peter: I do not think this has been resolved. If you are sending traffic over IPSec tunnels, you must disable the helpers, and if you are sending traffic across the NAT, you must enable them. If you need to *BOTH* send traffic over NAT *AND* send traffic over IPSec, I think you are out of luck.

Since I originally filed the bug report, the option to enable/disable the helpers in the WUI has been added, but it still leaves those who need to send traffic over both IPSec and NAT in the lurch. It is still my opinion that the helpers should not be intercepting traffic destined for IPSec connections, but I don't know what, if anything, can be done about that.

As for your problem, it is my experience that SIP calls that successfully connect, but have audio problems (one-way audio or similar) have a NAT problem. Make sure that your PBX has the remote LANs on the other side of the tunnel defined as a local network so it does not try to replace its internal address with its external address, causing the packets to get routed out the Red interface and be blocked on the way back in. What SIP solution are you using?
Comment 11 Peter Müller 2017-08-30 17:10:51 UTC
(In reply to Tom Rymes from comment #10)
> Peter: I do not think this has been resolved. If you are sending traffic
> over IPSec tunnels, you must disable the helpers, and if you are sending
> traffic across the NAT, you must enable them. If you need to *BOTH* send
> traffic over NAT *AND* send traffic over IPSec, I think you are out of luck.
Yes, that is exactly the problem.

Further, if the SIP ALG is disabled, my VoIP-ATAs register with their local addresses at the SIP provider (who sent me a note about this a while ago). If the ALG is enabled, the firewall machine rewrites the IP to the external one (RED).
> 
> Since I originally filed the bug report, the option to enable/disable the
> helpers in the WUI has been added, but it still leaves those who need to
> send traffic over both IPSec and NAT in the lurch. It is still my opinion
> that the helpers should not be intercepting traffic destined for IPSec
> connections, but I don't know what, if anything, can be done about that.
This puzzles me since Michael said in comment #5:
[...] 
>> These are always disabled for IPsec. They are NAT helpers and there is no
>> NAT for IPsec.
So, some NAT seems to interfere although there should not be any NAT. Strange.
> 
> As for your problem, it is my experience that SIP calls that successfully
> connect, but have audio problems (one-way audio or similar) have a NAT
> problem. Make sure that your PBX has the remote LANs on the other side of
> the tunnel defined as a local network so it does not try to replace its
> internal address with its external address, causing the packets to get
> routed out the Red interface and be blocked on the way back in. What SIP
> solution are you using?
I am using Grandstream ATAs (mostly 802, but some old 701/702) which do not have that sophisticated network configuration pages. The VoIP calls are placed by the "Direct IP call" feature.

SIP seems to work, but audio fails in both directions.
Comment 12 Tom Rymes 2017-08-31 14:31:40 UTC
Peter: You might be able to enable a setting on the Grandstream units that will cause them to rewrite the packets themselves with the proper external address. There is usually a place to input the external address in the settings, or you can use a STUN server or similar. Unfortunately, because you have the Helper disabled, the RTP audio stream will not be properly forwarded to the ATA. You could manually forward the ports to the ATA, but only if you have no more than one ATA, or perhaps if you configure them to use distinct port ranges for RTP. 

My personal suspicion here is that the logic that determines which packets are passed to the helper somehow sees any of the private IPv4 addresses (10.0.0.0 – 10.255.255.255, 172.16.0.0 – 172.31.255.255, or 192.168.0.0 – 192.168.255.255) and assumes that traffic needs to be processed by the helper. Or perhaps the system is using the helper if the traffic is destined for anywhere other than Green or Blue? Something is causing the helper to be used when sending across an IPSec tunnel, which should not be occurring.

This logic would normally be OK, as anything coming from a private IP range should be green or Blue and presumably headed to the outside world, crossing NAT. It goes off the rails once you introduce IPSec, though, as packets from private ranges might be destined to another private range on the far side of the tunnel, with no NAT in-between (from the packet's point of view - in reality it has crossed NAT twice while encapsulated, but the packet can't tell). I cannot remember if SIP traffic between Blue and Green works with the helper enabled.
Comment 13 Peter Müller 2017-09-02 22:20:58 UTC
(In reply to Tom Rymes from comment #12)
> Peter: You might be able to enable a setting on the Grandstream units that
> will cause them to rewrite the packets themselves with the proper external
> address. There is usually a place to input the external address in the
> settings, or you can use a STUN server or similar. Unfortunately, because
> you have the Helper disabled, the RTP audio stream will not be properly
> forwarded to the ATA. You could manually forward the ports to the ATA, but
> only if you have no more than one ATA, or perhaps if you configure them to
> use distinct port ranges for RTP. 
The RTP helper is not a problem, since there are port forwardings for SIP and RTP to the VoIP-ATA. So it works fine with and without the ALG helper.

Enabling or disabling the RTP ALG has no effect at all, neither on calls to POTS nor to internal IP addresses. With SIP ALG, it is the same, with the difference that the device registeres with its private IP to the SIP server when the helper is disabled.

At the moment, I have not found a proper setting in the configuration pages, but I am still searching. Placing direct IP calls to another device in the same internal network works, which indicates that the problem is related to IPFire.
> 
> My personal suspicion here is that the logic that determines which packets
> are passed to the helper somehow sees any of the private IPv4 addresses
> (10.0.0.0 – 10.255.255.255, 172.16.0.0 – 172.31.255.255, or 192.168.0.0 –
> 192.168.255.255) and assumes that traffic needs to be processed by the
> helper. Or perhaps the system is using the helper if the traffic is destined
> for anywhere other than Green or Blue? Something is causing the helper to be
> used when sending across an IPSec tunnel, which should not be occurring.
Yes, that sounds plausible.
> 
> This logic would normally be OK, as anything coming from a private IP range
> should be green or Blue and presumably headed to the outside world, crossing
> NAT. It goes off the rails once you introduce IPSec, though, as packets from
> private ranges might be destined to another private range on the far side of
> the tunnel, with no NAT in-between (from the packet's point of view - in
> reality it has crossed NAT twice while encapsulated, but the packet can't
> tell). I cannot remember if SIP traffic between Blue and Green works with
> the helper enabled.
I will report my findings (if any), but at the moment, I have absolutely no idea where to begin. Setting certain firewall rules allowing incoming SIP/RTP traffic with private IPs is not a solution, since my company requires encryption for confidential issues.

Switching to OpenVPN is impossible, too, because we have some 3rd-party-devices here which do not support that. Personally, I like IPsec more than OVPN anyway.

@Michael: Your posts sound like this issue does not appear with your systems. Would you mind giving us a hint on configuration etc.? Thanks in advance. :-)
Comment 14 Peter Müller 2017-09-08 06:50:30 UTC
There was no significant progress in the last time:

(a) Experiencing trouble with SIP ALG this morning. I enabled it yesterday, and now my ATA fails to register (one account only, the other still works - quite strange). Disabled and rebooted IPFire, and now everything is fine except that the SIP registrations use the LAN IP of the ATA, but this seems to work.

(b) Making "Direct IP calls" inside the GREEN network works without any issues. So, it is now sure that the problem is somewhere at the side of IPFire.

(c) Setting a STUN server does not make any difference. Neither some other possible related network settings on the ATA, but I have not finished here. I will try to post my conntrack entries database here, perhaps this might help.

My system status:
- SIP ALG: disabled
- RTP ALG: disabled
- Port forwardings for SIP and RTP port (ranges) have been set manually so the SIP server and SBC can reach the ATA.
- Core Update 113 on x86_64.
Comment 15 Tom Rymes 2017-09-24 00:53:43 UTC
I was looking at the output of 'iptables -L' and I think this is the source of the issues we are seeing:

Chain CONNTRACK (3 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate ESTABLISHED
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     icmp --  anywhere             anywhere             ctstate RELATED
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "h323"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "pptp"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "irc"


Basically, it looks to me that we are handling traffic from anywhere to anywhere with the helpers. I would suggest that these chains need to be modified to exclude all traffic bound *to* any RFC-1918 private address (10.0.0.0/8,172.16.0.0/12, or 192.168.0.0/16).

Provided that we aren't leaking any private addresses across Red, and provided red isn't connected to carrier grade NAT (which is a BIG assumption), that should resolve the IPSec issue. 

Another option (probably better) is to add lines excluding traffic belonging to the far side of an IPSec tunnel.

I am way over my head with regard to IPTables, so please correct me if I have missed anything here.
Comment 16 Peter Müller 2017-10-08 21:07:44 UTC
Surprisingly, it works here now with SIP and H323 ALGs _enabled_ on the first IPFire machine.

Further, I needed to change the configuration of one VoIP-ATA (thanks to Arne for the hint!) so it enumerates its public IP on its own. STUN does not work here.

I will change this on the 2nd machine, too, but if this works, my problem has been solved. (This gives no satisfaction since the reason for this could not be enumerated, I know...)
Comment 17 Peter Müller 2017-11-12 06:35:38 UTC
Experiencing some issues with the SIP/H.232 helpers here:

udp      17 3524 src=10.XXX.XXX.XXX dst=217.10.XXX.XXX sport=5064 dport=5060 packets=144 bytes=80941 [UNREPLIED] src=217.10.XXX.XXX dst=10.XXX.XXX.XXX sport=5060 dport=5064 packets=0 bytes=0 mark=0 helper=sip use=1
udp      17 3585 src=10.XXX.XXX.XXX dst=217.10.XXX.XXX sport=5068 dport=5060 packets=126 bytes=16518 src=217.10.XXX.XXX dst=84.153.XXX.XXX sport=5060 dport=5068 packets=17 bytes=7430 [ASSURED] mark=0 helper=sip use=1

My VoIP ATA can only register the second number ([ASSURED]), but not the first one. (But this is not related with IPsec, although I thought you might want to know.)

I am on Core Update 116, with SIP and H.232 ALGs enabled.
Comment 18 Tom Rymes 2017-11-12 06:59:44 UTC
Peter: I don't know much about H.323, but it sounds like a similar issue that can occur with SIP and NAT traversal. I have often seen that it is recommended that different ports be used for multiple devices behind NAT. I don't know if this is an issue with multiple registrations from the same device.
Comment 19 Peter Müller 2017-11-12 07:52:22 UTC
(In reply to Tom Rymes from comment #18)
> Peter: I don't know much about H.323, but it sounds like a similar issue
> that can occur with SIP and NAT traversal. I have often seen that it is
> recommended that different ports be used for multiple devices behind NAT. I
> don't know if this is an issue with multiple registrations from the same
> device.

Hello Tom,

I do use different SIP and RTP ports here.

Surprisingly, after executing "conntack -D -s 10.XXX.XXX.XXX -d 217.10.XXX.XXX", everything works fine now. Will test behaviour after reboot later.
Comment 20 Peter Müller 2017-11-12 12:57:28 UTC
After a reboot it shows the same behaviour as Tom reported initially:

* SIP & H.323 ALG enabled
* "normal" VoIP call possible after manual adjustments to conntrack
* VoIP call over VPN fails.

This issue can therof reproduced here.
Comment 21 Peter Müller 2018-02-06 21:07:19 UTC
FYI: Discussed with Michael a few days ago, but reason seems to be unclear, yet.
Comment 22 Michael Tremer 2018-02-12 00:27:58 UTC
(In reply to Peter Müller from comment #21)
> FYI: Discussed with Michael a few days ago, but reason seems to be unclear,
> yet.

This very likely is a kernel bug. Could someone affected test with 4.14 and confirm that this problem still exists?

https://people.ipfire.org/~arne_f/highly-experimental/kernel/
Comment 23 Tom Rymes 2018-02-13 23:08:16 UTC
I'll give it a go. How do I revert to the normal kernel if I have any undesirable outcomes?
Comment 24 Michael Tremer 2018-02-14 12:47:16 UTC
(In reply to Tom Rymes from comment #23)
> I'll give it a go. How do I revert to the normal kernel if I have any
> undesirable outcomes?

If you extract the kernel archive and run "update-bootloader" you can choose between both kernels. So if something goes terribly wrong you can just boot the old kernel again.

One known bug is that IPsec compression doesn't work any more (#11550).
Comment 25 Michael Tremer 2018-06-18 14:25:36 UTC
Tom, can you confirm that this is resolved with the new kernel?
Comment 26 Tom Rymes 2018-06-18 18:12:34 UTC
Crud, I forgot I was supposed to be testing this, my apologies.

It still is a problem on 4.14.2. I was going to update to the newer kernel Arne had up, but it looks to have been deleted.

[root@ossipee ~]# uname -a
Linux ossipee 4.14.2-ipfire #1 SMP Sun Nov 26 13:39:59 GMT 2017 x86_64 Intel(R) Celeron(R) CPU N3060 @ 1.60GHz GenuineIntel GNU/Linux
Comment 27 Michael Tremer 2018-06-18 18:16:09 UTC
Yes, we are on ~4.14.50 now. You should be able to see the update in testing
already I think...
Comment 28 Peter Müller 2018-06-18 18:29:20 UTC
I am currently updating the conntrack tools. Maybe that will solve the issue...
Comment 29 Michael Tremer 2018-06-18 18:30:26 UTC
No that is just userspace and showing what conntrack states the kernel has...
Comment 30 Tom Rymes 2018-06-18 18:57:41 UTC
I found an old archive of the 4.14.44 kernel that I had applied to another machine, and we still have the same issue if the ALG is enabled.
Comment 31 Peter Müller 2018-06-19 20:27:09 UTC
Experiencing issues with VoIP over IPsec N2N again. One callee can be heard always, but the other can't (or just in one direction). Both machines are IPFire systems and no configuration changes were made.

Further, the issue cannot be reliable reproduced. There are no logs, no firewall hits, nothing. *sigh*

Any thoughts?
Comment 32 Michael Tremer 2018-06-19 20:35:44 UTC
Created attachment 596 [details]
attachment-17713-0.html

With SIP ALG on or off or both?
Comment 33 Tom Rymes 2018-06-19 22:26:40 UTC
Peter: Perhaps there is an issue related to this bug that you are experiencing, but my experience with ~75 remote SIP devices on the far sides of ~20 IPSec tunnels doesn't match up with what you are reporting. My experience:

- SIP ALG On: SIP IN/OUT via Red works. SIP traffic across IPSec does not work (NAT issues manifested as one-way audio or no audio).

- SIP ALG Off: SIP across IPSec works, but SIP IN/OUT via Red does not work.

Keep in mind that your provider and your device(s) generally have a *bunch* of different settings that will affect whether a given device's packets will have the correct IP Address on them. We use Asterisk for our PBX, for example, and if we do not define the local networks explicitly, it will try to do NAT handling on those extensions and all sorts of weird stuff will happen. The weirdest was that a device connected via IPSec and whose traffic should have never left our internal networks would only work when port forwarded in. It turned out that one of the NAT settings was causing that device to rewrite its packets with the external IP, and it was sending traffic out via Red and back in to the PBX via the port forward. It was utterly not anticipated, but it worked until we disabled the port forward one day and couldn't figure out why the phone stopped working.

In other words, I wouldn't be certain that you don't have an unrelated problem. If memory serves, you have a device connecting to a PBX/provider that is not local to you, is that right? Do the devices also have to communicate with each other via the local network? SIP Reinvite could also cause havoc if you're using it.

Tom
Comment 34 Peter Müller 2018-06-20 16:57:18 UTC
The situation here is:

Firewall A:
- using upstream VoIP provider 1&1/Versatel
- SIP ALG on
- H323 ALG on

Firewall B:
- using upstream VoIP provider sipgate
- SIP ALG off
- H323 ALG off

Both machines are connected via an IPsec N2N and need to place phone calls via POTS and internal VPN. I cannot tell when problems occured again exactly, but I am unaware of any configuration changes.

Of course, every provider is using its own settings (often poorly documentated), and it was quite hard to get thinks working. Right now, a user behind B cannot hear A via VPN. If A initiated the call, he can and everything works fine.

@Michael: Doesn't the libnetfilter_conntrack package has something to do with kernel space?
Comment 35 Peter Müller 2018-06-28 19:22:42 UTC
Things are now working again here. Some misconfiguration at one ATA caused the trouble. Sorry for the rush.
Comment 36 Michael Tremer 2018-07-04 18:54:25 UTC
(In reply to Peter Müller from comment #34)
> @Michael: Doesn't the libnetfilter_conntrack package has something to do
> with kernel space?

No.

Could you guys try the following:

> iptables -t raw -I CONNTRACK -s GREEN_SUBNET -d IPSEC_SUBNET -j RETURN

Add this on both sides. This should skip the connection tracking check. This might not be it, but I would like to know what happens...
Comment 37 Tom Rymes 2018-07-04 19:17:48 UTC
I will give that a go. Just run that command from the shell on both ends after re-enabling the helper and rebooting, right?
Comment 38 Michael Tremer 2018-07-04 19:19:39 UTC
Enable the helpers first, then reboot, then run the command with the subnets
replaced obviously.

They are not persistent. So another reboot will get rid of them again.

Happy Independence Day!
Comment 39 Michael Tremer 2018-07-20 16:24:30 UTC
Any luck?
Comment 40 Peter Müller 2019-10-13 10:32:09 UTC
Just for the records: I just bumped into this again.

Since VoIP calls over VPN used to work for some time, I am not sure whether an updated changed the conntrack behaviour or I made some mistakes at the VoIP configuration. :-/
Comment 41 Peter Müller 2019-10-28 15:16:37 UTC
For the records: Setting "Remove OBP from Route Header" to "No" solved the problem for me. It looks like the conntrack tools need that information to process the actual destination of a RTP connection. :-/
Comment 42 Peter Müller 2019-11-10 11:10:02 UTC
(In reply to Peter Müller from comment #41)
> For the records: Setting "Remove OBP from Route Header" to "No" solved the
> problem for me. It looks like the conntrack tools need that information to
> process the actual destination of a RTP connection. :-/

Well, actually, it did not. Some VoIP calls via VPN work, but most do not.
This seems to be dependent on which ATA initiates the RTP connection (snippet
from connection tracking table):

(a) UDP 	10.XXX.XXX.XXX [ATA 1] > 87.XXX.XXX.XXX 	5060 		DE 	212.XXX.XXX.XXX 	5060 	DE 	15k  /  6k 		0:59:54
(b) UDP 	10.XXX.XXX.XXX [ATA 2] 				5064	 		10.XXX.XXX.XXX [ATA 1] 	5060 		 3k  /  4k 		0:59:54
(c) UDP 	10.XXX.XXX.XXX [ATA 1] > 87.XXX.XXX.XXX 	5060 		DE 	212.XXX.XXX.XXX 	5060 	DE 	78k  /  2k 		0:32:14
(d) UDP 	10.XXX.XXX.XXX [ATA 2] 				5004 			10.XXX.XXX.XXX [ATA 1] 	5004 		54k  / 54k 		0:02:59
(e) UDP 	10.XXX.XXX.XXX [ATA 1] > 87.XXX.XXX.XXX 	5005	 	DE 	10.XXX.XXX.XXX [ATA 2]  5005 		300B /  0B 		0:00:29
(f) UDP 	10.XXX.XXX.XXX [ATA 2] 				5005 > 1024 		10.XXX.XXX.XXX [ATA 1] 	5005 		114B /  0B 		0:00:29

Connections (a) and (c) are just normal SIP keepalives to the ISP and can be
omitted here. Connection (b) is the SIP connection from one VoIP ATA to the other,
which works fine every time I tried this and in every constellation.

Connection (d) is the RTP channel initiated by ATA 2, and eventually successful
since the send/receive data amount is the same. Connections (e) and (f) are bogus,
since both are either NATted (the first one is rewritten to the RED IP address of
the firewall, the second has a different source port number).

H.323 conntrack helper was disabled on both sites, SIP helper is enabled on site 1.

I will try the iptables command Michael posted a while ago and report back.
Comment 43 Peter Müller 2020-05-25 14:14:43 UTC
> I will try the iptables command Michael posted a while ago and report back.

Well, I have tried that meanwhile and it solves the problem. What shall we do about this? Should we bypass connection tracking helpers for IPsec connections in general?
Comment 44 Michael Tremer 2020-05-26 10:05:54 UTC
(In reply to Peter Müller from comment #43)
> > I will try the iptables command Michael posted a while ago and report back.
> 
> Well, I have tried that meanwhile and it solves the problem. What shall we
> do about this? Should we bypass connection tracking helpers for IPsec
> connections in general?

No, we should not bypass the connection tracking, but we should always bypass the NAT helpers which seem to get confused with IPsec.

Peter, are you going to submit a patch? Is it clear what has to be done?
Comment 45 Peter Müller 2020-05-26 13:56:26 UTC
Not really. Could you describe it more detailled? :-)
Comment 46 trevlyng 2020-05-28 14:25:16 UTC
I was working with Tom to test this, like Michael suggested I enabled the H.323 and SIP ALG on both ends of the IPSec tunnel and rebooted the machines to apply the changes. Pasting in the command suggested did not work however as it would not apply the changes to the iptables: 

[root@siteA ~]# iptables -t raw -I CONNTRACK -s 192.xxx.x.x/24 -d 192.xxx.x.x/24 -j RETURN
[root@siteA ~]# iptables -L CONNTRACK
Chain CONNTRACK (3 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate ESTABLISHED
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     icmp --  anywhere             anywhere             ctstate RELATED
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "sip"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "h323"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "pptp"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "irc"

I instead had to remove the -t argument and then was able to add the rule:

[root@siteA ~]# iptables -I CONNTRACK -s 192.xxx.x.x/24 -d 192.xxx.x.x/24 -j RETURN
[root@siteA ~]# iptables -L CONNTRACK
Chain CONNTRACK (3 references)
target     prot opt source               destination
RETURN     all  --  192.xxx.x.x/24       192.xxx.x.x/24
ACCEPT     all  --  anywhere             anywhere             ctstate ESTABLISHED
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     icmp --  anywhere             anywhere             ctstate RELATED
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "sip"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "h323"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "pptp"
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED helper match "irc"

After applying the iptables changes to both ends I was able to get successful SIP phone calls from across the IPSec tunnel.
Comment 47 Michael Tremer 2020-05-29 08:42:59 UTC
No, the raw table and the filter table are something different.

The command to view the added rules would have been "iptables -t raw -L CONNTRACK". This is where the NAT stuff is.

You still want to keep the regular connection tracking in the filter table. Deactivating that has some other side effects and I would not be in favour of that.

So can you confirm that only adding this rule on both sides works?

> iptables -t raw -I CONNTRACK -s 192.xxx.x.x/24 -d 192.xxx.x.x/24 -j RETURN
Comment 48 trevlyng 2020-05-29 14:09:12 UTC
Thank you for the clarification, in that case I tested it with only the applied changes in the raw table on both machines and removing the filter table changes:

[root@siteA ~]# iptables -t raw -L CONNTRACK
Chain CONNTRACK (1 references)
target     prot opt source               destination
RETURN     all  --  192.xxx.x.x/24       192.xxx.x.x/24
CT         udp  --  anywhere             anywhere             udp dpt:sip CT helper sip
CT         tcp  --  anywhere             anywhere             tcp dpt:sip CT helper sip
CT         udp  --  anywhere             anywhere             udp dpt:h323gatestat CT helper RAS
CT         tcp  --  anywhere             anywhere             tcp dpt:h323hostcall CT helper Q.931
CT         tcp  --  anywhere             anywhere             tcp dpt:pptp CT helper pptp
CT         tcp  --  anywhere             anywhere             tcp dpt:6667 CT helper irc

Once the changes were made I lost audio on both ends of SIP calls through the IPSec connection. It works with the changes made in both tables but not if it is only applied to the raw table.
Comment 49 Michael Tremer 2020-05-29 18:06:30 UTC
Okay. Thank you testing this.

What happens when you flush the whole raw CONNTRACK chain?

> iptables -t raw -F CONNTRACK

It might happen than exiting SIP connections need to be restarted because once it has found out that a connection is SIP it will activate the helpers for as long as it exists.
Comment 50 trevlyng 2020-06-03 14:11:18 UTC
So I tested the command and flushed the raw CONNTRACK chain but ended up with the same result, dead air on both ends of the call. This was tested with and without the changes to the raw table just to confirm for both scenarios that we got the same result.
Comment 51 Michael Tremer 2020-06-04 11:01:48 UTC
Thanks for testing.

Do you have the output of "iptables -t raw -L -nv" for me after testing?

If also would help to have a look at what is in /proc/net/nf_conntrack.
Comment 52 trevlyng 2020-06-05 15:45:22 UTC
Of course, and sure thing, this is the output of Site A while on the SIP call from Site B and grepping the phone IP in Site B:

[root@SiteA ~]# grep 192.yyy.y.yyy  /proc/net/nf_conntrack
ipv4     2 udp      17 28 src=192.xxx.x.x dst=192.yyy.y.yyy sport=10861 dport=21645 packets=1 bytes=68 [UNREPLIED] src=192.yyy.y.yyy dst=24.xxx.xx.xxx sport=21645 dport=10861 packets=0 bytes=0 mark=0 zone=0 use=2
ipv4     2 udp      17 7 src=192.xxx.x.x dst=192.yyy.y.yyy sport=17694 dport=26576 packets=164 bytes=32728 [UNREPLIED] src=192.yyy.y.yyy dst=192.xxx.x.x sport=26576 dport=17694 packets=0 bytes=0 mark=0 zone=0 use=2
ipv4     2 udp      17 29 src=192.xxx.x.x dst=192.yyy.y.yyy sport=10860 dport=21644 packets=347 bytes=69400 [UNREPLIED] src=192.yyy.y.yyy dst=24.xxx.xx.xxx sport=21644 dport=10860 packets=0 bytes=0 mark=0 zone=0 use=45
ipv4     2 udp      17 3592 src=192.yyy.y.yyy dst=192.xxx.x.x sport=5060 dport=5060 packets=41780 bytes=29294057 src=192.xxx.x.x dst=192.yyy.y.yyy sport=5060 dport=5060 packets=42225 bytes=22761499 [ASSURED] mark=0 zone=0 use=5

and

[root@SiteA ~]# iptables -t raw -L -nv
Chain PREROUTING (policy ACCEPT 29M packets, 15G bytes)
 pkts bytes target     prot opt in     out     source             destination
  80M   44G CONNTRACK  all  --  *      *       0.0.0.0/0          0.0.0.0/0

Chain OUTPUT (policy ACCEPT 4241K packets, 2088M bytes)
 pkts bytes target     prot opt in     out     source             destination

Chain CONNTRACK (1 references)
 pkts bytes target     prot opt in     out     source             destination
 403K  166M RETURN     all  --  *      *       192.xxx.x.x/24     192.yyy.y.y/24
Comment 53 Tom Rymes 2020-06-05 16:53:39 UTC
Trevlyn: Can you clarify if the output you just posted is when you have a working call in progress? If so, can you also reconfirm which steps you took to achieve that?
Comment 54 trevlyng 2020-06-05 17:12:41 UTC
Sorry, to clarify this output was from a non-working call. This is from adding the changes to the raw table "iptables -t raw -I CONNTRACK -s GREEN_SUBNET -d IPSEC_SUBNET -j RETURN" without making any changes to the filter table, resulting in the call going through but dead air on both ends once the call is picked up on the receiving end. The output was taken mid-call while we had the line open.
Comment 55 Peter Müller 2021-09-04 10:27:18 UTC
Thanks to the "NAT Slipstreaming" vulnerability, we had to remove all Connection Tracking helpers in Core Update 155. Therefore, I consider this bug as no longer being valid ("CANTFIX"), and am closing it. Please reopen, if necessary.

https://blog.ipfire.org/post/security-announcement-mitigating-nat-slipstreaming
https://blog.ipfire.org/post/ipfire-2-25-core-update-155-released-security-advisory