Bug 12574

Summary: OpenVPN Internal server error when returning after generating root/host certificates
Product: IPFire Reporter: Adolf Belka <adolf.belka>
Component: openvpnAssignee: Erik Kapfer <ummeegge>
Status: CLOSED FIXED QA Contact:
Severity: Minor Usability    
Priority: Will affect all users CC: michael.tremer, peter.mueller, ummeegge
Version: 2   
Hardware: unspecified   
OS: Unspecified   
Attachments: httpd error log message
Fixes response header contains invalid characters

Description Adolf Belka 2021-02-26 14:21:26 UTC
Created attachment 861 [details]
httpd error log message

After generating the OpenVPN root/host certificates the wui shows an Internal Server error. The error message is attached.  All the certificates have been successfully created and if you refresh the screen or select the IPFire url and go to the OpenVPN wui page then the certificates are present and can be viewed. The OpenVPN server can also then be successfully started.

This is happening with Core Update 153 and also 154.

It does not occur with Core Update 152.

On a Core 153 installation I installed the Core 152 version of ovpnmain.cgi
The same problem still occurred.

I installed Core Update 152 and successfully generated the root/host certificates with no Internal Server error.
Then I installed the OpenVPN 2.5.0 binary from a Core Update 153 installation. Deleting and regenerating the root/host certificates then had the same Internal Server Error.
This bug appears to be related to the upgrade of OpenVPN from 2.4.9 to 2.5.0
It only looks to occur when the root/host certificates are generated and the wui is going back to the top level ovpnmain.cgi page

This will affect all users that generate or need to regenerate the root/host certificates in OpenVPN
Comment 1 Adolf Belka 2021-02-26 15:20:04 UTC
This was raised in the following IPFire Community thread

https://community.ipfire.org/t/error-generating-root-host-certificates-for-openvpn-header-name-contains-invalid-characters/4664
Comment 2 Adolf Belka 2021-02-26 17:36:50 UTC
I am finding it hard to figure out how having the binary either being version 2.4.9 or 2.5.0 is making a difference for this bug. During the Generation of the Root/Host certificates the OpenVPN binary is stopped.

I went back to my VM testbed setup and installed Core Update 152 and removed the certificates and then generated them and did that twice and there was no Internal Error message.

Keeping a backup of the 2.4.9 binary I then copied in to /usr/sbin the 2.5.0 version of the binary. In both cases I confirmed the version by running /usr/sbin/openvpn --version
I then removed the root/host certificates and generated them again. Internal error message came up. I repeated this and the same message happened.

I then changed back to 2.4.9 binary and no Internal error message and then again back to 2.5.0 binary and again the Internal error message and then back to 2.4.9 and again no Internal error message.

Clearly the different binary is causing the problem with the changing back to the main wui page after generating the root/host certificates but I can't see where in the ovpnmain.cgi that this could be caused by.

This lack of understanding of the mechanism is making me hesitate to pick up this bug.
Comment 3 Erik Kapfer 2021-03-01 14:48:31 UTC
Hi Adolf,
the "Response Header" message appears also new to me. As far as i can see it is an 'http:error' so Apache cuts the date to slices have it here like this '2021-03-01 13' whereby the hour is in the output but nothing else. Not sure about Apache but the date format seems to causes problems (may spaces?!) ? In Apaches debug mode the following appears:

[Mon Mar 01 13:45:51.313280 2021] [http:error] [pid 31140:tid 124481574008384] [client 192.168.:49446] AH02429: Response header name '2021-03-01 13' contains invalid characters, aborting request, referer: https://192.168.:444/

[Mon Mar 01 13:45:51.313441 2021] [headers:debug] [pid 31140:tid 124481574008384] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()

so the ap_headers_error_filter jumps there in. Since i am not that familiar with Apache am not really sure where the problem comes from.

Have tested a 

Header unset X-Powered-By

in httpd.conf and the Response Header message seems to disappear but another one comes up then which is a 'cgid:error'

[Mon Mar 01 14:11:19.217833 2021] [cgid:error] [pid 2514:tid 124481087460928] [client 192.168.:49904] Script timed out before returning headers: ovpnmain.cgi, referer: https://192.168.

the access log gives a 504 for the Content-Lenght 247

192.168. - admin [01/Mar/2021:14:06:11 +0100] "POST /cgi-bin/ovpnmain.cgi HTTP/1.1" 504 247

which takes place from the dh-paramter generation to the ROOTCERT_SUCCESS jump as far as i can see.

Sorry no fix so far only a little more investigation.

Best,

Erik
Comment 4 Adolf Belka 2021-03-09 09:49:25 UTC
As the problem only occurs when the 2.5.0 binary is installed, even if it is not running then it seemed to me that there must be some steps in the certificate generation where the binary is used and that must have some linkage to the header issue.

Searching through the ovpnmain.cgi the openvpn binary is used to create the ta.key for the server certificates. This is carried out just before the dh parameter generation.

While the ta.key generation may have some linkage to this problem, I have not been able to see how it could affect the header.


It has become clear to me that I definitely don't have the capabilities or understanding to pick this bug up, so I will leave it someone more capable to take on.
Comment 5 Erik Kapfer 2021-11-10 10:47:43 UTC
Hi Adolf,
sorry for the long time with no response. There might be a solution for this if the DH generation time does not takes too long (weak boards) but in that case there comes another message (will post it at the bottom).

The specific message 'Response header name ‘2021-02-24 18’ contains invalid characters' comes from a OpenVPN warning which are not logged in Apaches error_log. Have investigated it by printing the fatals and warnings to the browser which you can also check if you put the following lines in ovpnmain.cgi

use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
print header();
warningsToBrowser(1);

If you create then a new PKI the following warning appears:

2021-11-10 11:00:25 WARNING: Using --genkey --secret filename is DEPRECATED. Use --genkey secret filename instead.

.

OpenVPN lines the following --> https://community.openvpn.net/openvpn/wiki/DeprecatedOptions?__cf_chl_jschl_tk__=nydA_3cgvnqkQWKqgBi.lQIgXUMNJwDc6tVo0_sGGOY-1636537828-0-gaNycGzNCNE#Option:--secret
as deprecated out . So the problem is the double minus '--' before 'secret' in the TLS-Authentication key command.

Will add a fix into the attachment may you can also go for a try.

If the time takes too long (weak board, not enough entropy) the following message appears in error_log:

[Wed Nov 10 11:05:26.046581 2021] [core:error] [pid 17712:tid 132439074666048] (70007)The timeout specified has expired: [client 192.168.1.3:34506] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: https://192.168.1.1:444/

but this should be not a part from this bug.

Best,

Erik
Comment 6 Erik Kapfer 2021-11-10 10:48:59 UTC
Created attachment 953 [details]
Fixes response header contains invalid characters
Comment 7 Adolf Belka 2021-11-10 12:24:17 UTC
Hi Erik,
Thanks for the update. Also thanks for the hint on how to get5 more error messages when I am fault finding.

I will give your patch a go when I have finished some other builds I am working on and will provide feedback into here.
Comment 8 Adolf Belka 2021-11-14 12:49:37 UTC
I tried re-creating the root/host certificates with Core Update 160 on my vm and confirmed the problem still existed.

Then I made the changes as per your patch, removal of the -- in from of the "secret" option and re-tried creating the root/host certificates again and can confirm that this solves the problem.

Will you submit this as a patch now?
Comment 9 Erik Kapfer 2021-11-14 14:02:18 UTC
Hi Adolf,
glad to hear that it works. Have currently no development environment around so it might be great if you can deliver the patch.

Best,

Erik
Comment 10 Adolf Belka 2021-11-14 17:06:25 UTC
(In reply to Erik Kapfer from comment #9)
> Hi Adolf,
> glad to hear that it works. Have currently no development environment around
> so it might be great if you can deliver the patch.
> 
> Best,
> 
> Erik

No problems Erik, I will submit the patch on your behalf.
Comment 11 Erik Kapfer 2021-11-14 18:02:39 UTC
Great,
thanks for testing and solving this bug and for delivering the patch.

Best,

Erik
Comment 12 Adolf Belka 2021-11-14 20:45:40 UTC
Patch from Erik submitted to Dev Mailing List and Patchwork.

https://patchwork.ipfire.org/project/ipfire/patch/20211114204252.3464019-1-adolf.belka@ipfire.org/
Comment 13 Adolf Belka 2021-12-03 14:04:52 UTC
Patch has been merged into next - Core Update 162

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=acbd6ff4dbbfa248b00d3922f666da7e6fabcc6c
Comment 15 Adolf Belka 2021-12-11 14:22:34 UTC
I clones a Core Update 161 vm on my testbed and confirmed that the problem was present when trying to create the root/host certificates.

Then I upgraded the vm to Core Update 162 Testing and repeated creating the root/host certificates. I can confirm that this worked without any error messages confirming that the bug has been fixed.

Status changed to Verified.