Bug 13996

To avoid modifying all pages, it is therefore preferable to modify the escape and cleanhtml functions as follows.

// var/ipfire/header.pl
628: sub escape($) {
629: 	my $outstring = shift;
630: 	# decode the UTF-8 text so that characters with diacritical marks such as
631: 	# umlauts are treated correctly by the escape command
632: 	$outstring = &Encode::decode("UTF-8",$outstring);
633: 	$outstring = HTML::Entities::encode_entities($outstring);
634: 	$outstring = &Encode::encode("UTF-8",$outstring);
635: 	return $outstring;
636: }
637:
...
650: sub cleanhtml {
651: 	my $outstring =$_[0];
652: 	$outstring =~ tr/,/ / if not defined $_[1] or $_[1] ne 'y';
653: 	$outstring = escape($outstring);
654: 	return $outstring;
655: }
656:

I tested it in French, It works, but there are some side effects; I found one in firewall.cgi.
// html/cgi-bin/firewall.cgi
2351: 		$fwdfwsettings{'ruleremark'}=~ s/,/;/g;
2352: 		utf8::decode($fwdfwsettings{'ruleremark'});
2353: 		$fwdfwsettings{'ruleremark'}=&Header::escape($fwdfwsettings{'ruleremark'});
2354:

After removing the 2352 line the remark field with accents seem correct

It would need more extensive testing;

I think the two function do slightly different operations.

encode just converts a Perl string to a valid HTML text.
cleanhtml takes an UTF8 coded string and produces a HTML text coded with UTF8.

Your suggestion uses all string parameters as UTF8 encoded.
This means some unnecessary decode/encode operations ( runtime! ) and trusts that the Encode::encode() and Encode::decode() functions are identy for unencoded strings.

The problem is caused by the string type of Perl. First it was just a sequence of bytes. With occurence of utf it was expanded to a sequence of double byte integers. But the standard for UTF-8 doesn't allow all values as valid characters.
This produces a mixture of string values. ( See the description of Encode module in the Perl docs).

My suggestion is leaving the function definitions.
encode() operates on 'real' Perl strings.
cleanhtml() operates on any UTF8 encoded texts.
This means the WUI sources have to be checked for the right usage. My short search did not find too much occurances.

Thanks Phil for opening this. The problem has been fixed here:

> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a6c585b677e3c4b5bdcf0ea8c3886c9cb20ae623

The cleanhtml() function has been removed, too:

> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=88a825787408112cd34f60612898b572c67be154

(In reply to Michael Tremer from comment #3)
> Thanks Phil for opening this. The problem has been fixed here:
> 
> > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a6c585b677e3c4b5bdcf0ea8c3886c9cb20ae623
> 
> The cleanhtml() function has been removed, too:
> 
> > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=88a825787408112cd34f60612898b572c67be154

Agreed. This means a clear, unified processing for all types of strings.

Good work.

I just tested the web interface on the unstable CU203 cumulative update.

All French accents display correctly on pages and forms 

I only detected one last problem on the captive.cgi page, in the TERM field.

(In reply to Phil SCAR from comment #5)
> I only detected one last problem on the captive.cgi page, in the TERM field.

And what is that problem?

Created attachment 1716 [details]
captive.cgi Term problem with french

Add text
testaccentséèêàâùûôçe;,:/
After save
testaccents���������e;,:/

see captive.pdf attachment

The problem comes from using escape before save the file

> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=ac3fc5987f549140cbee5c99c380dd909e8232a1

Cool. Fixed.

I've made the corrections from the last commit in index.cgi and captive.cgi.
The display of French accents is now correct in the forms on the captive.cgi page.

However, it seems the Captive Portal page is no longer displaying.
I'm getting an ERR_CONNECTION_TIMED_OUT error in Microsoft Edge, and in Firefox, I'm redirected to the IPFire homepage.

This happens with or without accents in the forms.

If I restore the old version of index.cgi and captive.cgi, it displays correctly as I put it in the PDF.

But perhaps I've missed something; I'm not used to using the captive portal. Could you check?

Created attachment 1718 [details]
Patch captive/index.cgi

The issue appears to be caused by requiring header.pl from index.cgi. 
After removing require "${General::swroot}/header.pl", adding use Encode, and copying only the needed escape function into index.cgi, the captive portal page works again. 
This suggests that header.pl introduces a side effect affecting captive portal output, encoding, or HTTP response handling.

Here is the patch I tested. OK

Hello,

please don't add another unrelated bug to this report. If you think that there is a problem in the captive portal, please open a new report.

The original localisation problem has been fixed.

https://bugzilla.ipfire.org/show_bug.cgi?id=14006

Thank you.

https://www.ipfire.org/blog/ipfire-2-29-core-update-203-is-available-for-testing