Bug 12922

I believe that this should be feasible as long as the logging is directed to a syslog, which is possible according to the documentation. Will need to try it out on a vm based system.

Yes, Matthias has added this for lots of tools so far. It could be a one-liner like this:

> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=0b2aa5173d60878533ac72403d15ef9f97f8810f

So basically my HAProxy now logs into syslog with just a single line in /etc/haproxy/haproxy.cfg:

In section *global*, I've just added one line to make this happen. After a reload of HAProxy , each line that was previously logged into /var/log/haproxy is now forwarded into /var/log/messages, with facility: local0, see below.

global
  #Log to haproxy.log
  #log         127.0.0.1 local1
	
  #Log to local syslog server/UDP
  log 		127.0.0.1:514 local0

If desired, one can adjust the log format according to the infos provided here https://www.haproxy.com/documentation/hapee/1-8r1/onepage/#8.2.2

For my needs I've added this line in haproxy.cfg below the *frontend http_https* section. This is my custom config and may change on other installations.
  log-format "%ci:%cp %ft %b/%s %ST %B %tsc %{+Q}r"

The output in /var/log/messages look like this:

Sep 12 16:32:06 localhost.localdomain haproxy[29465]: 100.20.190.139:46614 http_https~ collab_server/collab01 200 1968 --VN "GET /dav.php/principals/User/ HTTP/1.1"

So, it's up to you to add an appropriate section in System Logs to filter for those log entries!

Thanks!

As Michael wrote, one line in '/srv/web/ipfire/cgi-bin/logs.cgi/log.dat' could perhaps be sufficient:

Under 'my %sections = ):'

...
'HAProxy' => '(haproxy\[.*?\]: )',
...

As I see it, we even wouldn't need a translation string.

But: I can't test this here...

@Adolf:
Can you test it?

(In reply to Matthias Fischer from comment #4)
> As Michael wrote, one line in '/srv/web/ipfire/cgi-bin/logs.cgi/log.dat'
> could perhaps be sufficient:
> 
> Under 'my %sections = ):'
> 
> ...
> 'HAProxy' => '(haproxy\[.*?\]: )',
> ...
> 
> As I see it, we even wouldn't need a translation string.
> 
> But: I can't test this here...
> 
> @Adolf:
> Can you test it?

Hi @Matthias

Yes, I plan to have a look at this after I have dealt with the NUT log reporting which will be done after another bug I am working on. Unfortunately I have had other non IPFire things come up in the last week which have diverted me but I will be trying to get back to things as quickly as I can.

I've tested it and actually it requires two addition lines in log.dat:

The one for the var/array: %sections
...
'haproxy' => '(haproxy\[.*?\]: )',
...

and one for the caption/translation of the drop down box: %trsections
...
'haproxy' => 'HAProxy',
...

Similar like those two lines above. Both are now adding a new section in WebIf.
BUT, unfortunately no messages are getting filtered, although I've got many of them in /var/log/messages, like:

Sep 13 19:13:14 localhost.localdomain haproxy[29465]: 100.20.194.139:47216 http_https~ collab_server/collab01 401 1013 --VN "PROPFIND /dav.php/addressbooks/family/ HTTP/1.1"
Sep 13 19:13:14 localhost.localdomain haproxy[29465]: 100.20.194.139:47216 http_https~ collab_server/collab01 207 2609 --VN "PROPFIND /dav.php/addressbooks/family/ HTTP/1.1"

I assume that (haproxy\[.*?\]: ) is a RegEx, right? 
So, basically it should find any occurrences of haproxy[nnnnn]: in the logs, shouldn't it?

Strange enough, when replacing the first line above, for testing purposes, with
'haproxy' => '(ipfire: )',
I get a result. Of course with logs from ipfire.

Testing the RegEx (haproxy\[.*?\]: ) in an online tool hits exactly two times with sample logs above. So why does this filter not work in WebIf?

Maybe those lines are too long for processing?

Further testing:

I've added a dummy line to /var/log/messages with cmd:

echo 'Sep 13 19:23:54 localhost.localdomain haproxy[29465]: 100.20.194.139:54682' >> /var/log/messages

and re-tested, to no avail either!

Sorry for so much replies :-(

When adding a new line, e.g. to messages:

echo 'Sep 13 19:54:59 test haproxy[29465]: 100.100.143.137:54682' >> /var/log/messages

This one gets now filtered in WebIf!

Is the cgi-script not capable of handling a dot, like in "localhost.localdomain"?

Further tests showed that this is true, because neither this line:

echo 'Sep 13 19:56:59 test.test haproxy[29465]: 100.100.143.137:54682' >> /var/log/messages

nor that line:

echo 'Sep 13 19:57.01 test. haproxy[29465]: 100.100.143.137:54682' >> /var/log/messages

is filtered out. See "test.test" or "test." as opposed to "test" above.

I leave it up to you now for a deeper analysis ;-)

Couldn't resist :-)

I assume this line in log.dat is causing the issue I'm currently facing:

foreach $_ (@log)
{
	/^... (..) (..:..:..) [\w\-]+ ${section}(.*)$/;

This looks like a RegEx, too(?) and maybe [\w\-]+ fails because of the dot in "localhost.localdomain"

Tested.

Confirmed.

But sorry, I have no idea yet... :-(

Looking at the usual messages in /var/log/messages, your HAProxy messages and the regular expression for filtering, it is clear to me that there is a problem.
Usually the messages have as field #4 the host name. '[\w\-]' is a correct expression describing the syntax of a host name. It is the same as function validhostname() in general-functions.pl accepts ( according to RFC1035 ).
Therefore HAProxy logging fills in an invalid host name.

So I've asked at Reddit for the correct syntax or how to change the behaviour of HAProxy.
https://www.reddit.com/r/haproxy/comments/xeu85n/haproxy_writes_non_rfc_compliant_host_name_to/

I answered  that the correct host name is even different: ipfire.ourhome
However, if Haproxy would use the real one, thks won't solve the issue either.

So how to proceed?

In IPFire internal FQDNs are of the form <host>.<domain>, your IPFire host name can not be ipfire.ourhome.
I suppose the host is named 'ipfire', with domain name 'ourhome'.
This is set by setup for example.

This is my Fireinfo from WebIF:

IPFire version	IPFire 2.27 (x86_64) - core167
Pakfire version	2.27-x86_64
Kernel version	Linux ipfire.ourhome 5.15.35-ipfire #1 SMP Tue Apr 26 10:56:57 GMT 2022 x86_64 AMD GX-412TC SOC AuthenticAMD GNU/Linux

And I understand that this is a FQDN: hostname.domain

Anyway, Haproxy uses the wrong host name. 


Where would you lookup the wrong hostname, in IPfire's setup?

Nada!

Just checked tbe setup and the hostname is: ipfire

So basically this is correct! I'm running a local Active Directory on a Synology NAS on the domain name is ourhome.local.

This works on all Windows clients and a PC's fully qualified domain name is e.g. pc1234.ourhome.local

For 'pc1234.ourhome.local' IPFire's interpretation would be
host: pc1234 ( the left-most label of the FQDN )
domain: ourhome.local ( which is also the domain of your NAS )

(In reply to Bernhard Bitsch from comment #16)
> For 'pc1234.ourhome.local' IPFire's interpretation would be
> host: pc1234 ( the left-most label of the FQDN )
> domain: ourhome.local ( which is also the domain of your NAS )

That's my understanding, too, but does not explain ipfire.ourhome nor even localhost.localdomain

Found it in /etc/hosts 

#cat hosts
127.0.0.1       localhost.localdomain localhost
172.17.0.1      gateway

Is this a bug, did not modify this file. To be exactly I've added  many hosts, using WebIf, but none of them are kn /etc/host

Can I change this  to:
127.0.0.1       ipfire localhost

without further issues?

I've just looked into the HAProxy docs about logging.
There is a field 'log-send-hostname' which should switch to RFC 3164 mode.
Default is the 'host name' ( whatever HAProxy uses for it ), but you could set it to the real name 'ipfire'.

(In reply to Michael from comment #18)
> Found it in /etc/hosts 
> 
> #cat hosts
> 127.0.0.1       localhost.localdomain localhost
> 172.17.0.1      gateway
> 
> Is this a bug, did not modify this file. To be exactly I've added  many
> hosts, using WebIf, but none of them are kn /etc/host
> 
> Can I change this  to:
> 127.0.0.1       ipfire localhost
> 
> without further issues?

My /etc/hosts file looks the same. So I wouldn't change it. 
The file doesn't contain information about the local namings.

(In reply to Bernhard Bitsch from comment #20)
> My /etc/hosts file looks the same. So I wouldn't change it. 
> The file doesn't contain information about the local namings.
Nevertheles, HAProxy obviously uses it!?

Will give the option log-send-hostname a try and report back.

localhost.localdomain is a synonym for the FQDN. You can actually use the hostname localhost to mean the same as your defined hostname. The IP for localhost is 127.0.0.1 and just means the computer that you are on.

So localhost.localdomain is the local FQDN for the computer you are on.

So technically HAProxy are correct to put localhost.localdomain to refer to the computer that they are logging on but it is not the most helpful, especially if you were logging to a remote syslog because you wouldn't then know which localhost it was referring to.

The log line can be whatever HAProxy want it to be as there is no standardised log line format as far as I am aware.
They have obviously decided to put the FQDN in that location instead of just the hostname as others have used, and then they have decided not to convert the local FQDN to a full FQDN.

Having a FQDN instead of just a hostname is not a big deal, the regex just needs to be modified to take account of that and the presence of one or more dots for the separators.

It might also be possible to find the actual hostname and domain name from IPFire and replace the localhost.localdomain entry with the global FQDN.

(In reply to Michael from comment #18)
> Found it in /etc/hosts 
> 
> #cat hosts
> 127.0.0.1       localhost.localdomain localhost
> 172.17.0.1      gateway
> 
> Is this a bug, did not modify this file. To be exactly I've added  many
> hosts, using WebIf, but none of them are kn /etc/host
> 
> Can I change this  to:
> 127.0.0.1       ipfire localhost
> 
> without further issues?

That explains what the problem is.

In my Arch Linux hosts file is where I have added the FQDN together with the localhost.localdomain localhost line and the /etc/hosts file is where programs usually look for the FQDN etc.

In IPFire this is stored in /var/ipfire/main/hostname.conf

When I get round to this I can look at using the hostname.conf data to replace the localhost.localdomain portion of the log line, unless doing that takes too long to execute.

(In reply to Bernhard Bitsch from comment #19)
> I've just looked into the HAProxy docs about logging.
> There is a field 'log-send-hostname' which should switch to RFC 3164 mode.
> Default is the 'host name' ( whatever HAProxy uses for it ), but you could
> set it to the real name 'ipfire'.
Added the option to haproxy.cfg and thats tbe result in /var/log/messages

   Sep 15 21:46:09 localhost.localdomain ipfire haproxy[10763]: 

Do you spot the difference? The option inserts a string, that I've used kn haproxy.cfg instead lf replacing the wrong hostname.

(In reply to Adolf Belka from comment #22)
> Having a FQDN instead of just a hostname is not a big deal, the regex just
> needs to be modified to take account of that and the presence of one or more
> dots for the separators.

Already tried that kn a quick way, yesterday, but did not succeed. So please can you modify the regex?

Thanks!

(In reply to Michael from comment #25)
> (In reply to Adolf Belka from comment #22)
> > Having a FQDN instead of just a hostname is not a big deal, the regex just
> > needs to be modified to take account of that and the presence of one or more
> > dots for the separators.
> 
> Already tried that kn a quick way, yesterday, but did not succeed. So please
> can you modify the regex?
> 
> Thanks!

Not of the top of my head. I am not that familiar with regex. However I know there are routines in ipfire that take a FQDN and separate it into a hostname by splitting at the first dot and then finding the end of the FQDN when there are no more dots before a space.
 So I know its been done and I just need to go and look for it and use it as a basis for this bug.

Originally you had the NUT logging in System Log being the highest priority with HAProxy next. As I have submitted the previous bug I was working on I was going to move onto the NUT one. Is that one still highest priority or should I try and work on HAProxy before NUT?

No, no please stick with nut logging of course.

I was trying to provide some helpful information with HAProxy logging and as yoh already know, I was stopped by some small issues 😞

Thank you!

(In reply to Michael from comment #24)
> (In reply to Bernhard Bitsch from comment #19)
> > I've just looked into the HAProxy docs about logging.
> > There is a field 'log-send-hostname' which should switch to RFC 3164 mode.
> > Default is the 'host name' ( whatever HAProxy uses for it ), but you could
> > set it to the real name 'ipfire'.
> Added the option to haproxy.cfg and thats tbe result in /var/log/messages
> 
>    Sep 15 21:46:09 localhost.localdomain ipfire haproxy[10763]: 
> 
> Do you spot the difference? The option inserts a string, that I've used kn
> haproxy.cfg instead lf replacing the wrong hostname.

That is really irritating. 
If I've understood the docs right, HAProxy doesn't set the host name in format local. With switching to RFC3164 format a host name shall be included. Your log message shows 2 host names ( localhost.localdomain and ipfire ).

BTW: I investigated about logging in IPFire. pakfire, for example, just calls the logger utility, without giving a host name. This part is inserted by logger, I think.

Just to clarify, if mistaken:

I've added the line 

  log-send-hostname  ipfire 

to haproxy.cfg and restarted HAProxy.


Before this modification, /var/log/messages showed:

   Sep 15 21:46:09 localhost.localdomain haproxy[10763]: xxxxx

and after the restart it now logs, e.g.:

   Sep 15 21:46:09 localhost.localdomain ipfire haproxy[10763]: xxxxx

So basically things got even worse after adding the option.

(In reply to Bernhard Bitsch from comment #11)
> Usually the messages have as field #4 the host name. '[\w\-]' is a correct
> expression describing the syntax of a host name. It is the same as function

I've put the following regex into an evaluation tool

  ^... (..) (..:..:..) [\w\-\.]+ haproxy(.*)$

using this line from logs

  Sep 15 21:46:09 localhost.localdomain haproxy[10763]: xxxxx

The result was OK, the regex recognized the line! All I did is to add \. inside the brackets [\w\-\.] and used a fix string instead of the var section.

After replacing the original line in log.dat, with 

 /^... (..) (..:..:..) [\w\-\.]+ ${section}(.*)$/

still no logs are shown in WebIF. Is there another code line to search and replace any regex?

At least in the loop where /var/log/message is searched, I did not found any further occurrence of an additional regex...

(In reply to Michael from comment #29)
> Just to clarify, if mistaken:
> 
> I've added the line 
> 
>   log-send-hostname  ipfire 
> 
> to haproxy.cfg and restarted HAProxy.
> 
> 
> Before this modification, /var/log/messages showed:
> 
>    Sep 15 21:46:09 localhost.localdomain haproxy[10763]: xxxxx
> 
> and after the restart it now logs, e.g.:
> 
>    Sep 15 21:46:09 localhost.localdomain ipfire haproxy[10763]: xxxxx
> 
> So basically things got even worse after adding the option.

I know. Reading a bit more of the docs reveals that the log message consists of two parts.
- the syslog part ( date host )
- HAProxy log message ( the remainder of the line )
So the host name comes from the syslog utility, the 'log-send-hostname' settings just prepends this name to the standard log message. 

Therefore we should find the reason why logging by HAProxy generates this host name. Other IPFire parts using the syslog facilty generate the 'right' name, as you can see in /var/log/messages.

(In reply to Michael from comment #30)
> (In reply to Bernhard Bitsch from comment #11)
> > Usually the messages have as field #4 the host name. '[\w\-]' is a correct
> > expression describing the syntax of a host name. It is the same as function
> 
> I've put the following regex into an evaluation tool
> 
>   ^... (..) (..:..:..) [\w\-\.]+ haproxy(.*)$
> 
> using this line from logs
> 
>   Sep 15 21:46:09 localhost.localdomain haproxy[10763]: xxxxx
> 
> The result was OK, the regex recognized the line! All I did is to add \.
> inside the brackets [\w\-\.] and used a fix string instead of the var
> section.
> 
> After replacing the original line in log.dat, with 
> 
>  /^... (..) (..:..:..) [\w\-\.]+ ${section}(.*)$/
> 
> still no logs are shown in WebIF. Is there another code line to search and
> replace any regex?
> 
> At least in the loop where /var/log/message is searched, I did not found any
> further occurrence of an additional regex...

I wouldn't recommend this. As stated in my former post, all IPFire parts logging to /var/log/messages produce the single name ( leftmost label of the FQDN ). It is not easy to find all places, where this fact is used.
Therefore we should concentrate on transforming the exception ( HAProxy ) to standard and not the opposite. IMHO

I agree with that opinion, nevertheless for the solely purpose of showing up (correct) logs in WebIf, it could be the way to go.

I've asked for help on HAproxy support community. Maybe someone can explain this behavior.

Will report back...

(In reply to Michael from comment #33)
> I agree with that opinion, nevertheless for the solely purpose of showing up
> (correct) logs in WebIf, it could be the way to go.
> 
> I've asked for help on HAproxy support community. Maybe someone can explain
> this behavior.
> 
> Will report back...

We'll wait for the answer. :)

Just to check your system, how is the entry in /var/log/messages for
   logger -t test "This is a test message"
?

The command puts out

  Sep 16 16:48:09 ipfire test: This is a test message

IMO, perfect!

Created attachment 1090 [details]
Fitlering HAproxy logs

Hah! Fixed! 

OK, hostname is still with an inline dot, still waiting for an answer from the HAProxy dev community, but nevertheless got the filter running in log.dat.

Obviously, there are more than one line to be patched:
Running core 167, don't know if there were recent changes in code.

First patch in regex, line 230: added \. in [\w\-]

225  # now read file if existing
226  if (open (FILE,($filestr =~ /.gz$/ ? "gzip -dc $filestr |" : $filestr))) {
227	 #&General::log("reading $filestr");
228      READ:while (<FILE>) {
229	 my $line = $_;
230      if ($line =~ /^${monthstr} ${daystr} ..:..:.. [\w\-\.]+ ${section}(.*)/) {

Second patch in regex, line 275: added \. in [\w\-]

273	foreach $_ (@log)
274	{
275		/^... (..) (..:..:..) [\w\-\.]+ ${section}(.*)$/;

Third patch in regex, line 398: added \. in [\w\-]

394 $lines = 0;
395 #print '<tt>';
396 foreach $_ (@log)
397 {
398 	/^... (..) (..:..:..) [\w\-\.]+ ${section}(.*)$/;

After inserting three times \. the section HAProxy in WebIf properly filters those logs.

Hope, it's of some help!
Michael

In the meantime, the hostname issue, has been solved. 

The cause for this, is still unknown: https://community.ipfire.org/t/unbound-and-other-programs-logging/654/30

(In reply to Bernhard Bitsch from comment #31)
> 
> I know. Reading a bit more of the docs reveals that the log message consists
> of two parts.
> - the syslog part ( date host )
> - HAProxy log message ( the remainder of the line )
> So the host name comes from the syslog utility, the 'log-send-hostname'
> settings just prepends this name to the standard log message. 
> 
> Therefore we should find the reason why logging by HAProxy generates this
> host name. Other IPFire parts using the syslog facilty generate the 'right'
> name, as you can see in /var/log/messages.

Has any conclusion been reached about why the syslog part does not have the correct hostname as all the other packages seem to?

I am a bit hesitant to change the regex in three places for all log entries to be able to match up with what is coming from HAProxy.

I would prefer to understand why we get the anomalous entry from HAProxy but not from other entries.

Sorry no insight from my side, as the hosts file /etc/hosts is now correct after adding a new host within the WebIf, as written in the forums.

So probably HAProxy is working correctly and the messages have been written correctly since adding a new host.

Unfortunately, I do not know why or what has previously saved wrong lines into /etc/hosts.

What is in the /etc/hosts file should not be causing the problem.

With the localhost.localdomain in my /etc/hosts file monit ends up with my correct hostname in the messages log file.

If monit, clamav etc can end up with the correct lines in the messages logfile then haproxy should also be able to do that. I just can't figure out why something different is ending up there.

By the way, your modified /etc/hosts file will revert back to the original form when you do a reboot of IPFire. I have confirmed that on  my system.

I just had a look through RFC 5424 and looked at the section on Hostname

The HOSTNAME field identifies the machine that originally sent the
   syslog message.

   The HOSTNAME field SHOULD contain the hostname and the domain name of
   the originator in the format specified in STD 13 [RFC1034].  This
   format is called a Fully Qualified Domain Name (FQDN) in this
   document.

   In practice, not all syslog applications are able to provide an FQDN.
   As such, other values MAY also be present in HOSTNAME.  This document
   makes provisions for using other values in such situations.  A syslog
   application SHOULD provide the most specific available value first.
   The order of preference for the contents of the HOSTNAME field is as
   follows:

   1.  FQDN

   2.  Static IP address

   3.  hostname

   4.  Dynamic IP address

   5.  the NILVALUE

So most of the programs such as monity and clamav etc are using the 3rd option in the list but the top preferred info is a FQDN.

This suggests that we do need to add in the check for a dot separator in the HOSTNAME field of the log line as that is the primary expected return info.

So I will create a patch set based on the changes shown by @Michael with the first patch being the change to add HAProxy to the system logs list and the second patch to modify the regex to work if a FQDN is used for the HOSTNAME portion of the logging line.

For the RegEx part you can have a look at comment #36,at least for the CGI in logs.dat.

Sorry for the delay in moving forward with this.

Patches have been submitted into the dev mailing list and into patchwork.

https://lists.ipfire.org/pipermail/development/2023-March/015620.html
https://lists.ipfire.org/pipermail/development/2023-March/015621.html
https://patchwork.ipfire.org/project/ipfire/list/?series=3498

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=452fd239f00121e957c932f909409485f26332d6
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=282e3fe4fb6c71942ac37c925ad541f14973365f

https://blog.ipfire.org/post/ipfire-2-27-core-update-174-is-available-for-testing

I have tested this on my vm testbed with CU174 Testing.

HAProxy now shows up in the system logs drop down box.

However it would be good if @Michael, the original poster could confirm that the logs are correctly filtered out due to the need to filter for just the hostname when a FQDN has been provided in the logs.

https://blog.ipfire.org/post/ipfire-2-27-core-update-174-released