classification
Title: Logging SyslogHandler not appending '\n' to the end
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: elelement, reahaas, vinay.sajip
Priority: normal Keywords:

Created on 2016-10-10 12:14 by elelement, last changed 2021-02-17 15:33 by reahaas. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24556 reahaas, 2021-02-17 15:33
Messages (4)
msg278412 - (view) Author: José Manuel (elelement) Date: 2016-10-10 12:14
I'm using SyslogHandler from logging.handlers to send syslog messages to a Fluentd input (https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_syslog.rb), both in TCP and UDP. UDP works fine, but TCP does not work. 

The "problem" is that the handler is not ending messages with a new line '\n' character (I realized that using tcpdump). I've temporarily added this to line 855 of handlers.py: 
            msg = prio + msg + '\n' 
And now is working. 

Now I'm confused because maybe this is not an issue but a problem of Fluentd. For the time, I will create a new class extending SyslogHandler and override the emit function.

Thank you for your time.
msg278417 - (view) Author: José Manuel (elelement) Date: 2016-10-10 14:43
After reading the RFC5424 it seems that there is no such "new line message delimiter":

--------------------------------
4.3.1.  Message Length

   The message length is the octet count of the SYSLOG-MSG in the
   SYSLOG-FRAME.  A transport receiver MUST use the message length to delimit a syslog message
--------------------------------

So I think it must be a Fluentd error. This is what caused my confusion:

From in_syslog.rb (https://github.com/athenahealth/fluent-plugin-newsyslog/blob/master/lib/fluent/plugin/in_newsyslog.rb):

--------------------------------
# syslog family add "\n" to each message and this seems only way to split messages in tcp stream
Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log, "\n", callback)
--------------------------------
msg278443 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-10-10 20:37
> So I think it must be a Fluentd error.

So I'll close this.
msg278807 - (view) Author: José Manuel (elelement) Date: 2016-10-17 13:27
Sorry to bother you again, but I've tested this not only with Fluentd, but with a RSYSLOG server and it does not work with TCP except if you manually add the trailer LF character. Other than that, UDP default transport protocol has no issues and works fine with both systems. Here's my simple code:

-------
sHandler = logging.handlers.SysLogHandler(address=(address[0], address[1]), socktype = socket.SOCK_STREAM)
sHandler.setFormatter(logging.Formatter(fmt=MSG_SYSLOG_FORMAT, datefmt=DATE_FMT))
self.addHandler(sHandler)
-------

After reading RFC 6587 I think the SyslogHandler class should implement at least one of the framing mechanisms proposed by this RFC, meant for TCP transmission:
- Octet counting
- Trailer character (e.g. LF)

Besides, I've being checking out the library "pyloggr" (https://github.com/stephane-martin/pyloggr) and they are implementing both mechanisms. As for SyslogHandler, it will be as simple as adding another field to the class constructor (use_delimiter?) and to add these lines to the emit code (it works):

-------
if (self.use_delimiter):
    msg = msg + '\n'
else:
    msg = str(len(msg)) + ' ' + msg # default behavior
-------

Thank you again
History
Date User Action Args
2021-02-17 15:33:48reahaassetnosy: + reahaas

pull_requests: + pull_request23337
2019-12-06 07:01:49vinay.sajipsetstatus: pending -> closed
resolution: not a bug -> out of date
2016-10-17 13:27:25elelementsetstatus: closed -> pending

messages: + msg278807
2016-10-10 20:37:05vinay.sajipsetstatus: open -> closed
resolution: not a bug
messages: + msg278443

stage: resolved
2016-10-10 19:40:11berker.peksagsetnosy: + vinay.sajip
2016-10-10 14:43:26elelementsetmessages: + msg278417
2016-10-10 12:14:22elelementcreate