This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: logging.handlers.RotatingFileHandler rotation broken under gunicorn
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Bruce Edge, berker.peksag, vinay.sajip
Priority: normal Keywords: patch

Created on 2016-12-17 20:00 by Bruce Edge, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9616 closed pathakumesh, 2018-09-28 23:09
Messages (3)
msg283511 - (view) Author: Bruce Edge (Bruce Edge) Date: 2016-12-17 20:00
I've been seeing some funny behavior in log files form a logging.handlers.RotatingFileHandler.

Here's my handler config snippet, in a "log.cfg":

[handlers]
keys=log_file

[formatters]
keys=access

[formatter_access]
format=%(asctime)s - %(name)s(%(funcName)s:%(lineno)s)[%(process)d] - %(levelname)s: %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

[handler_log_file]
class=logging.handlers.RotatingFileHandler
formatter=access
args=('/var/log/ml-api/access.log', 'a', 10000000, 16, 'utf8')

Specified using:

 /usr/local/bin/gunicorn --worker-class=gevent --bind unix:ml-api.sock -m 007 --workers=8 --log-config=log.cfg --log-level=DEBUG app

I ran this script to show the progression of log file creation:

while true ; do ls -lrt && sleep 10; done

Initially there's one incrementing log file, fine:

-rw-r--r-- 1 ubuntu www-data 9765042 Dec 16 17:50 access.log
total 9572
-rw-r--r-- 1 ubuntu www-data 9796828 Dec 16 17:50 access.log
total 9656
-rw-r--r-- 1 ubuntu www-data 9881053 Dec 16 17:50 access.log
total 9708
-rw-r--r-- 1 ubuntu www-data 9936776 Dec 16 17:50 access.log
total 9756
-rw-r--r-- 1 ubuntu www-data 9984782 Dec 16 17:50 access.log
total 9816

But as soon as it gets rotated, I immediately get 8 log files:

-rw-r--r-- 1 ubuntu www-data 9999911 Dec 16 17:50 access.log.7
-rw-r--r-- 1 ubuntu www-data    2578 Dec 16 17:50 access.log.5
-rw-r--r-- 1 ubuntu www-data    2578 Dec 16 17:50 access.log.3
-rw-r--r-- 1 ubuntu www-data    6871 Dec 16 17:50 access.log.2
-rw-r--r-- 1 ubuntu www-data    5122 Dec 16 17:50 access.log.1
-rw-r--r-- 1 ubuntu www-data   11165 Dec 16 17:50 access.log.4
-rw-r--r-- 1 ubuntu www-data    1718 Dec 16 17:50 access.log
-rw-r--r-- 1 ubuntu www-data    2905 Dec 16 17:50 access.log.6
total 9864
-rw-r--r-- 1 ubuntu www-data 9999911 Dec 16 17:50 access.log.7
-rw-r--r-- 1 ubuntu www-data    8921 Dec 16 17:50 access.log.6
-rw-r--r-- 1 ubuntu www-data   15460 Dec 16 17:50 access.log
-rw-r--r-- 1 ubuntu www-data   10313 Dec 16 17:51 access.log.2
-rw-r--r-- 1 ubuntu www-data    7699 Dec 16 17:51 access.log.3
-rw-r--r-- 1 ubuntu www-data   21471 Dec 16 17:51 access.log.4
-rw-r--r-- 1 ubuntu www-data    6874 Dec 16 17:51 access.log.5
-rw-r--r-- 1 ubuntu www-data   11989 Dec 16 17:51 access.log.1
total 9892
-rw-r--r-- 1 ubuntu www-data 9999911 Dec 16 17:50 access.log.7
-rw-r--r-- 1 ubuntu www-data    7699 Dec 16 17:51 access.log.3
-rw-r--r-- 1 ubuntu www-data   12849 Dec 16 17:51 access.log.1
-rw-r--r-- 1 ubuntu www-data   14936 Dec 16 17:51 access.log.6
-rw-r--r-- 1 ubuntu www-data   30068 Dec 16 17:51 access.log.4
-rw-r--r-- 1 ubuntu www-data   19755 Dec 16 17:51 access.log
-rw-r--r-- 1 ubuntu www-data   11170 Dec 16 17:51 access.log.5
-rw-r--r-- 1 ubuntu www-data   15466 Dec 16 17:51 access.log.2
total 9932

Is this a consequence of the gunicorn --workers=8?

Does logging.handlers.RotatingFileHandler not work with gunicorn workers?

I tried --worker-class=gevent as well with the same result.
msg326666 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-09-29 04:46
It's not Gunicorn-specific - more a case of multiple processes writing to a single log file. This is not supported directly, as documented here:

https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

As the cookbook says, you can set up a separate process to handle logging to file, which listens with a socket server or a multiprocessing QueueListener; your worker processes would communicate with it using either a SocketHandler or a QueueHandler, as appropriate.
msg346059 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-06-19 15:38
Closing, as per last comment.
History
Date User Action Args
2022-04-11 14:58:40adminsetgithub: 73187
2019-06-19 15:38:59vinay.sajipsetstatus: open -> closed
resolution: not a bug
messages: + msg346059

stage: patch review -> resolved
2018-09-29 04:49:11berker.peksagsetnosy: + berker.peksag
2018-09-29 04:46:43vinay.sajipsetmessages: + msg326666
2018-09-28 23:12:07cheryl.sabellasetnosy: + vinay.sajip

versions: + Python 3.8, - Python 2.7
2018-09-28 23:09:45pathakumeshsetkeywords: + patch
stage: patch review
pull_requests: + pull_request9027
2016-12-17 20:00:59Bruce Edgecreate