classification
Title: Gzip old log files in rotating handlers
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vinay.sajip Nosy List: ramhux, vinay.sajip
Priority: normal Keywords: patch

Created on 2011-12-01 18:00 by ramhux, last changed 2012-01-20 11:37 by vinay.sajip. This issue is now closed.

Files
File name Uploaded Description Edit
log.patch ramhux, 2011-12-01 18:00 patch to add gzip to rotating handlers review
Messages (10)
msg148732 - (view) Author: Raul Morales (ramhux) Date: 2011-12-01 18:00
Sometimes log files grow very quickly and consume too much disk space (e.g. DEBUG), so compress old log files saves disk space without losing the information from log files.

I propose to add a "gzip" or "compress" argument to RotatingFileHandler and TimedRotatingFileHandler to select the number of old files you want to compress.

For example, if you set the argument to 0 (default) no files are gzipped , but if you set it to 3 you get the following log files:
app.log
app.log.1
app.log.2
app.log.3.gz
app.log.4.gz
...
app.log.n.gz

For TimedRotatingFileHandler it works similar, gzipping from the n-th newer log file to the oldest log file:
app.log
app.log.2011-12-01
app.log.2011-11-30
app.log.2011-11-29.gz
app.log.2011-11-28.gz
...

A possible patch is attached
msg149254 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-11 22:42
Some people might want other compression methods, e.g. bz2, zip, lzma ...

Have you considered subclassing the existing handler classes as in the following example?

http://code.activestate.com/recipes/502265-timedcompressedrotatingfilehandler/

Possibly I could change the implementation in 3.3 to make this easier to do ... I will give it some thought.
msg149349 - (view) Author: Raul Morales (ramhux) Date: 2011-12-12 19:34
I use a similar code in my scripts, but I thought it could be useful to have this feature built into python.

If you prefer subclassing for compression, what about a compressing subclass built into logging package?

If you think it is a good feature, I will be able to work on it next week, and to add support for other formats.
msg149355 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-12 21:16
I have worked out a possible approach. I will post about it soon, and add a link to it from this issue.
msg149364 - (view) Author: Raul Morales (ramhux) Date: 2011-12-12 23:10
Interesting, then I will wait your post. Thanks.
msg149381 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-13 10:30
See this for the proposed resolution:

http://plumberjack.blogspot.com/2011/12/improved-flexibility-for-log-file.html
msg149469 - (view) Author: Raul Morales (ramhux) Date: 2011-12-14 20:16
I have just posted a comment, too.

http://plumberjack.blogspot.com/2011/12/improved-flexibility-for-log-file.html?showComment=1323891345946#c2875224484376643310

With this approach, anyone can implement support for any format easily. It is powerful. But IMHO, built-in support for Gzip and Zip formats would cover the basic needs of almost all platforms where python run.

Mix-in classes can be implemented in just a few lines of code, and it matches with your idea. Of course, IMHO.
msg149474 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-14 20:46
It seems that you agree that the basic mechanism is good enough to build on, so I'd rather leave the proposed stdlib change as is, but provide examples of how to achieve gzip/zip compression for rotated logs in the Logging Cookbook. The reason I don't want to add these in the stdlib is that I have to support anything I add indefinitely, so I don't want to add anything which will not be often used. This is the first time the issue has come up in all the years since the rotating file handlers have been around, so while I want to support your use case, I want to minimise stdlib changes and additions as much as I can.
msg149482 - (view) Author: Raul Morales (ramhux) Date: 2011-12-14 22:06
Ok, it is reasonable. It has no sense add support for compression since I am the only user who want it.

Maybe in the future ;)
msg151687 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-01-20 11:37
Refactoring in 57295c4d81ac supports this use case.
History
Date User Action Args
2012-01-20 11:37:39vinay.sajipsetstatus: open -> closed
resolution: fixed
messages: + msg151687

stage: resolved
2011-12-14 22:06:48ramhuxsetmessages: + msg149482
2011-12-14 20:46:01vinay.sajipsetmessages: + msg149474
2011-12-14 20:16:49ramhuxsetmessages: + msg149469
2011-12-13 10:30:25vinay.sajipsetmessages: + msg149381
2011-12-12 23:10:30ramhuxsetmessages: + msg149364
2011-12-12 21:16:36vinay.sajipsetmessages: + msg149355
2011-12-12 19:34:29ramhuxsetmessages: + msg149349
2011-12-11 22:42:17vinay.sajipsetassignee: vinay.sajip
messages: + msg149254
2011-12-01 18:00:22ramhuxcreate