This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Gzip old log files in rotating handlers
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vinay.sajip Nosy List: markgrandi, ramhux, vinay.sajip
Priority: normal Keywords: patch

Created on 2011-12-01 18:00 by ramhux, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
log.patch ramhux, 2011-12-01 18:00 patch to add gzip to rotating handlers review
Messages (13)
msg148732 - (view) Author: Raul Morales (ramhux) Date: 2011-12-01 18:00
Sometimes log files grow very quickly and consume too much disk space (e.g. DEBUG), so compress old log files saves disk space without losing the information from log files.

I propose to add a "gzip" or "compress" argument to RotatingFileHandler and TimedRotatingFileHandler to select the number of old files you want to compress.

For example, if you set the argument to 0 (default) no files are gzipped , but if you set it to 3 you get the following log files:
app.log
app.log.1
app.log.2
app.log.3.gz
app.log.4.gz
...
app.log.n.gz

For TimedRotatingFileHandler it works similar, gzipping from the n-th newer log file to the oldest log file:
app.log
app.log.2011-12-01
app.log.2011-11-30
app.log.2011-11-29.gz
app.log.2011-11-28.gz
...

A possible patch is attached
msg149254 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-11 22:42
Some people might want other compression methods, e.g. bz2, zip, lzma ...

Have you considered subclassing the existing handler classes as in the following example?

http://code.activestate.com/recipes/502265-timedcompressedrotatingfilehandler/

Possibly I could change the implementation in 3.3 to make this easier to do ... I will give it some thought.
msg149349 - (view) Author: Raul Morales (ramhux) Date: 2011-12-12 19:34
I use a similar code in my scripts, but I thought it could be useful to have this feature built into python.

If you prefer subclassing for compression, what about a compressing subclass built into logging package?

If you think it is a good feature, I will be able to work on it next week, and to add support for other formats.
msg149355 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-12 21:16
I have worked out a possible approach. I will post about it soon, and add a link to it from this issue.
msg149364 - (view) Author: Raul Morales (ramhux) Date: 2011-12-12 23:10
Interesting, then I will wait your post. Thanks.
msg149381 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-13 10:30
See this for the proposed resolution:

http://plumberjack.blogspot.com/2011/12/improved-flexibility-for-log-file.html
msg149469 - (view) Author: Raul Morales (ramhux) Date: 2011-12-14 20:16
I have just posted a comment, too.

http://plumberjack.blogspot.com/2011/12/improved-flexibility-for-log-file.html?showComment=1323891345946#c2875224484376643310

With this approach, anyone can implement support for any format easily. It is powerful. But IMHO, built-in support for Gzip and Zip formats would cover the basic needs of almost all platforms where python run.

Mix-in classes can be implemented in just a few lines of code, and it matches with your idea. Of course, IMHO.
msg149474 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-14 20:46
It seems that you agree that the basic mechanism is good enough to build on, so I'd rather leave the proposed stdlib change as is, but provide examples of how to achieve gzip/zip compression for rotated logs in the Logging Cookbook. The reason I don't want to add these in the stdlib is that I have to support anything I add indefinitely, so I don't want to add anything which will not be often used. This is the first time the issue has come up in all the years since the rotating file handlers have been around, so while I want to support your use case, I want to minimise stdlib changes and additions as much as I can.
msg149482 - (view) Author: Raul Morales (ramhux) Date: 2011-12-14 22:06
Ok, it is reasonable. It has no sense add support for compression since I am the only user who want it.

Maybe in the future ;)
msg151687 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-01-20 11:37
Refactoring in 57295c4d81ac supports this use case.
msg273049 - (view) Author: Mark Grandi (markgrandi) * Date: 2016-08-18 17:33
I just ran into this myself, and would challenge the notion that just because few people complain about the fact that built in compression isn't built in for logging handlers (such as TimedRotatingFileHandler), that it isn't a needed feature. This is such a common feature in pretty much everything that it would be nice to have it built in rather than having a custom RotatingHandler subclass that just compresses the logs and does nothing else in every project I work with.

While not python, i searched github for logback/slf4j (java logging framework) entries for their equivilent of TimedRotatingFileHandler and I found quite a few results:

https://github.com/search?utf8=%E2%9C%93&q=fileNamePattern%2F%3E*.gz+language%3AXML&type=Code&ref=searchresults

you don't even have to change the API of it, it could act the same way as the tarfile module, where if the 'filename' argument ends in one of "bz2", "gz" or 'xz" then it is compressed with that compression type, and if not, then it doesn't compress it (what it does now)

and, that is how logback/slf4j does it as well (http://logback.qos.ch/manual/appenders.html#fwrpFileNamePattern)
msg273050 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-08-18 18:00
You don't need a subclass - you can specify your own function to do it, as in the cookbook example:

https://docs.python.org/3/howto/logging-cookbook.html#using-a-rotator-and-namer-to-customize-log-rotation-processing
msg273058 - (view) Author: Mark Grandi (markgrandi) * Date: 2016-08-18 19:01
While I will say that is slightly easier than subclassing, it still requires a function, or a variation of that to be present in every project that wants to have a RotatingHandler + compression, and extra lines if you use logging.config.dictConfig, as you will need to reference an external function rather than just specifying the filename = *.gz

I don't see this this simple feature would add much maintainer overhead, as lzma.open, bz2.open and gz.open are all pretty simple to use and are all already in the stdlib
History
Date User Action Args
2022-04-11 14:57:24adminsetgithub: 57725
2016-08-18 19:01:09markgrandisetmessages: + msg273058
2016-08-18 18:00:55vinay.sajipsetmessages: + msg273050
2016-08-18 17:33:42markgrandisetnosy: + markgrandi
messages: + msg273049
2012-01-20 11:37:39vinay.sajipsetstatus: open -> closed
resolution: fixed
messages: + msg151687

stage: resolved
2011-12-14 22:06:48ramhuxsetmessages: + msg149482
2011-12-14 20:46:01vinay.sajipsetmessages: + msg149474
2011-12-14 20:16:49ramhuxsetmessages: + msg149469
2011-12-13 10:30:25vinay.sajipsetmessages: + msg149381
2011-12-12 23:10:30ramhuxsetmessages: + msg149364
2011-12-12 21:16:36vinay.sajipsetmessages: + msg149355
2011-12-12 19:34:29ramhuxsetmessages: + msg149349
2011-12-11 22:42:17vinay.sajipsetassignee: vinay.sajip
messages: + msg149254
2011-12-01 18:00:22ramhuxcreate