Sometimes log files grow very quickly and consume too much disk space (e.g. DEBUG), so compress old log files saves disk space without losing the information from log files.

I propose to add a "gzip" or "compress" argument to RotatingFileHandler and TimedRotatingFileHandler to select the number of old files you want to compress.

For example, if you set the argument to 0 (default) no files are gzipped , but if you set it to 3 you get the following log files:
app.log
app.log.1
app.log.2
app.log.3.gz
app.log.4.gz
...
app.log.n.gz

For TimedRotatingFileHandler it works similar, gzipping from the n-th newer log file to the oldest log file:
app.log
app.log.2011-12-01
app.log.2011-11-30
app.log.2011-11-29.gz
app.log.2011-11-28.gz
...

A possible patch is attached

Some people might want other compression methods, e.g. bz2, zip, lzma ...

Have you considered subclassing the existing handler classes as in the following example?

http://code.activestate.com/recipes/502265-timedcompressedrotatingfilehandler/

Possibly I could change the implementation in 3.3 to make this easier to do ... I will give it some thought.

I use a similar code in my scripts, but I thought it could be useful to have this feature built into python.

If you prefer subclassing for compression, what about a compressing subclass built into logging package?

If you think it is a good feature, I will be able to work on it next week, and to add support for other formats.

I have worked out a possible approach. I will post about it soon, and add a link to it from this issue.

Interesting, then I will wait your post. Thanks.

See this for the proposed resolution:

http://plumberjack.blogspot.com/2011/12/improved-flexibility-for-log-file.html

I have just posted a comment, too.

http://plumberjack.blogspot.com/2011/12/improved-flexibility-for-log-file.html?showComment=1323891345946#c2875224484376643310

With this approach, anyone can implement support for any format easily. It is powerful. But IMHO, built-in support for Gzip and Zip formats would cover the basic needs of almost all platforms where python run.

Mix-in classes can be implemented in just a few lines of code, and it matches with your idea. Of course, IMHO.

It seems that you agree that the basic mechanism is good enough to build on, so I'd rather leave the proposed stdlib change as is, but provide examples of how to achieve gzip/zip compression for rotated logs in the Logging Cookbook. The reason I don't want to add these in the stdlib is that I have to support anything I add indefinitely, so I don't want to add anything which will not be often used. This is the first time the issue has come up in all the years since the rotating file handlers have been around, so while I want to support your use case, I want to minimise stdlib changes and additions as much as I can.

Ok, it is reasonable. It has no sense add support for compression since I am the only user who want it.

Maybe in the future ;)

Refactoring in 57295c4d81ac supports this use case.

I just ran into this myself, and would challenge the notion that just because few people complain about the fact that built in compression isn't built in for logging handlers (such as TimedRotatingFileHandler), that it isn't a needed feature. This is such a common feature in pretty much everything that it would be nice to have it built in rather than having a custom RotatingHandler subclass that just compresses the logs and does nothing else in every project I work with.

While not python, i searched github for logback/slf4j (java logging framework) entries for their equivilent of TimedRotatingFileHandler and I found quite a few results:

https://github.com/search?utf8=%E2%9C%93&q=fileNamePattern%2F%3E*.gz+language%3AXML&type=Code&ref=searchresults

you don't even have to change the API of it, it could act the same way as the tarfile module, where if the 'filename' argument ends in one of "bz2", "gz" or 'xz" then it is compressed with that compression type, and if not, then it doesn't compress it (what it does now)

and, that is how logback/slf4j does it as well (http://logback.qos.ch/manual/appenders.html#fwrpFileNamePattern)

You don't need a subclass - you can specify your own function to do it, as in the cookbook example:

https://docs.python.org/3/howto/logging-cookbook.html#using-a-rotator-and-namer-to-customize-log-rotation-processing

While I will say that is slightly easier than subclassing, it still requires a function, or a variation of that to be present in every project that wants to have a RotatingHandler + compression, and extra lines if you use logging.config.dictConfig, as you will need to reference an external function rather than just specifying the filename = *.gz

I don't see this this simple feature would add much maintainer overhead, as lzma.open, bz2.open and gz.open are all pretty simple to use and are all already in the stdlib