classification
Title: Make logging configuration files easier to use
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: vinay.sajip Nosy List: gjb1002, vinay.sajip
Priority: normal Keywords:

Created on 2009-05-28 07:09 by gjb1002, last changed 2009-07-30 07:30 by vinay.sajip. This issue is now closed.

Messages (16)
msg88467 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-05-28 07:09
Recently tried to use the logging configuration file format as
understood by logging.config.fileConfig() and found it very unwieldy for
normal usage. I feel it needs to "scale down" better. Some thoughts:

a) It creates handlers whether they're needed or not. This means you
cannot leave the handler sections present in your configuration file and
just disable and enable the logger by changing the level, or your
application will open the files/sockets etc. anyway, whether the loggers
are enabled or not. This is bordering on being a bug rather than just an
annoyance...

b) There is a lot of unnecessary cruft. For example it should be
possible to eliminate the [loggers], [handlers] and [formatters]
sections. I gather they're there due to a limitation of ConfigParser but
I assume this is historical as I can see no good reason for it now.
Also, entries should default sensibly, e.g. "qualname" should not be
treated as vital but should default to the name in the section header.

c) I'd also suggest providing a newer, non-back-compatible format
alongside the existing one that was less wordy for normal use. Get rid
of the separate "handlers" and "formatters" and make each logger have a
section of its own containing all "handling" and "formatting"
information: most users aren't going to want to think about these things
as separate entities and in any case there is the hierarchical mechanism
to prevent too much copying.

In fact, I'd suggest extending the "basicConfig" idea to be able to call
it on individual loggers, and build a new format around that, where each
section is read and a logger created with the contents passed as keyword
arguments to "basicConfig".

I might have time to do some of this myself but I want to be sure people
think it's a good idea before investing time in it.
msg88738 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-02 11:48
Re. point (a): The configuration logic assumes that if you put things in
the configuration file, you want them included - that means
instantiating handlers, which will generally open their output files
(uness the delay parameter is set) and sockets. If you use append
semantics for files and configure levels as you want them, the files are
opened but not written to (and closed when the handlers are closed or
the application exits). This is not a bug, as it was designed to work
like this.

Re. point (b): Agreed that the presence of the sections may not seem
necessary as you could achieve the same with loggers=, handlers= etc.
The sections were originally placed there as a place to hang other
settings which could apply across loggers, handlers etc. - which never
materialised. In any case, it's not a big problem and not worth breaking
backward compatibility for. The qualname is not always the same as the
value in the section header - see the documentation for an example of
this, search for "compiler.parser". The values in e.g. the section
header and the handlers= section are just names to allow the
configuration to cross-refer different parts of the configuration - so
you could use h1, h2 etc. as handler names.

Re. point (c): there are alternative ways of configuring logging; for
one example, see http://www.red-dove.com/python_config.html and look at
"logconfig.cfg" and "logconfig.py" in the linked tarball/zip.
msg88763 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-06-02 19:34
a) What is the point of opening files that will never be written to and
sockets that will never be used? 

A large application might have a great many loggers for debugging which
are off by default, and this means you have to either put up with lots
of empty files being created all the time, or tell everyone that they
need to change the configuration in two places each time they enable or
disable a logger.

Logging config files need to be easy to tweak, even for people who
aren't coders: it should be quick and obvious how to enable or disable a
logger.

b) I don't see why making those sections optional would break backward
compatibility. It would be easy to just silently ignore them if they
were present (or call today's code that uses them). 

I'm aware that "qualname" isn't always the same as the section name. My
point is that it should not be *compulsory*, not that it shouldn't
exist. It has an obvious default value so it's wrong to fail with a
python stack if it isn't present.

c) I know there are other packages out there. I've been using log4py for
years, which is now abandoned. But Python now has an official way to do
logging and I think it should be more user-friendly for simple usage
than it is. I can write my own config file format without too much
difficulty but it seems a shame if everyone ends up doing this.

(The one you linked to seemed to have wider ambitions than logging and
its format seemed even more unwieldy. Curly braces in Python?!)
msg88772 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-02 22:47
a) Loggers don't have a one-to-one correspondence to files that are
opened, so I don't quite understand your point. Perhaps you are
conflating loggers with handlers. The common way of using logging is to
have a small number of file-based handlers and perhaps a socket handler
and console handler, working with a potentially much larger number of
loggers.

b) If you want to propose a patch (incl. tests and docs) which maintains
backward compatibility, I'll certainly look at it.

the qualname *is* compulsory as it determines exactly which logger is
instantiated. So I don't understand your statement.

c) Having used log4py (which I'm not familiar with), it may be that you
have to consider that Python's logging package might do some things
differently. You are of course free to use your own configuration format
- it's one of those areas where personal taste is more of a factor. I
don't know of any alternative formats which have a lot of traction for
configuring logging specifically, though the programmatic API for
configuring logging is pretty simple and almost any other configuration
approach could be made to fit.

It's funny to hear you comment on the inappropriateness of curly braces
in Python, how do you manage without dict literals? ;-)

It's probably best not to continue this discussion on the issue tracker
- it's not the best place as it's not its intended purpose, and also the
audience here will be much smaller than, say, comp.lang.python. Before
rolling your own config format, I would suggest posting on c.l.py with
your difficulties with logging configuration and see how others have
coped with those or similar problems.
msg88820 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-06-03 18:08
OK, I'll take point (c) further on comp.lang.python. As for the others,
it would be useful if you could at least understand my points.

a) I'm aware that there isn't necessarily a one-to-one correspondence
between loggers and files, don't see why that's relevant. I have no idea
what the "common" way of using logging is, if there is one. Having lots
of files in a logging set up doesn't seem to me in any way unusual, even
if the number of loggers is potentially even larger.

The main question is the one I posed before: what is the point of
opening files that will never be written to and sockets that will never
be used? Or to put it another way, if I submitted a patch that only
created handlers that were used by at least one logger, would you look
at it?

b) "compulsory" as in "compulsory as an entry in the config file". The
code would be

if "qualname" in opts:
    qn = cp.get(sectname, "qualname")
else:
    qn = log

To make life easier for those of us who don't see the point in naming
loggers differently in the config file and in the code...
msg88865 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-04 07:02
Re point a) and opening files - why not use the "delay" parameter on
FileHandlers, which delays opening until a message is actually logged? I
can understand why one wouldn't want empty log files cluttering up the
place, but the same argument doesn't apply to sockets. If you don't want
an always-open connection, you could also use UDP. While the delay
parameter takes care of files, why does it matter so much about sockets?
Is it actually causing a practical problem, or does it just offend your
aesthetic sensibilities?

If you submit a patch for fileConfig that only created handlers that
were used by at least one logger, I would certainly look at it - I have
both accepted and rejected numerous patches over the years. Would your
patch also consider unused formatters? As the new behaviour could
conceivably create a backward-compatibility problem (some users may be
relying on the current behaviour, no matter how broken it seems to you),
the new behaviour should be accessible through another config setting
(e.g. ignore_unused) which defaults to False (the existing behaviour) if
not specified in the config file.

Re point b), it only really covers the case where you have single-level
loggers (qualname = word), and not actually a multi-level hierarchy
(qualname = words-separated-by-dots). So I don't feel it's worth doing
this, especially as there seems to be no widespread demand for this
functionality.

I'll mark the issue as pending for now.
msg88959 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-06-05 18:24
OK, I hadn't seen the "delay" parameter until now. I guess this is new
in Python 2.6? Good that there is already a way to avoid lots of empty
files, though it'll be a while before I can assume Python 2.6
unfortunately... that probably renders point (a) moot.

As for (b), do you not think a large number of users will not bother
with the hierarchical aspect of the logging framework? I'd say you need
to be pretty advanced/large scale before that becomes interesting.

I don't really understand why accepting such a patch would be a problem,
as it's a simple change that wouldn't break backwards compatibility.
It's surely got to be better than exiting with a python stack, which is
what happens today.

(To give an idea of the bloat-factor, since migrating to the logging
framework a typical configuration file for my system is now roughly 3
times the size it used to be for the same functionality)
msg88963 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-05 19:01
"As for (b), do you not think a large number of users will not bother
with the hierarchical aspect of the logging framework? I'd say you need
to be pretty advanced/large scale before that becomes interesting."

I disagree with this. The hierarchical set up is one of the key points
of the approach. Casual users may not use it, but there are a lot of
users who do - and not only on advanced or large-scale systems. Users wh
o don't need hierarchies don't have to use them, but the system has to
support those who find them useful. 

"I don't really understand why accepting such a patch would be a
problem, as it's a simple change that wouldn't break backwards
compatibility. It's surely got to be better than exiting with a python
stack, which is what happens today."

Any software might result in an exception if you don't interface to it
in the expected and documented way. And, in my previous comment, I
merely set some conditions which a patch would have to meet. Also, as it
is definitely the case that many users use the hierarchical feature,
even if you don't, I expect that the qualname setting will stay.

"(To give an idea of the bloat-factor, since migrating to the logging
framework a typical configuration file for my system is now roughly 3
times the size it used to be for the same functionality)"

How big is that in KB? Disk space is pretty cheap these days. If it is a
big problem for you, you can always try using your existing
configuration format, reading it yourself and using the programmatic API
to configure logging yourself. It should be a small bit of up-front work
which can then be used on all your future projects.
msg88968 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-06-05 19:40
Who said anything about not supporting users who want the hierarchy? I'm
talking about making "qualname" ****optional****, not removing it
entirely! I even supplied the entirety of the code (all 4 lines of it)
to be clear what I meant a few comments ago.

The files have gone from about 5kb to about 15kb. Of course diskspace is
not a problem in itself, but these files need to be read and edited by
non-coders and they're a lot scarier (and harder to tweak) than the old
ones were. Basically they're full of abstract technical concepts
("qualname", "handler") and bits of python code to be eval'ed. 

Yes, I can write my own format. But I can't see anything about my case
which is unusual and I'm sure there must be a demand for something
simpler, which is why I bothered to report this issue at all. 

It's not particularly hard to find people out there raising this if you
google a bit. But I shall raise this on comp.lang.python as you suggest.
msg89045 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-07 16:11
"Who said anything about not supporting users who want the hierarchy?
I'm talking about making "qualname" ****optional****, not removing it
entirely!"

Ok, I see - sorry for the misunderstanding on my part.

"... these files need to be read and edited by non-coders and they're a
lot scarier (and harder to tweak) than the old ones were. Basically
they're full of abstract technical concepts ("qualname", "handler") and
bits of python code to be eval'ed."

The configuration format is not (and was never) intended for
non-technical end users. Because of the way it works, typos in the
elements which are eval'd can throw exceptions, which is obviously
undesirable. If you want such users to be able to change the logging
configuration, I would advise you implement your own layer which does
not use any technical terminology such as "handler" or "qualname", and
from that layer either update the logging configuration via the logging
API, or write a standard logging config file.

"It's not particularly hard to find people out there raising this if you
google a bit."

Well, I did do a search for "+python +logging +config +problems" which
didn't throw up much (other than this issue). I'd be grateful for some
specific links to recent items which you have found.
msg89050 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-06-07 19:23
If it was never intended for end users, then perhaps you could rename
this request as "provide an end-user-friendly log configuration format"
(that would live alongside the current one). It's hardly unusual that
users should be able to troubleshoot systems themselves by requesting
more (or different) logging, is it?

For example, log4py's format looked like this:
[my_logger]
LogLevel: Normal
Target: my_filename.log

Pretty much anyone can edit a bunch of sections that look like that
(with optional extras such as formatting where appropriate).

------------------------

As for mentions of this issue online, I linked to one such on my
comp.lang.python posting. It isn't recent, but the points about the
config file still apply as it hasn't substantially changed since then as
far as I can see. 3 different users raise essentially the same point
(and you also contributed to this thread)

http://www.mechanicalcat.net/richard/log/Python/Simple_usage_of_Python_s_logging_module

Here's another (only some of the discussion is relevant, but this exact
point is raised by the original poster and agreed with by the responder):

http://markmail.org/message/amxocg2nskd72qaf
msg89063 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-08 00:56
"If it was never intended for end users, then perhaps you could rename
this request as "provide an end-user-friendly log configuration format"
(that would live alongside the current one). It's hardly unusual that
users should be able to troubleshoot systems themselves by requesting
more (or different) logging, is it?"

It's unlikely that this would be provided as part of the Python stdlib:
requirements which are truly end-user-friendly are also likely to be
application-specific. When the configuration format was devised I did
seek feedback - but people with strong opinions on the issue told me
they'd prefer to roll their own. I'm fine with that; you can't please
all of the people all of the time and I sadly can't devote much time to
this at the moment.

"For example, log4py's format looked like this:
[my_logger]
LogLevel: Normal
Target: my_filename.log"

Looks fine for logging to files only. Does log4py support the wide range
of output sinks that Python logging does? If not, then the more
complicated logging configuration would seem reasonable as there are
more options in Python logging.

"As for mentions of this issue online, I linked to one such on my
comp.lang.python posting."

I'm not sure which posting you mean - I didn't see any recent postings
by you on c.l.py (I look at it via the corresponding Google group).

Re. the links you posted - Richard Jones' thread came out shortly after
logging was released and was largely about documentation and the need
for casual-use functionality which was subsequently provided by
basicConfig(). The MarkMail thread which you linked to doesn't appear to
raise the same points you did - Steve Hindle's original post of 17 May
2006 mentions only in point 4 about "crufty config file syntax", with no
specifics, and I can only find 3 messages in that thread - 2 by Steve
Hindle and one by Aahz which simply suggests moving the discussion to
c.l.py. I checked the Baypiggies archive on mail.python.org - same thing.

My suggestion would be - if you want to promote a simpler config file
syntax for logging, implement one and upload it on PyPI, and post
announcements on c.l.py and c.l.py.announce. If there is demand for this
functionality, people can easily download your package from PyPI and use
it. After all, logging.config is a separate sub-module and need not be
loaded at all.
msg89161 - (view) Author: Geoffrey Bache (gjb1002) Date: 2009-06-09 19:47
My comp.lang.python thread is here:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/a0c35c3c9ad210a4

It was met by deafening silence though.

I don't see why a simple format would need to be customer-specific.
log4py's isn't/wasn't and that worked just fine. It did support logging
to more than files but its configuration file scaled down much more
gracefully for simple usage, mostly because it didn't expose its
internal design like the logging one does. It had only loggers instead
of loggers/handlers/formatters.

But log4py is discontinued now as a project and I can't face maintaining
my own copy of it any more.

I'm getting the feeling you're just trying to fob me off here. You
dismiss the threads I found as being "mostly about other things" or "not
mentioning specifics". That may be so, but the fact is, in those threads
you have five other people expressing in one way or another that the
configuration file is too complex - and I'm sure I could find more if
you really want. If you prefer to ignore them and me there's not much
point in discussing further.

I'm not demanding that you do this work. I'm simply trying to raise the
issue and asking you to consider accepting such a patch if I or somebody
else produce it.
msg89170 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-06-09 22:03
"It was met by deafening silence though."

Give it time - it's only been a few days. For some reason, Google Groups
doesn't show your post in the first page of results when I search for
logging configuration by date (i.e. most recent on top):

http://groups.google.com/group/comp.lang.python/search?q=logging+configuration&start=0&scoring=d

The same problem if I search for "logging configuration" the phrase -
again, it doesn't show up.

However, if after a while there's still not much response, it would
indicate that this is not perhaps such an important issue for the
community as you feel it is. This doesn't stop you from rolling your own
format, but gives less justification for adding a patch to the core stdlib.

"[log4py's] configuration file scaled down much more
gracefully for simple usage, mostly because it didn't expose its
internal design like the logging one does. It had only loggers instead
of loggers/handlers/formatters."

Yes - Python logging is more complex because that's what's useful for
developers. It's not really intended for end-users to change - in fact
once something is in production, typically only levels need to change.
This is surely editable by end users even with the existing config file
format, as long as they're not too fazed by the other stuff which they
don't need to touch. If they are - then a much simpler,
application-specific, end-user friendly format seems more in order.

"But log4py is discontinued now as a project and I can't face
maintaining my own copy of it any more."

What's to maintain? Python logging has been pretty stable now for a long
time, and log4py being simpler shouldn't need any particular maintenance
(since it has worked for you in the past).

"I'm getting the feeling you're just trying to fob me off here. You
dismiss the threads I found as being 'mostly about other things' or 'not
mentioning specifics'. That may be so, but the fact is, in those threads
you have five other people expressing in one way or another that the
configuration file is too complex - and I'm sure I could find more if
you really want. If you prefer to ignore them and me there's not much
point in discussing further."

I'm not trying to fob you off - I just stated what I found about those
posts. The complaints you refer to were not specific enough to suggest
improvements, and anyone can write comments about how crufty they think
something is - it doesn't exactly tell the maintainer which direction
they would like to go in. I'm not saying that applies to anything you
personally have said - I'm referring to the comments in those posts you
referred to. All of us in open source development have to balance a
number of different issues and we all have different agendas and
priorities. My position is that the logging configuration system, while
not perfect, works and is used by quite a lot of people without
problems. It's just not high on my list of priorities to tinker with the
format, because the feedback I've had in the past is that those people
who care a lot about configuration will roll their own anyway. I'm never
going to be able to please them all, so why not focus my energies elsewhere?

"I'm not demanding that you do this work. I'm simply trying to raise the
issue and asking you to consider accepting such a patch if I or somebody
else produce it."

As I've said before, I've accepted numerous patches from numerous people
in the past. You can confirm this from SVN where commit messages
generally refer to issue numbers on this tracker. Clearly I can't make
promises in advance to accept any future patch, but I've indicated where
I'd set the bar (backward compatibility, doc changes, test changes) for
a patch to be considered.
msg90531 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-07-15 08:32
There hasn't been any activity for around a month now - marking as
pending, will close shortly unless there's some reason not to.
msg91087 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-07-30 07:30
No feedback, closing.
History
Date User Action Args
2009-07-30 07:30:58vinay.sajipsetassignee: vinay.sajip
2009-07-30 07:30:43vinay.sajipsetstatus: pending -> closed

messages: + msg91087
2009-07-15 08:32:24vinay.sajipsetstatus: open -> pending
resolution: wont fix
messages: + msg90531
2009-06-09 22:03:38vinay.sajipsetmessages: + msg89170
2009-06-09 19:47:33gjb1002setstatus: pending -> open

messages: + msg89161
2009-06-08 00:56:39vinay.sajipsetstatus: open -> pending

messages: + msg89063
2009-06-07 19:23:14gjb1002setstatus: pending -> open

messages: + msg89050
2009-06-07 16:11:34vinay.sajipsetstatus: open -> pending

messages: + msg89045
2009-06-05 19:40:11gjb1002setstatus: pending -> open

messages: + msg88968
2009-06-05 19:01:32vinay.sajipsetstatus: open -> pending

messages: + msg88963
2009-06-05 18:24:59gjb1002setstatus: pending -> open

messages: + msg88959
2009-06-04 07:03:35vinay.sajipsetstatus: open -> pending

messages: + msg88865
2009-06-03 18:08:41gjb1002setmessages: + msg88820
2009-06-02 22:47:58vinay.sajipsetmessages: + msg88772
2009-06-02 19:34:16gjb1002setmessages: + msg88763
2009-06-02 11:48:41vinay.sajipsetmessages: + msg88738
2009-05-28 22:43:28pitrousetpriority: normal
nosy: + vinay.sajip

versions: + Python 2.7, Python 3.2
2009-05-28 07:09:58gjb1002create