classification
Title: Add Python command line flags to configure logging
Type: enhancement Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: doughellmann, eric.araujo, eric.smith, exarkun, georg.brandl, michael.foord, ncoghlan, pitrou, r.david.murray, theller, vinay.sajip
Priority: normal Keywords: patch

Created on 2009-09-21 11:23 by theller, last changed 2011-02-04 23:34 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
logging.patch theller, 2009-09-21 11:23 Proof of concept
Messages (23)
msg92932 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-09-21 11:23
I want the Python executable to have command line flags which allow
simple configuration of the logging module.  Use cases are to run
applications/scripts (which use libraries that use logging calls) with
different logging output without having to edit (ugly) logging config files.

The attached patch demonstrates the idea; it allows to run 'python -l
<options> ...' and passes <options> to a new function
logging._parse_python_options(<options>).
msg92945 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-09-21 16:59
I get the idea. The Python part of the patch demonstrates what you're
getting at, though it can't be used as is - for example the
getattr(logging, a, a) could lead to problems. However a more
intelligent parser (which looked for specific keywords recognised by
basicConfig(), and got the correct values accordingly) wouldn't be much
more complicated. (I'll look at enhancing this part.)

As for the changes to main.c - I am a C/C++ developer but have not made
any changes to Python C code so far - it would be good if a more
experienced committer reviews this part (not sure who - can someone
please reassign/add to nosy list)? Thanks.
msg92946 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-09-21 17:15
> I get the idea. The Python part of the patch demonstrates what you're
> getting at, though it can't be used as is - for example the
> getattr(logging, a, a) could lead to problems. However a more
> intelligent parser (which looked for specific keywords recognised by
> basicConfig(), and got the correct values accordingly) wouldn't be much
> more complicated. (I'll look at enhancing this part.)
> 
> As for the changes to main.c - I am a C/C++ developer but have not made
> any changes to Python C code so far - it would be good if a more
> experienced committer reviews this part (not sure who - can someone
> please reassign/add to nosy list)? Thanks.

Both parts of the patch are only thought to demonstrate the idea.
You said you'll attack the Python part - good.

For the C part, the most prominemt things that are missing are these:

- free(logopt) should be called at the end of the 'if (logopts != NULL) {' block.

- error handling should be improved.  Errors in this block should probably exit Python.
msg92947 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-09-21 17:34
If you want to add flags to the main executable, it deserves a
discussion on python-dev IMO.
It could be obtained through an environment variable, e.g.
PYLOGGING_CONFIG; which has the nice side-effect of working for
executable scripts too.
msg92949 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2009-09-21 18:23
The C code looks basically okay to me. I'd probably use strdup() instead
of the malloc/strlen/strcpy calls. And as Thomas said, the cleanup needs
a little bit of work.
msg92950 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-09-21 18:31
Also possibly relevant was the recent python-dev discussion about
replicating (some of) the command line argument parsing in python so
that it can be used by things other than the 'python' command:

http://mail.python.org/pipermail/python-dev/2009-August/091484.html
msg92952 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-09-21 18:44
Also, don't forget to update the "python -h" help text.
msg92953 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2009-09-21 18:47
Why does this need to be built into the interpreter? The script / app
should have logging config support.
msg92956 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-09-21 20:16
> Why does this need to be built into the interpreter? The script / app
> should have logging config support.

It does not need to, but it would be nice.
I think the '-l' flag should be similar to the -W flag.
Or consider for example using unittest.main() as script/app.
msg92957 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-09-21 20:32
If we do include interpreter support, there should be an option to
invoke a configuration file, too:

-l config=path

Mutually exclusive with all the other options. So, you can either use it
to invoke basicConfig or to invoke an arbitrary configuration in a file.

Opinions?
msg92958 - (view) Author: Doug Hellmann (doughellmann) * (Python committer) Date: 2009-09-21 20:45
I think I'm with Michael on this one.  I'd rather add logging
configuration to any stdlib modules that support being run directly and
want to support logging.
msg92959 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-09-21 20:58
I don't think this is about logging in stdlib modules that are run
directly. I think this is about a library that contains logging calls
(eg: multiprocessing), and is used in j-random-application, and while
prototyping/debugging the application you want to access that logging
information without having to write a logging infrastructure into your
app first.  Or you want to access it while debugging someone _else_'s
application...
msg92960 - (view) Author: Jean-Paul Calderone (exarkun) * (Python committer) Date: 2009-09-21 21:02
How about putting this into the logging module instead?  Instead of
"python <logging options> <program> <program options>", making it
"python -m logging <logging options> <program> <program options".

This:

  * involves no changes to the core interpreter
  * only requires Python to be written, not C
  * Lets other VMs benefit from the feature immediately
  * Follows what seems to be the emerging convention for this kind of
(eg pdb, unittest)
  * Still allows logging to be configured for arbitrary programs that
wouldn't otherwise be configurable in this way
msg92961 - (view) Author: Doug Hellmann (doughellmann) * (Python committer) Date: 2009-09-21 21:19
How do these "global" settings (either via the interpreter or a wrapper
in the logging module) change what an app might do on its own?  IOW, if
my app is already written to configure logging, and someone invokes it
with these other settings, which settings are used?
msg92994 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-09-22 13:50
Both Doug and Jean-Paul have made good points, IMO.
msg93007 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-09-22 19:01
> Jean-Paul Calderone:
> 
> How about putting this into the logging module instead?  Instead of
> "python <logging options> <program> <program options>", making it
> "python -m logging <logging options> <program> <program options".
> 
> This:
> 
>   * involves no changes to the core interpreter
>   * only requires Python to be written, not C
>   * Lets other VMs benefit from the feature immediately
>   * Follows what seems to be the emerging convention for this kind of
> (eg pdb, unittest)
>   * Still allows logging to be configured for arbitrary programs that
> wouldn't otherwise be configurable in this way

This is a very good idea, and fully covers my use case.
msg93008 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-09-22 19:03
> How do these "global" settings (either via the interpreter or a wrapper
> in the logging module) change what an app might do on its own?  IOW, if
> my app is already written to configure logging, and someone invokes it
> with these other settings, which settings are used?

IMO the same would happen as if logging.basicConfig() would have been called
followed by calls to the custom configuration.
msg93010 - (view) Author: Doug Hellmann (doughellmann) * (Python committer) Date: 2009-09-22 19:30
@theller, I'm not sure what your point is.  I'm asking what the defined
behavior is if we provide some sort of global way to run a program with
logging configured, and then that app turns around and tries to
reconfigure it.  Should the last one to call the configuration
function(s) win, or the first?

I like the idea of adding this feature to the logging module better than
building it into the interpreter, but I still think it opens up areas
for unexpected behavior, and it would be better to just let each
application set up its own logging.
msg93012 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-09-22 19:45
@theller: Although your use case just covers using basicConfig, I can
just see users expecting the same mechanism to invoke configuration
stored in a logging configuration file, which is why I suggested the
config= variant. However, doughellmann raises the valid point of what
the semantics would be if the program you invoke via logging -m does its
own configuration.

With a simple implementation: If a script calls basicConfig after it has
been invoked with logging -m etc. then the basicConfig call won't do
anything, as if the root logger already has some handlers, the call is
essentially a no-op. If the called script loads a configuration file,
that will overwrite the configuration specified via logging -m.

Either way, it's not consistent IMO - sometimes the logging -m
configuration will seem to win, and other times, the called script's
configuration.

Of course, it could be argued that logging -m is intended for scripts
which don't do explicit configuration - I'm not sure of how strong an
argument this is.

Thinking caps on :-)
msg93013 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-09-22 19:46
Ummm, s/logging -m/-m logging/
msg93418 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-10-01 18:35
I retract this request.  It seems the idea is not liked or a solution is
not easy.

(The solution I now use is to start Python from a batch file that parses
some command line flags itself, sets environment variables, and my
sitecustomize.py file configures logging).

Thanks for the comments.
msg93425 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-10-01 22:46
Ok. I think it's a reasonable request, but I can see that implementing
it naively without some further thought will lead to (justifiable)
complaints from some users that the behaviour is contradictory or
unintuitive in terms of which configuration "wins" in the event of
ambiguity.
msg127952 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-02-04 23:34
I think Antoine’s envvar idea was a good one, similar to PYTHONWARNINGS.
History
Date User Action Args
2011-02-04 23:34:10eric.araujosetnosy: + eric.araujo
messages: + msg127952
2009-10-01 22:46:17vinay.sajipsetmessages: + msg93425
2009-10-01 18:35:02thellersetstatus: open -> closed
resolution: not a bug
messages: + msg93418
2009-09-22 19:46:44vinay.sajipsetmessages: + msg93013
2009-09-22 19:45:38vinay.sajipsetmessages: + msg93012
2009-09-22 19:30:31doughellmannsetmessages: + msg93010
2009-09-22 19:03:36thellersetmessages: + msg93008
2009-09-22 19:01:54thellersetmessages: + msg93007
2009-09-22 13:50:59vinay.sajipsetmessages: + msg92994
2009-09-21 21:19:00doughellmannsetmessages: + msg92961
2009-09-21 21:02:35exarkunsetnosy: + exarkun
messages: + msg92960
2009-09-21 20:58:34r.david.murraysetmessages: + msg92959
2009-09-21 20:45:52doughellmannsetmessages: + msg92958
2009-09-21 20:32:12vinay.sajipsetmessages: + msg92957
2009-09-21 20:16:46thellersetmessages: + msg92956
2009-09-21 18:47:37michael.foordsetnosy: + michael.foord
messages: + msg92953
2009-09-21 18:44:21georg.brandlsetnosy: + georg.brandl
messages: + msg92952
2009-09-21 18:31:00r.david.murraysetnosy: + r.david.murray, ncoghlan
messages: + msg92950
2009-09-21 18:23:01eric.smithsetmessages: + msg92949
2009-09-21 17:34:02pitrousetnosy: + pitrou
messages: + msg92947
2009-09-21 17:15:56thellersetmessages: + msg92946
2009-09-21 16:59:43vinay.sajipsetmessages: + msg92945
2009-09-21 12:21:57doughellmannsetnosy: + doughellmann
2009-09-21 12:15:38r.david.murraysetpriority: normal
nosy: + vinay.sajip

versions: + Python 2.7, Python 3.2
2009-09-21 12:07:32eric.smithsetnosy: + eric.smith
2009-09-21 11:23:16thellercreate