Issue26710
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2016-04-07 15:10 by Marc.Abramowitz, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (6) | |||
---|---|---|---|
msg262989 - (view) | Author: Marc Abramowitz (Marc.Abramowitz) * | Date: 2016-04-07 15:10 | |
My expectation was that any defaults I passed to ConfigParser when creating one would override values in the DEFAULT section of the config file. This is because I'd like the DEFAULT section to have the default values, but then I want to be able to override those with settings from environment variables. However, this is not the way it works. The defaults in the file take precedence over the defaults passed to the constructor. I didn't see a mention of this in the docs, but I might've missed it. Take this short program (`configparsertest.py`): ``` import configparser cp = configparser.ConfigParser({'foo': 'dog'}) print(cp.defaults()) cp.read('app.ini') print(cp.defaults()) ``` and this config file (`app.ini`): ``` [DEFAULT] foo = bar ``` I was expecting that I would see foo equal to dog twice, but what I get is: ``` $ python configparsertest.py OrderedDict([('foo', 'dog')]) OrderedDict([('foo', 'bar')]) ``` The reason that I want the programmatic default values to override the default values in the file is that I want the file to have low-precedence defaults that are used as a last resort, and I want to be able to override the defaults with the values from environment variables. As a concrete example, imagine that I have a config file for the stdlib `logging` module that looks something like this: ``` [DEFAULT] logging_logger_root_level = WARN ... [logger_root] level = %(logging_logger_root_level)s handlers = console ``` The desired behavior is that normally the app would use the WARN level for logging, but I'd like to be able to do something like: ``` $ LOGGING_LOGGER_ROOT_LEVEL=DEBUG python my_app.py ``` to get DEBUG logging. Maybe there is some other mechanism to accomplish this? |
|||
msg262990 - (view) | Author: SilentGhost (SilentGhost) * | Date: 2016-04-07 15:21 | |
You can override the level if an environmental variable was defined. Not sure why it needs to be responsibility of the ConfigParser. While I'm not going to immediately close this issue, I don't think such a backward-incompatible proposal is viable. |
|||
msg262992 - (view) | Author: Marc Abramowitz (Marc.Abramowitz) * | Date: 2016-04-07 15:41 | |
Some more info on the logging example I gave. So here is a program called `my_app.py`: ``` import os import logging.config logging.config.fileConfig('logging.ini', defaults=os.environ) logger = logging.getLogger(__name__) logger.debug('debug msg') logger.info('info msg') logger.warn('warn msg') logger.error('error msg') root_logger = logging.getLogger() print('root_logger.level = %d; logging.WARN = %d; logging.DEBUG = %d' % (root_logger.level, logging.WARN, logging.DEBUG)) ``` Note that it calls logging.config.fileConfig with defaults=os.environ so that environment variables can be used to affect the logging configuration. And here is `logging.ini`: ``` ### # logging configuration # http://docs.pylonsproject.org/projects/pyramid/en/1.7-branch/narr/logging.html ### [loggers] keys = root, fakeproject, sqlalchemy [handlers] keys = console [formatters] keys = generic [logger_root] level = %(logging_logger_root_level)s handlers = console [logger_fakeproject] level = DEBUG handlers = qualname = fakeproject [logger_sqlalchemy] level = INFO handlers = qualname = sqlalchemy.engine # "level = INFO" logs SQL queries. # "level = DEBUG" logs SQL queries and results. # "level = WARN" logs neither. (Recommended for production systems.) [handler_console] class = StreamHandler args = (sys.stderr,) level = NOTSET formatter = generic [formatter_generic] format = %(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s ``` Note that in the `logger_root` section of `logging.ini`, the variable `logging_logger_root_level` is referenced but not defined. For now, we are going to rely on getting that from an environment variable. If I provide an environment variable when running the program: ``` $ LOGGING_LOGGER_ROOT_LEVEL=DEBUG python my_app.py 2016-04-07 08:26:36,184 DEBUG [__main__:6][MainThread] debug msg 2016-04-07 08:26:36,184 INFO [__main__:7][MainThread] info msg 2016-04-07 08:26:36,184 WARNI [__main__:8][MainThread] warn msg 2016-04-07 08:26:36,184 ERROR [__main__:9][MainThread] error msg root_logger.level = 10; logging.WARN = 30; logging.DEBUG = 10 ``` then it works and the root logger level is DEBUG as expected. Great! But what happens if the user leaves out the environment variable? ``` $ python my_app.py Traceback (most recent call last): File "my_app.py", line 4, in <module> logging.config.fileConfig('logging.ini', defaults=os.environ) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/config.py", line 86, in fileConfig _install_loggers(cp, handlers, disable_existing_loggers) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/config.py", line 196, in _install_loggers level = cp.get(sectname, "level") File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 623, in get return self._interpolate(section, option, value, d) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 669, in _interpolate option, section, rawval, e.args[0]) ConfigParser.InterpolationMissingOptionError: Bad value substitution: section: [logger_root] option : level key : logging_logger_root_level rawval : %(logging_logger_root_level)s ``` An error occurs as expected. But I'd like to be able to provide a default value so that user doesn't have to set the environment variable, so let's add this to the top of `logging.ini`: ``` [DEFAULT] logging_logger_root_level = WARN ``` Now let's run the program again without the environment variable to see if that fixed the problem: ``` $ python my_app.py 2016-04-07 08:33:07,101 WARNI [__main__:8][MainThread] warn msg 2016-04-07 08:33:07,101 ERROR [__main__:9][MainThread] error msg root_logger.level = 30; logging.WARN = 30; logging.DEBUG = 10 ``` Awesome! It worked and set the root logger level to the default of WARN, as expected. Now what happens if we try to override the default with an environment variable? ``` $ LOGGING_LOGGER_ROOT_LEVEL=DEBUG python my_app.py 2016-04-07 08:33:56,047 WARNI [__main__:8][MainThread] warn msg 2016-04-07 08:33:56,048 ERROR [__main__:9][MainThread] error msg root_logger.level = 30; logging.WARN = 30; logging.DEBUG = 10 ``` Doh! The root logger level is still WARN. So the default in the ini file took precedence over what the user provided. This is unfortunate. So how does one provide defaults while also allowing overriding those defaults? |
|||
msg262993 - (view) | Author: Marc Abramowitz (Marc.Abramowitz) * | Date: 2016-04-07 15:43 | |
So I think changing the behavior of `defaults` might break backwards compatibility for folks who are relying on the old behavior. So I guess I would propose adding a new parameter called `overrides`. These would take precedence over `defaults` and that allows retaining backwards compatibility. |
|||
msg264682 - (view) | Author: Łukasz Langa (lukasz.langa) * | Date: 2016-05-02 23:19 | |
Having both `defaults` and `overrides` would confuse the API, especially given that we already have `fallback` arguments to `get*` methods. What's the point of having overrides *over* a file? Defaults were thought of as a way to say: "unless the user bothered to modify those, the values are as follows". That's the point of configuration. We have defaults, that the *user* overrides. Why would you want to override what the user specified? I fail to understand the logic behind this. |
|||
msg264685 - (view) | Author: Marc Abramowitz (Marc.Abramowitz) * | Date: 2016-05-02 23:47 | |
Well my thought is that the configuration file has defaults that a user may want to override at runtime using an environment variable or command-line switch. I guess as SilentGhost pointed out, maybe this is not the responsibility of the ConfigParser as this behavior is not related to config files per se, though it's curious that the ConfigParser allows the user to pass in values in the constructor. Since two people are -1 on this, it seems pretty clear that I am on the wrong track with this, so I think we can close this. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:29 | admin | set | github: 70897 |
2016-05-02 23:47:05 | Marc.Abramowitz | set | status: open -> closed resolution: rejected messages: + msg264685 |
2016-05-02 23:19:40 | lukasz.langa | set | messages: + msg264682 |
2016-04-07 15:43:22 | Marc.Abramowitz | set | messages: + msg262993 |
2016-04-07 15:41:54 | Marc.Abramowitz | set | messages: + msg262992 |
2016-04-07 15:21:54 | SilentGhost | set | nosy:
+ SilentGhost, lukasz.langa type: behavior -> enhancement messages: + msg262990 |
2016-04-07 15:10:26 | Marc.Abramowitz | create |