classification
Title: ConfigParser: Values in DEFAULT section override defaults passed to constructor
Type: enhancement Stage:
Components: Library (Lib) Versions:
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: Marc.Abramowitz, SilentGhost, lukasz.langa
Priority: normal Keywords:

Created on 2016-04-07 15:10 by Marc.Abramowitz, last changed 2016-05-02 23:47 by Marc.Abramowitz. This issue is now closed.

Messages (6)
msg262989 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2016-04-07 15:10
My expectation was that any defaults I passed to ConfigParser when creating one would override values in the DEFAULT section of the config file. This is because I'd like the DEFAULT section to have the default values, but then I want to be able to override those with settings from environment variables.

However, this is not the way it works. The defaults in the file take precedence over the defaults passed to the constructor. I didn't see a mention of this in the docs, but I might've missed it.

Take this short program (`configparsertest.py`):

```
import configparser

cp = configparser.ConfigParser({'foo': 'dog'})
print(cp.defaults())
cp.read('app.ini')
print(cp.defaults())
```

and this config file (`app.ini`):

```
[DEFAULT]
foo = bar
```

I was expecting that I would see foo equal to dog twice, but what I get is:

```
$ python configparsertest.py
OrderedDict([('foo', 'dog')])
OrderedDict([('foo', 'bar')])
```

The reason that I want the programmatic default values to override the default values in the file is that I want the file to have low-precedence defaults that are used as a last resort, and I want to be able to override the defaults with the values from environment variables.

As a concrete example, imagine that I have a config file for the stdlib `logging` module that looks something like this:

```
[DEFAULT]
logging_logger_root_level = WARN
...
[logger_root]
level = %(logging_logger_root_level)s
handlers = console
```

The desired behavior is that normally the app would use the WARN level for logging, but I'd like to be able to do something like:

```
$ LOGGING_LOGGER_ROOT_LEVEL=DEBUG python my_app.py
```

to get DEBUG logging.

Maybe there is some other mechanism to accomplish this?
msg262990 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2016-04-07 15:21
You can override the level if an environmental variable was defined. Not sure why it needs to be responsibility of the ConfigParser.

While I'm not going to immediately close this issue, I don't think such a backward-incompatible proposal is viable.
msg262992 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2016-04-07 15:41
Some more info on the logging example I gave.

So here is a program called `my_app.py`:

```
import os
import logging.config

logging.config.fileConfig('logging.ini', defaults=os.environ)
logger = logging.getLogger(__name__)
logger.debug('debug msg')
logger.info('info msg')
logger.warn('warn msg')
logger.error('error msg')
root_logger = logging.getLogger()
print('root_logger.level = %d; logging.WARN = %d; logging.DEBUG = %d'
      % (root_logger.level, logging.WARN, logging.DEBUG))
```

Note that it calls logging.config.fileConfig with defaults=os.environ so that environment variables can be used to affect the logging configuration.


And here is `logging.ini`:

```
###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.7-branch/narr/logging.html
###

[loggers]
keys = root, fakeproject, sqlalchemy

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = %(logging_logger_root_level)s
handlers = console

[logger_fakeproject]
level = DEBUG
handlers =
qualname = fakeproject

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither.  (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s
```

Note that in the `logger_root` section of `logging.ini`, the variable `logging_logger_root_level` is referenced but not defined. For now, we are going to rely on getting that from an environment variable.


If I provide an environment variable when running the program:

```
$ LOGGING_LOGGER_ROOT_LEVEL=DEBUG python my_app.py
2016-04-07 08:26:36,184 DEBUG [__main__:6][MainThread] debug msg
2016-04-07 08:26:36,184 INFO  [__main__:7][MainThread] info msg
2016-04-07 08:26:36,184 WARNI [__main__:8][MainThread] warn msg
2016-04-07 08:26:36,184 ERROR [__main__:9][MainThread] error msg
root_logger.level = 10; logging.WARN = 30; logging.DEBUG = 10
```

then it works and the root logger level is DEBUG as expected. Great!

But what happens if the user leaves out the environment variable?

```
$ python my_app.py
Traceback (most recent call last):
  File "my_app.py", line 4, in <module>
    logging.config.fileConfig('logging.ini', defaults=os.environ)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/config.py", line 86, in fileConfig
    _install_loggers(cp, handlers, disable_existing_loggers)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/config.py", line 196, in _install_loggers
    level = cp.get(sectname, "level")
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 623, in get
    return self._interpolate(section, option, value, d)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 669, in _interpolate
    option, section, rawval, e.args[0])
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
	section: [logger_root]
	option : level
	key    : logging_logger_root_level
	rawval : %(logging_logger_root_level)s
```

An error occurs as expected. 

But I'd like to be able to provide a default value so that user doesn't have to set the environment variable, so let's add this to the top of `logging.ini`:

```
[DEFAULT]
logging_logger_root_level = WARN
```

Now let's run the program again without the environment variable to see if that fixed the problem:

```
$ python my_app.py
2016-04-07 08:33:07,101 WARNI [__main__:8][MainThread] warn msg
2016-04-07 08:33:07,101 ERROR [__main__:9][MainThread] error msg
root_logger.level = 30; logging.WARN = 30; logging.DEBUG = 10
```

Awesome! It worked and set the root logger level to the default of WARN, as expected.

Now what happens if we try to override the default with an environment variable?

```
$ LOGGING_LOGGER_ROOT_LEVEL=DEBUG python my_app.py
2016-04-07 08:33:56,047 WARNI [__main__:8][MainThread] warn msg
2016-04-07 08:33:56,048 ERROR [__main__:9][MainThread] error msg
root_logger.level = 30; logging.WARN = 30; logging.DEBUG = 10
```

Doh! The root logger level is still WARN. So the default in the ini file took precedence over what the user provided. This is unfortunate.

So how does one provide defaults while also allowing overriding those defaults?
msg262993 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2016-04-07 15:43
So I think changing the behavior of `defaults` might break backwards compatibility for folks who are relying on the old behavior.

So I guess I would propose adding a new parameter called `overrides`. These would take precedence over `defaults` and that allows retaining backwards compatibility.
msg264682 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2016-05-02 23:19
Having both `defaults` and `overrides` would confuse the API, especially given that we already have `fallback` arguments to `get*` methods.

What's the point of having overrides *over* a file? Defaults were thought of as a way to say: "unless the user bothered to modify those, the values are as follows". That's the point of configuration. We have defaults, that the *user* overrides. Why would you want to override what the user specified? I fail to understand the logic behind this.
msg264685 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2016-05-02 23:47
Well my thought is that the configuration file has defaults that a user may want to override at runtime using an environment variable or command-line switch.

I guess as SilentGhost pointed out, maybe this is not the responsibility of the ConfigParser as this behavior is not related to config files per se, though it's curious that the ConfigParser allows the user to pass in values in the constructor.

Since two people are -1 on this, it seems pretty clear that I am on the wrong track with this, so I think we can close this.
History
Date User Action Args
2016-05-02 23:47:05Marc.Abramowitzsetstatus: open -> closed
resolution: rejected
messages: + msg264685
2016-05-02 23:19:40lukasz.langasetmessages: + msg264682
2016-04-07 15:43:22Marc.Abramowitzsetmessages: + msg262993
2016-04-07 15:41:54Marc.Abramowitzsetmessages: + msg262992
2016-04-07 15:21:54SilentGhostsetnosy: + SilentGhost, lukasz.langa
type: behavior -> enhancement
messages: + msg262990
2016-04-07 15:10:26Marc.Abramowitzcreate