The error message from the title is not very intuitive from a user point of view. For instance, in the following traceback, only the first one gives a meaning to this error, through the leaked KeyError, thus the user knows that home_dir1 is invalid and missing. But the second message with "Bad value substitution" doesn't emphasize the actual problem and you have to stare at it for a couple of moments to figure out what's happening and why. Maybe changing the "Bad value substitution" with a "Missing value" would suffice. This came up in bpo-19546.

Traceback (most recent call last):
  File "C:\Python34\lib\configparser.py", line 410, in _interpolate_some
    v = map[var]
  File "C:\Python34\lib\collections\__init__.py", line 789, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "C:\Python34\lib\collections\__init__.py", line 781, in __missing__
    raise KeyError(key)
KeyError: 'home_dir1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\configparser.py", line 1204, in __getitem__
    return self._parser.get(self._name, key)
  File "C:\Python34\lib\configparser.py", line 773, in get
    d)
  File "C:\Python34\lib\configparser.py", line 374, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
  File "C:\Python34\lib\configparser.py", line 413, in _interpolate_some
    option, section, rest, var)
configparser.InterpolationMissingOptionError: Bad value substitution:
        section: [Paths]
        option : my_dir
        key    : home_dir1
        rawval : /lumberjack

Another problem with that error message is that it seems confusing as to where the bad value *reference* came from, given that rawval doesn't seem to include any interpolation point.

Those exception messages are really old. I agree we should fix them, we have to think about possible compatibility breakage though. I'll look into it.

So the diff would look like this (please find it attached). It does two things:

• changes messages on InterpolationMissingOptionError and InterpolationDepthError to be more helpful to the end user
• fixes rawval to actually hold the raw value of an option and not the "rest after substitution" like before

Fred, R. David, I have two questions for you:

• the reliable method of getting the arguments of both exceptions is the .args attribute; this didn't change but the message did. It's unlikely anybody would be parsing the old message string, but even if there was I'm inclined to treat any code doing so as broken by design. Do you agree?

• without the diff, the rawval argument in those exceptions holds a value that exposes internal implementation and is not generally useful for user code. It wasn't exposed directly as an attribute in those exceptions (section, option and reference are). That being said, fixing this is a change in logic of sorts. Do you see any danger of third-party code relying on the old behaviour?

Actually, I have a hypothetical third question:

• Should I make sure that those exceptions are unpicklable by older releases of Python 3?

I'm asking because if there's no such expectation, we could add .rawval as a direct attribute to InterpolationMissingOptionError, and introduce reference to InterpolationDepthError (currently the exception message can say that "option O in section S contains an interpolation key that cannot be substituted" but it doesn't say which interpolation key).

Anyway, the first two questions are most important because they basically decide whether we can change the exceptions at all at this point. I'm inclined to say "yes", Python 3 did that with a number of exceptions both built-in and in the standard library.

We never promise that the messages won't change (they are not part of the API), so that part isn't a problem. We do try to be backward compatible there when it comes to args beyond the message text. I don't think unpickleability is an issue; at least there isn't any discussion of it in PEP-3161, and I don't remember any around the implementation.

I think this is fine. I don't think we should worry about someone who is actually using rawval/rest pulled out of the exception args. You could put a porting note in What's New, but my guess is that the chances of anyone being inconvenienced by this are pretty near to zero.

I find the new error messages clear and straight to the point. It would be nice if this would get into 3.5.

LGTM - lukasz, do you want to commit this, or would you like someone else to if you're too busy? Looks like we should patch this in 3.4/3.5./3.6 at this point.

New changeset 267422f7c927 by Robert Collins in branch '3.4':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/267422f7c927

New changeset 1a144ff2d78b by Robert Collins in branch '3.5':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/1a144ff2d78b

New changeset fb4e67040779 by Robert Collins in branch 'default':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/fb4e67040779

I've applied this since it seems Lukasz was busy. Thanks for the patch Lukasz!

You forgot to merge default with 3.5.

Huh? I definitely did. I can see there's a extra head now, but I did the merge up per protocol locally.

New changeset 7191910aeb45 by Robert Collins in branch 'default':
Issue bpo-21159: Improve message in configparser.InterpolationMissingOptionError.
https://hg.python.org/cpython/rev/7191910aeb45

anyhow fixed

Thanks for handling this, I was indeed busy.

Reopening because we're starting to see regressions caused by the fix for this bug, e.g.: https://bugs.launchpad.net/configglue/+bug/1504288

The linked bug doesn't include any information as to what the observed failure mode(s) are. Can you provide details, please?

The failure is in the configglue test suite, but apparently also kazan and qutebrowser are also affected by this change. In the Launchpad bug there's a link to a librarian build log result.

The problem is that doing the .get() requires that subclasses implement a compatible API, which wasn't required previous to this change. But apparently adding the fallback argument to the subclass's .get() doesn't entirely fix the problem, it leads to other failures.

OK, so I guess it should be backed out in 3.4. But the since the patch does not change the signature of get, it seems like it is a legitimate change for 3.5. It is using the public API, after all.

Or are you saying the existing code is actually depending on the old value of rawval, which is what Łukasz was worried about?

By the way, I won't argue a lot if you say we should go for the strict backward compatibility view even in 3.5, I'm more raising the question.

This change is causing a problem for boto under 3.5.1 (works on 3.5.0):

TypeError: get() got an unexpected keyword argument 'raw'

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/configparser.py(406)_interpolate_some()
-> rawval = parser.get(section, option, raw=True, fallback=rest)

boto bug report: boto/boto#3433

This is because boto is subclassing ConfigParser and its get method does not include the raw argument. A quick search shows that this also affects Kazam am circus.

Circus fixed it by adding **kwargs to the method. circus-tent/circus@d0d2ac4

I think we should close this again: if you subclass something you need to implement the full public API. raw has been in that API since 2010!.

It's a shame that folk have been implementing just-enough, rather than the actual API, but I don't see that has all that much bearing on the go-forward.

Rolling it back out of 3.4 could be done - but since the problematic subclasses are fixing themselves now 3.5 is out, that should take care of issues on 3.4 too.

At this point I agree with Robert. It's public API that predates my changes which was inadequately mocked.

I've seen similar problems in the past when we've changed reading file objects to use plain iteration instead of the .readlines() method. I've had people complain that their "file-like object" implementation provided just .readlines() and suddenly stopped working.

I can relate to how this caused painful churn. We do try to minimize it. However, it's hard to evolve any code if we can't touch it based on fear of breaking incompletely duck typed third-party code :(

Barry, would you still like this to be reverted on Python 3.4?

Closing since there's apparently nothing actionable at this point.