classification
Title: Undocumented different between METH_KEYWORDS and **kws
Type: behavior Stage:
Components: Documentation Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: abacabadabacaba, alex, barry, docs@python, eric.smith, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-07-22 18:51 by barry, last changed 2016-05-16 07:12 by python-dev.

Files
File name Uploaded Description Edit
test_args_kwargs_types.patch serhiy.storchaka, 2016-05-06 07:51 review
ceval_kwargs_exact_dict.patch serhiy.storchaka, 2016-05-06 08:30 review
Messages (15)
msg193559 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-07-22 18:51
A colleague discovered an interesting implementation different between C-defined functions and Python-defined functions called with **kws arguments.  He tried to do this:

>>> from collections import defaultdict
>>> '{foo}{bar}'.format(**defaultdict(str))
''

He wondered how this could work, especially given:

>>> def what(**kws):
...   print(type(kws), repr(kws))
... 
>>> what(**defaultdict(str))
<class 'dict'> {}

The Language Reference clearly states that when what() is called, a new dictionary is create, which is populated by keyword arguments not consumed by positional args.

http://docs.python.org/3/reference/compound_stmts.html#function-definitions
http://docs.python.org/3/reference/expressions.html#calls

What isn't described is the behavior when the function is defined in C using METH_KEYWORDS.  In that case, the object passed through the ** argument is passed unscathed to the underlying methodobject defined to take METH_KEYWORDS.  str.format() is of the latter type so it gets the actual defaultdict.  what() of course gets the defined behavior.

It would be nice if the implementation details of the function did not leak into this behavior, but this is admittedly a corner case of the CPython implementation.  I haven't checked any of the alternative implementations to see what they do.  My guess is that CPython won't change for practical and backward compatibility reasons, thus I've opened this issue as a documentation bug.  At the very least, an implementation note in the Language Reference should be added.
msg193560 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2013-07-22 18:53
I'll confirm that PyPy raises a KeyError on the format() code.
msg264943 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-06 07:40
Yet one difference is that kwargs can be NULL in C function if no keyword arguments are supplied.

Here is a patch that adds tests for exact types of args and kwargs arguments in C functions.

I think that we should keep passing NULL as kwargs for performance reasons. But shouldn't we convert dict subclass to exact dict? I think there are good reasons to do this. Some functions can save kwargs for latter use. And this makes the implementation more consistent with PyPy.

If defaultdict behavior is needed for formatting, there is format_map().
msg264947 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-06 08:30
Proposed patch converts var-keyword arguments to exact dict. Since it changes the behavior of str.format() it can be pushed only in 3.6.
msg264979 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-06 15:21
Eric should chime in on the str.format() implications.
msg264982 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-05-06 15:41
I think it's okay to change this as far as str.format() goes.

Interestingly enough, the motivating case to add str.format_map() in issue 6081 was exactly this kind of code. Although I notice that the example in that issue, which it shows as failing, works in both 2.7 and 3.4 (the only versions I have handy). So I'm not sure what changed after 2009 to cause that code to start working, but I'd be okay with it breaking again. But maybe we should track it down, in case it was deliberately changed and is important to someone.

In any event, since str.format_map() is well-known (to me, at least!), and is the approved way of getting the behavior of passing a specific map in to the format machinery, I'm okay with changing **dict to create an exact new dict, at least as far as str.format() goes. I can't speak to the overall implications (such as other APIs and performance).

I agree it would be a 3.6 only change (and I've change the versions to reflect that).
msg264985 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-06 15:46
On May 06, 2016, at 03:41 PM, Eric V. Smith wrote:

>I agree it would be a 3.6 only change (and I've change the versions to
>reflect that).

The reason the other versions were included was because this was originally
reported as a documentation bug.  It should probably still be documented in
those older releases.
msg264994 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-05-06 16:52
Okay. Adding back 3.5 and 2.7, in case someone wants to document the behavior there. I'm not sure if we fix docs in 3.4 still.
msg265160 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-08 19:44
str.format() uses PyDict_GetItem() in 2.7, but PyObject_GetItem() in 3.x. A comment before:

        /* Use PyObject_GetItem instead of PyDict_GetItem because this
           code is no longer just used with kwargs. It might be passed
           a non-dict when called through format_map. */

Thus the behavior of str.format() was changed by accident.
msg265166 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-05-08 20:37
New changeset e4835b1ed7b1 by Serhiy Storchaka in branch 'default':
Issue #18531: Single var-keyword argument of dict subtype was passed
https://hg.python.org/cpython/rev/e4835b1ed7b1
msg265171 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-05-08 22:59
I'm not sure PyObject_GetItem instead of PyDict_GetItem in 3.x completely explains things, because the code in issue 6081 also works in 2.7.

But I don't think it's terribly important in any event, as long as we understand the 3.x difference. Thanks for researching!
msg265181 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-09 03:53
> I'm not sure PyObject_GetItem instead of PyDict_GetItem in 3.x completely
> explains things, because the code in issue 6081 also works in 2.7.

It doesn't work to me in current develop 2.7.11+. What version did you test?
msg265182 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-09 04:07
Now this is just the documentation issue. I think the change of the behavior in 3.6 should be documented too, but I even don't know where.
msg265190 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-05-09 12:13
Oops, sorry. I'm an idiot. I was running in a venv where python was python3. Checking with my installed 2.7.10, I see the expected error.
msg265674 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-05-16 07:12
New changeset e3283b4ae61d by Serhiy Storchaka in branch '3.5':
Backported tests for issue #18531.
https://hg.python.org/cpython/rev/e3283b4ae61d

New changeset 621e0a3249c2 by Serhiy Storchaka in branch '2.7':
Backported tests for issue #18531.
https://hg.python.org/cpython/rev/621e0a3249c2
History
Date User Action Args
2016-05-16 07:12:59python-devsetmessages: + msg265674
2016-05-09 12:13:23eric.smithsetmessages: + msg265190
2016-05-09 04:07:03serhiy.storchakasetmessages: + msg265182
2016-05-09 03:53:21serhiy.storchakasetmessages: + msg265181
2016-05-08 22:59:31eric.smithsetmessages: + msg265171
2016-05-08 20:37:07python-devsetnosy: + python-dev
messages: + msg265166
2016-05-08 19:44:43serhiy.storchakasetmessages: + msg265160
2016-05-06 16:52:37eric.smithsetmessages: + msg264994
versions: + Python 2.7, Python 3.5
2016-05-06 15:46:32barrysetmessages: + msg264985
2016-05-06 15:41:16eric.smithsetmessages: + msg264982
versions: - Python 2.7, Python 3.5
2016-05-06 15:21:26barrysetmessages: + msg264979
2016-05-06 15:20:45barrysetnosy: + eric.smith
2016-05-06 08:30:46serhiy.storchakasetfiles: + ceval_kwargs_exact_dict.patch

messages: + msg264947
2016-05-06 07:51:20serhiy.storchakasetfiles: + test_args_kwargs_types.patch
2016-05-06 07:51:07serhiy.storchakasetfiles: - test_args_kwargs_types.patch
2016-05-06 07:40:09serhiy.storchakasetfiles: + test_args_kwargs_types.patch

type: behavior
versions: + Python 3.5, Python 3.6, - Python 3.3, Python 3.4
keywords: + patch
nosy: + serhiy.storchaka

messages: + msg264943
2015-10-30 07:11:22abacabadabacabasetnosy: + abacabadabacaba
2013-07-22 18:53:42alexsetnosy: + alex
messages: + msg193560
2013-07-22 18:51:43barrycreate