classification
Title: Argument rules for callables do not apply when function implementation uses PyArg_ParseTuple
Type: behavior Stage:
Components: Documentation Versions: Python 3.0, Python 2.4, Python 2.6, Python 2.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: garden, georg.brandl, loewis, terry.reedy
Priority: normal Keywords:

Created on 2008-04-24 10:01 by garden, last changed 2008-04-27 09:40 by georg.brandl. This issue is now closed.

Messages (12)
msg65712 - (view) Author: Ludovico Gardenghi (garden) Date: 2008-04-24 10:01
(It seems strange to me that this issue hasn't been raised in the past,
maybe I just failed to find it in the BTS. In that case please excuse me
and please point me to the original discussion.)

The Language Reference, section 5.3.4, states that, for every callable
object:

"[...] If keyword arguments are present, they are first converted to
positional arguments, as follows. First, a list of unfilled slots is
created for the formal parameters. [...] Next, for each keyword
argument, the identifier is used to determine the corresponding slot (if
the identifier is the same as the first formal parameter name, the first
slot is used, and so on). [...]"

This is not true if the function is defined using the C function
PyArg_ParseTuple, and this happens a lot in the standard library. I
discovered it trying to call os.open("filename", flag=os.O_RDONLY), just
to make an example. In this case it seems useless to specify the
keyword, but I have to write a generic "wrapping" function that takes a
function name and its arguments (as keyword arguments) and call the
original function after having changed the content of some of the arguments.

Apart from the reason, I believe that this behavior is inconsistent with
the language definition and should be fixed. I'm very new to Python, but
maybe the format string of ParseTuple should be extended in order to
accept also the name of the positional arguments? Or something like
ParseTupleAndKeywords should be used instead?

I tried only on Python 2.4 and 2.5 but I took a look at the source code
of 2.6 and 3.0 and I believe the issue is still there.
msg65713 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-04-24 10:22
I fail to see the problem. The open function really doesn't have a named
parameter called flags; the positional parameters are unnamed. So there
is no violation of the language reference, AFAICT. Perhaps it would be
useful to point out that some parameters are available only for
positional passing, as they are unnamed.
msg65714 - (view) Author: Ludovico Gardenghi (garden) Date: 2008-04-24 10:35
I'd believe you when you say "positional parameters are unnamed", but:

- the language reference contains terms such as "first formal parameter
name". This means that positional parameters *may* have a name but may
also have no name?
- if you define a python function as "def f(a, b, c):" you can call it
using keyword arguments instead of positional (e.g. f(1, c=3, b=2)).

Could you please explain me what I'm still missing? (I repeat - I met
python for the first time 2 weeks ago, so it may very well be the case
that I'm completely wrong)
msg65715 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-04-24 10:50
> I'd believe you when you say "positional parameters are unnamed", but:
> 
> - the language reference contains terms such as "first formal parameter
> name". This means that positional parameters *may* have a name but may
> also have no name?

Correct (although that is actually my interpretation of the
implementation; it's probably not a feature of the language itself).

> - if you define a python function as "def f(a, b, c):" you can call it
> using keyword arguments instead of positional (e.g. f(1, c=3, b=2)).

Unnamed positional parameters are only available in C code. You cannot
write a function with unnamed parameters in Python.

> Could you please explain me what I'm still missing? (I repeat - I met
> python for the first time 2 weeks ago, so it may very well be the case
> that I'm completely wrong)

It's just how PyArg_ParseTuple works: it doesn't receive any parameter
names for the positional parameters, so it can't possibly do the
matching that the language prescribes for positional parameters.
Instead, PyArg_ParseTuple operates *just* by position, hence it
effectively implements unnamed positional parameters.

You are not completely wrong. It's just that this detail is something
most people recognize at some point and accept as a fact, regardless
of what the language specification says (and, as I claim, that text
isn't incorrect - or the implementation isn't incorrect -- it's just
underspecified, failing to mention a detail specific to CPython)
msg65716 - (view) Author: Ludovico Gardenghi (garden) Date: 2008-04-24 11:08
> You are not completely wrong. It's just that this detail is something
> most people recognize at some point and accept as a fact, regardless
> of what the language specification says (and, as I claim, that text
> isn't incorrect - or the implementation isn't incorrect -- it's just
> underspecified, failing to mention a detail specific to CPython)

Ok. I think I'll end up accepting it as a fact, too :) and work around
the issue. IMHO it would be perfectly acceptable to say "if you use
CPython and extend python with some C functions you must expect this
behavior", but it's slightly less acceptable that different modules from
the standard library have different behaviors (depending on which
language has been used to implement them):

- open(mode='r', name='filename') works
- os.open(flag=os.O_RDONLY, filename='filename') does not work
- calendar.weekday(2008, day=24, month=4) works
- math.fmod(x=10, y=3) does not work
- ...

From the point of view of someone who writes python code there should be
no difference between the behavior of these calls, as long as they are
included in the standard python library. IMHO, again.

Maybe yes, the easier but probably harmless solution is to change the
documentation and point out that "in general, you can't". Maybe this
somehow leans towards promoting a bug to the rank of feature? ;-)
msg65717 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-04-24 11:14
> Maybe yes, the easier but probably harmless solution is to change the
> documentation and point out that "in general, you can't". Maybe this
> somehow leans towards promoting a bug to the rank of feature? ;-)

The language spec is stuck between saying what the abstract Python
language is supposed to do, and describing what CPython precisely
does. You shouldn't use keyword arguments to pass non-optional
positional arguments, IMO, but the text describes precisely what
happens if you do - for Python functions. The more vague
specification then shouldn't say "you can't" (because that would
indicate that you get an exception when you try), but "it's
unspecified", then going on to say what CPython happens to do
in some release.
msg65718 - (view) Author: Ludovico Gardenghi (garden) Date: 2008-04-24 11:29
At present, "unspecified" is surely better than "you can't", that's a
good point. I understand the difficulties of balancing the reference
between the abstract definition and the actual implementation. But I
still believe that this should not be an unspecified behavior, either in
one direction or another. At least somewhere in the future.

Anyway, thanks for the explanation :)
msg65719 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-04-24 11:42
Making it a documentation issue; I don't think the implementation should
change. Georg, if you don't see the need for action, feel free to close it.
msg65820 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-04-25 22:20
I consider the difference between builtin and def'ed functions to be
something of an implementation wart -- one that I would like to see
someday removed if sensibly possible.  How is a beginner to know that
the parameter names used in the docs and help() responses are not really
parameter names?

In the meanwhile, I think something like the following in the doc would
help: "(Note: an implementation may provide builtin functions whose
positional parameters do not have names, even if they are 'named' for
the purpose of documentation, and which therefore cannot be supplied by
keyword.)"

Also in the meanwhile, the OP can def-wrap builtins
import builtins
def abs(number): return builtins.abs(number)
# but some like int require more care with its no-default option
msg65853 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-04-26 18:34
I'd love to add to the documentation, but I can't seem to find a proper
location - except the Tutorial?
msg65858 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-04-27 00:09
Ludovico quoted from LangReg 5.3.4 of old style doc.  The same paragraph
about keyword args is under Expressions/Primaries/Calls in the 2.6/3.0 docs
msg65882 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-04-27 09:40
Ah sorry, I should really read the OP next time.

Added a note in r62521.
History
Date User Action Args
2008-04-27 09:40:14georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg65882
2008-04-27 00:09:54terry.reedysetmessages: + msg65858
2008-04-26 18:34:19georg.brandlsetmessages: + msg65853
2008-04-25 22:20:38terry.reedysetnosy: + terry.reedy
messages: + msg65820
2008-04-24 11:42:10loewissetassignee: georg.brandl
messages: + msg65719
components: + Documentation, - Library (Lib)
nosy: + georg.brandl
2008-04-24 11:29:41gardensetmessages: + msg65718
2008-04-24 11:14:13loewissetmessages: + msg65717
2008-04-24 11:08:44gardensetmessages: + msg65716
2008-04-24 10:50:19loewissetmessages: + msg65715
2008-04-24 10:35:12gardensetmessages: + msg65714
2008-04-24 10:22:15loewissetnosy: + loewis
messages: + msg65713
2008-04-24 10:03:03gardensettitle: Argument rules in callables do not apply when function uses PyArg_ParseTuple -> Argument rules for callables do not apply when function implementation uses PyArg_ParseTuple
2008-04-24 10:01:23gardencreate