This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: accept keyword arguments on most base type methods and builtins
Type: enhancement Stage: patch review
Components: Extension Modules, Interpreter Core Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Julien.Palard, Ramchandra Apte, berker.peksag, brian.curtin, eric.araujo, eric.snow, ezio.melotti, gregory.p.smith, gruszczy, martin.panter, meatballhat, methane, nchammas, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2010-05-13 19:10 by gregory.p.smith, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
8706.patch gruszczy, 2012-02-26 13:00 review
8706_2.patch gruszczy, 2012-02-26 19:29 review
Messages (25)
msg105652 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-05-13 19:10
C Python has a real wart in that standard types and library functions that are implemented in C do not always accept keyword arguments:

>>> 'xxxxxx'.find('xx', 4)
4
>>> 'xxxxxx'.find('xx', start=4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: find() takes no keyword arguments
>>> 

While other things do accept keywords:

sorted(s, key=bla)


We should clean this up.  It is not well documented anywhere and I suspect other python implementations (haven't tested this) may accept keywords on these where C Python doesn't.

In string.find()'s case it looks like this is because it is an old style C method declaration that only gets an args tuple, no keyword args dict.
msg154340 - (view) Author: Filip Gruszczyński (gruszczy) Date: 2012-02-26 13:00
I don't know if this is exactly what you want, but this is an early patch.
msg154379 - (view) Author: Filip Gruszczyński (gruszczy) Date: 2012-02-26 19:29
With tests.
msg154491 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-02-27 17:10
At a quick glance, I think you've got the right idea.  Fixing this involves a lot of PyArg_ParseTuple -> PyArg_ParseTupleAndKeywords upgrades all over the place.

Obviously there are are a wide range of things that can use updating for this so in order to keep things sane, it makes sense to do them in stages with individual patches for particular libraries.

I like that your 8706_2.patch chose to attack stringlib to start with as that covers a lot of ground with one piece of code and that you updated the tests to confirm that ordered args and keyword args both work.

This issue as filed doesn't have a well defined "end goal" to know when it is complete.  Another good thing would be to come up with a final goal.  Perhaps an audit of the stdlib and types to list which things do not properly accept keyword arguments?  That way there is a list of things to check off.  And if desired we could file sub-issues for various components rather than letting this one get huge.
msg154493 - (view) Author: Filip Gruszczyński (gruszczy) Date: 2012-02-27 17:17
Do you think I should put everything into a single patch or rather slowly add new patches with different methods or method groups?

I would rather split it into several patches, I think it is easier to manage them, especially that this one is quite huge already.
msg154505 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-02-27 19:28
Several patches for sure!  and give the patch files useful names
indicating which things they touch.
msg154506 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-02-27 20:11
On one hand I agree that it would be nice to get rid of these implementation details that prevent some C functions/methods to accept keyword args, but on the other hand I'm not sure that changing them all is the right thing to do.
For some functions/methods being able to pass keyword args make the code more readable/flexible, but for some other there's no real gain.
It also seems to me that since the arguments where only positional, not much thought went into choosing an appropriate name for these arguments.
For example str.join() is documented as str.join(iterable), and the C function calls the argument 'data'.
If we use those names, we won't have a chance to fix them later, so we should be careful before doing a mass-update.
msg154507 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2012-02-27 20:22
> For some functions/methods being able to pass keyword args make the code more readable/flexible, but for some other there's no real gain.

I know what you're saying with the last part, but I think everyone becomes a winner in the consistency game if we expose kwargs regardless of level of improvement compared to other places.
msg154509 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-02-27 20:31
FWIW PyPy doesn't seem to support keyword args for e.g. str.join() (so that's extra work for them too), and I don't see what would be the advantage of being able to do '-'.join(iterable=a_list).  Even if I also don't see a valid reason why that shouldn't work and realize that it might be surprising for someone, I'm not sure it's worth changing it just for the sake of being consistent everywhere.
msg154510 - (view) Author: Filip Gruszczyński (gruszczy) Date: 2012-02-27 20:36
I would stay away from methods that accept just a single argument. For those that accept more, I think it's reasonable to allow keyword args.
msg154522 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-02-28 02:03
For many builtins, this would be a total waste, slowing down the calls, and supplying keyword names that would be meaningless or just weird.  For example, chr(i=65) is just a waste.

Keyword args should only be added where they would add clarity to something that would otherwise be ambiguous.

FWIW, that is the reason that sorted() required a keyword argument.  It prevents confusion between key-functions and cmp-functions.
msg154524 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-02-28 02:24
See also issue14081 which got fixed.
msg154528 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-02-28 02:32
Can we have keyword arguments to range([start],stop,[step])?
msg154534 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-02-28 05:16
range() has been around 20+ years and there has been zero need for keyword arguments for it.

FWIW, the maxsplit keyword argument was a case where a keyword argument added clarity, and there may be a handful of other cases that are also warranted.  Greg Smith's idea for "start" for str.find() may be one of those.   

People are welcome to propose individual suggestions in separate tracker items; however, if someone wants a wholesale set of changes to core builtins, then they should discuss it on python-ideas.  It would also be worthwhile for someone to benchmark the performance cost of switching from "METH_O" to "METH_VARARGS | METH_KEYWORDS".
msg154725 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-03-02 04:16
I agree with Ezio and Raymond.  Tentatively editing the title to reflect the reduction in scope.
msg154743 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-03-02 07:23
restricting the scope of this makes sense.

also: just because an argument is listed in the docs with a name does not mean that that name is the most appropriate; part of adding keyword support should be choosing a sensible name.  Keyword arguments, when used, should increase the readability of code rather than add to confusion.

I intend to bring this up for a brief discussion at the language summit next week as representatives of all the Python VMs will be in the same room at once.  Goal: define the appropriate scope or at the very least non-scope.

As for performance and memory use, yes, it could have a small impact but it should not be large [worth measuring] and that seems like something we should fix in a more general way rather than by limiting the way methods can be called based on how a given VM is implemented.
msg154745 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-03-02 07:45
> also: just because an argument is listed in the docs with a name does 
> not mean that that name is the most appropriate; part of adding keyword 
> support should be choosing a sensible name. 

I agree, but other implementations might not have this limitation and might already use the name that appears in the documentation/docstring -- or even a better one.

> I intend to bring this up for a brief discussion at the language summit
> next week

+1
msg154748 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-03-02 07:52
I kicked off a discussion on python-ideas.  Lets take this there for the time being.
msg154749 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-03-02 08:26
See also #13386 for the doc part.
msg223570 - (view) Author: Julien Palard (Julien.Palard) Date: 2014-07-21 10:13
I think for some builtins it may be usefull to have keyword arguments, in the case they take more than one parameter.

Typically, it's impossible to write:

    self.drop_elements(partial(isinstance, type(lxml.etree.Comment)))

Because isinstance take its argument in the "other" order, we may bypass this using keywords arguments:


    self.drop_elements(partial(isinstance, type=type(lxml.etree.Comment)))

But isinstance refuses keyword arguments, so there is no way to write this without a lambda:

    self.drop_elements(lambda x: isinstance(x,
                       type(lxml.etree.Comment)))

With is cool and work, I agree, it's just an example to explicitly show why keywords argument may be cool: functools.partial.
msg241589 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-20 02:18
See also Issue 23738, which identifies some functions whose documentation already suggests they accept keywords. Perhaps these functions could be prioritized.

Also, I think “version changed” notices should be added in the documentation when a function grows support for keyword arguments.
msg241606 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-20 05:31
Supporting keyword arguments has performance loss. For fast builtins it make be significant. We should defer adding keyword arguments support until more efficient parsing will implemented. Note that it is easier to implement efficient argument parsing for functions with positional-only arguments (see for example issue23867).
msg241610 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2015-04-20 06:05
I wouldn't make an efficiency argument against it without trying it and
showing reproducible degradation in the hg.python.org/benchmarks suite.

On Sun, Apr 19, 2015, 10:31 PM Serhiy Storchaka <report@bugs.python.org>
wrote:

>
> Serhiy Storchaka added the comment:
>
> Supporting keyword arguments has performance loss. For fast builtins it
> make be significant. We should defer adding keyword arguments support until
> more efficient parsing will implemented. Note that it is easier to
> implement efficient argument parsing for functions with positional-only
> arguments (see for example issue23867).
>
> ----------
> nosy: +serhiy.storchaka
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue8706>
> _______________________________________
>
msg285924 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-20 21:32
Most overhead of supporting keyword arguments when pass positional arguments was removed in issue29029. But still there is an overhead of passing argument by keywords. It can be decreased when convert functions to Argument Clinic or use new private argument parsing API. This is not very easy issue.
msg285942 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-01-21 07:13
TL;DR

Changing positional argument name doesn't break backward compatibility.
After start accepting keyword argument, it's name is part of API and should be stable.

For example, document says `str.startswith(prefix[, start[, end]])`.
But this patch seems using `sub` instead of `prefix`.
https://docs.python.org/3.5/library/stdtypes.html#str.startswith

Keyword name should be chosen carefully, like choosing method name.
So I'm -1 to adding keyword argument support to many method in one issue.
History
Date User Action Args
2022-04-11 14:57:00adminsetgithub: 52952
2017-01-21 07:13:37methanesetnosy: + methane
messages: + msg285942
2017-01-20 21:32:09serhiy.storchakasetmessages: + msg285924
versions: + Python 3.7, - Python 3.5
2016-02-14 18:01:41nchammassetnosy: + nchammas
2016-02-14 10:51:23martin.panterlinkissue26334 superseder
2015-04-20 06:05:46gregory.p.smithsetmessages: + msg241610
2015-04-20 05:31:29serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg241606
2015-04-20 02:18:29martin.pantersetnosy: + martin.panter
messages: + msg241589
2015-04-19 23:53:27berker.peksagsetnosy: + berker.peksag

components: + Extension Modules, Interpreter Core
versions: + Python 3.5, - Python 3.3
2014-07-21 10:13:53Julien.Palardsetnosy: + Julien.Palard
messages: + msg223570
2012-03-02 08:26:57eric.araujosetmessages: + msg154749
2012-03-02 07:52:39gregory.p.smithsetmessages: + msg154748
2012-03-02 07:45:52ezio.melottisetmessages: + msg154745
2012-03-02 07:23:50gregory.p.smithsetmessages: + msg154743
2012-03-02 04:16:20eric.araujosetnosy: + eric.araujo

messages: + msg154725
title: accept keyword arguments on all base type methods and builtins -> accept keyword arguments on most base type methods and builtins
2012-02-28 05:16:05rhettingersetmessages: + msg154534
2012-02-28 02:32:13Ramchandra Aptesetmessages: + msg154528
2012-02-28 02:24:14Ramchandra Aptesetnosy: + Ramchandra Apte
messages: + msg154524
2012-02-28 02:03:44rhettingersetmessages: + msg154522
2012-02-28 01:37:06eric.snowsetnosy: + eric.snow
2012-02-27 20:36:26gruszczysetmessages: + msg154510
2012-02-27 20:31:39ezio.melottisetmessages: + msg154509
2012-02-27 20:22:05brian.curtinsetnosy: + brian.curtin
messages: + msg154507
2012-02-27 20:11:05ezio.melottisetnosy: + rhettinger
messages: + msg154506
2012-02-27 19:28:08gregory.p.smithsetmessages: + msg154505
2012-02-27 17:17:28gruszczysetmessages: + msg154493
2012-02-27 17:10:06gregory.p.smithsetmessages: + msg154491
2012-02-27 15:18:29ezio.melottisetnosy: + ezio.melotti
stage: patch review

versions: - Python 3.2
2012-02-26 19:29:31gruszczysetfiles: + 8706_2.patch

messages: + msg154379
2012-02-26 18:00:22Arfreversetnosy: + Arfrever
2012-02-26 13:00:37gruszczysetfiles: + 8706.patch

nosy: + gruszczy
messages: + msg154340

keywords: + patch
2010-05-14 01:45:31meatballhatsetnosy: + meatballhat
2010-05-13 19:10:53gregory.p.smithcreate