classification
Title: json functions have too many positional parameters
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, bob.ippolito, ezio.melotti, gvanrossum, pitrou, python-dev, r.david.murray, rhettinger, serhiy.storchaka
Priority: low Keywords: patch

Created on 2013-08-13 11:18 by serhiy.storchaka, last changed 2016-07-01 06:10 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
json_keyword_only.patch serhiy.storchaka, 2016-05-06 16:43 review
Messages (17)
msg195068 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-08-13 11:18
The json module functions have too many positional parameters:

dump() -- 11
dumps() -- 10
load() -- 9
loads() -- 8

In most time only small part of these options is specified so users call these functions with keyword arguments for all parameters except mandatory ones.

Even worse, the simplejson module (an ancestor and an alternative to standard json module) have a very similar interface but with difference sequences of parameters (the second parameter of loads() and the ninth parameter of dumps() in simplejson is encoding). So in any case portable application should specify all but basic arguments as keyword arguments. If json will incorporate some features from simplejson in future positions of new parameters will be different.

I propose to deprecate specifying all but mandatory parameters of json functions as positional arguments in 3.4 and then forbid it in 3.5.

I.e. dumps() should be implemented in 3.4 as:

def dumps(obj, *args, **kwargs):
    if args:
        warnings.warn("The positional arguments are deprecated.",
                     DeprecationWarning, stacklevel=2)
    return _dumps(obj, *args, **kwargs)

def _dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True,
        allow_nan=True, cls=None, indent=None, separators=None,
        default=None, sort_keys=False, **kwargs):
    ...

and in 3.5 as:

def dumps(obj, *, skipkeys=False, ensure_ascii=True, check_circular=True,
        allow_nan=True, cls=None, indent=None, separators=None,
        default=None, sort_keys=False, **kwargs):
    ...
msg195070 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-08-13 12:46
This is not what we use keyword only arguments for.  The standard practice in the stdlib is that arguments are arguments unless there is a good reason to make one keyword only.  So I'm -1 on this proposal.
msg195071 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-08-13 12:57
Serhiy has a point though, I wouldn't want to see something like json.dumps(someobj, True, False, True, False).
msg195072 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-08-13 13:04
Ach.  I didn't read carefully enough (not awake yet, I guess).

Yes, boolean parameters are one of the things keyword only arguments are appropriate for.
msg196712 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2013-09-01 06:46
-1 Once the positional arguments are in the wild, you can't take them away without breaking a lot of code. 

This code was available as a third-party module prior to inclusion in Python and had many happy users.  If the positional arguments had been a sticking point, it would have been known long ago.

The time to express these concerns is when a module is being added.  Afterwards, the API churn just isn't worth it.  We don't want to create more obstacles to upgrading or converting from Python 2.
msg196717 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-01 08:44
> We don't want to create more obstacles to upgrading or converting from Python 2.

But these obstacles already exists.

The 10th parameter of dump() is "encoding" in Python 2 and simplejson, and "default" in Python 3.

The 12th parameter of dump() is "use_decimal" simplejson, and "sort_keys" in Python 2.

The 2nd parameter of load() is "encoding" in Python 2 and simplejson, and "cls" in Python 3.

Due to the fact that most arguments are boolean or accepts None some shifting of positional arguments can left unnoticed long time and causes subtle bugs when upgrading from Python 2 to Python 3 or switching between stdlib json module and simplejson.

Proposed change doesn't break any code at first stage, it just adds deprecation warning about using dangerous non-portable feature. We can defer forbidding positional parameters until Python 2 will gone and simplejson will be changed to keyword-only parameters too.
msg196747 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2013-09-01 22:46
Serhiy, please resist the urge to engage in API churn.  We've had zero user requests to abandon the current API.  I'm not a fan of the current positional arguments, but changing it isn't really worth it (creating disruption with nearly zero benefit, no new functionality and possibly slowing down the calls in module where people seem to really care about speed).

Bob, would you please weigh-in on this one.  You have the best appreciation for the needs of the established user base.
msg196751 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2013-09-01 23:10
I tend to agree with Raymond here. While I also don't like the positional arguments, I don't think the benefit of API churn them is high enough to remove them. Other than this issue, I have not seen any requests for this change.

I have seen problems because there are positional arguments, but only when people are subclassing these classes. Allowing for subclasses was never a good idea, and is actively discouraged in current simplejson documentation. The dumps default/loads object_hook functionality is sufficient and doesn't have the fragile base class problem. This functionality is is fully compatible with all versions of the library that are in use, back to v1.8 in March 2008, including Python 2.6's json library which IIRC is based on v1.9.
msg196791 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-02 17:12
But what you will do with discrepancies between Python 2, Python 3 and simplejson?
msg196792 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-02 17:15
> I have not seen any requests for this change.

That's probably because either people have not realized that their code might be buggy, or because they have not realized that keyword-only args might be a solution to this problem (or maybe they did but didn't bother suggesting it).
msg196799 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2013-09-02 18:11
Other than when subclassing (which is actively discouraged), I haven't seen anyone try and use positional args for these APIs. I simply don't think this is an issue in practice. If you go far enough back in simplejson history, these module-global functions were actually keyword-only.
msg196805 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-02 20:49
If no one uses positional arguments then converting parameters to keyword-only will not break any code.
msg196806 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2013-09-02 20:57
My evidence is only anecdotal. I haven't actively searched for code that uses json/simplejson to try and find cases that are using positional arguments.

One way to test this assumption would be to release a version of simplejson that deprecates using positional arguments and see if anyone notices. If you feel strongly about it, submit a pull request, and I'll review it (although my availability is spotty for the next two months due to travel).
msg264991 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-06 16:43
Guido's decision on similar issue25628 is that changing keyword-or-positional parameters to keyword-only is safe and can be made without deprecation.

If we started the deprecation period in 3.4, we could finish it in 3.6.

Proposed simple patch changes all optional parameters in functions and constructors that takes too much parameters in the json module to keyword-only without deprecation.
msg264993 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-06 16:51
> Guido's decision on similar issue25628 is that changing keyword-or-positional parameters to keyword-only is safe and can be made without deprecation.

> If we started the deprecation period in 3.4, we could finish it in 3.6.

That makes little sense. You can't start deprecation in 3.4 or in 3.5 since these have been released already.

But what I said implies that you can make these things mandatory in 3.6 without having to worry about deprecating them, so the start of the deprecation period is meaningless.

However I do encourage you to be conservative, and add a warning about this and similar cases when the alphas and betas go out; and if you get much pushback during beta consider changing your mind.
msg264996 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-06 16:58
My remark was just about that I proposed to start a deprecation period at the time of developing Python 3.4. If this did accepted, the deprecation period would already finished and we would come up to the same code in 3.6.

It doesn't matter now.
msg269024 - (view) Author: Roundup Robot (python-dev) Date: 2016-06-21 21:03
New changeset db5fe5c4d09d by Serhiy Storchaka in branch 'default':
Issue #18726: All optional parameters of the dump(), dumps(),
https://hg.python.org/cpython/rev/db5fe5c4d09d
History
Date User Action Args
2016-07-01 06:10:50serhiy.storchakasetstatus: open -> closed
assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2016-06-21 21:03:56python-devsetnosy: + python-dev
messages: + msg269024
2016-05-07 12:32:05rhettingersetassignee: rhettinger -> (no value)
2016-05-06 16:58:48serhiy.storchakasetmessages: + msg264996
2016-05-06 16:51:39gvanrossumsetmessages: + msg264993
2016-05-06 16:43:57serhiy.storchakasetfiles: + json_keyword_only.patch

versions: + Python 3.6, - Python 3.4
keywords: + patch
nosy: + gvanrossum

messages: + msg264991
stage: patch review
2013-09-02 20:57:11bob.ippolitosetmessages: + msg196806
2013-09-02 20:49:33serhiy.storchakasetmessages: + msg196805
2013-09-02 18:11:06bob.ippolitosetmessages: + msg196799
2013-09-02 17:15:19ezio.melottisetmessages: + msg196792
2013-09-02 17:12:09serhiy.storchakasetmessages: + msg196791
2013-09-01 23:10:55bob.ippolitosetassignee: bob.ippolito -> rhettinger
messages: + msg196751
2013-09-01 22:46:47rhettingersetpriority: normal -> low
assignee: rhettinger -> bob.ippolito
messages: + msg196747
2013-09-01 08:44:49serhiy.storchakasetmessages: + msg196717
2013-09-01 06:46:52rhettingersetassignee: rhettinger
messages: + msg196712
2013-08-13 15:14:50Arfreversetnosy: + Arfrever
2013-08-13 13:04:12r.david.murraysetmessages: + msg195072
2013-08-13 12:57:56ezio.melottisetmessages: + msg195071
2013-08-13 12:46:34r.david.murraysetnosy: + r.david.murray
messages: + msg195070
2013-08-13 11:18:14serhiy.storchakacreate