This issue is part of the Great Argument Clinic Conversion Derby,
where we're trying to convert as much of Python 3.4 to use
Argument Clinic as we can before Release Candidate 1 on January 19.

This issue asks you to change the following bundle of files:
    Objects/typeobject.c: 5 sites
    Objects/longobject.c: 5 sites
    Objects/listobject.c: 5 sites
    Objects/floatobject.c: 4 sites
    Modules/resource.c: 4 sites
    Modules/_threadmodule.c: 4 sites
    Modules/_bz2module.c: 4 sites
    Modules/_bisectmodule.c: 4 sites
    Python/marshal.c: 3 sites
    Objects/exceptions.c: 3 sites
    Modules/nismodule.c: 3 sites
    Modules/gcmodule.c: 3 sites
    Python/_warnings.c: 2 sites
    Modules/_cryptmodule.c: 1 sites

Talk to me (larry) if you only want to attack part of a bundle.

For instructions on how to convert a function to work with Argument
Clinic, read the "howto":
    http://docs.python.org/dev/howto/clinic.html

Antoine Pitrou took care of cryptmodule.c.

Here is the patch for Modules/resource.c.

However, I can not convert this method signature:

if (!PyArg_ParseTuple(args, _Py_PARSE_PID "i|(OO):prlimit",
                          &pid, &resource, &curobj, &maxobj))

_Py_PARSE_PID can be 'i' or 'l' or 'L'.

Here is the patch for Modules/gcmodule.c.

Here is the patch for Python/marshal.c.

A couple of issues:
1. I can not have bytes as argument.

    bytes: Py_buffer -> not possible

So I changed it to byte.

2. I can not give default value, marshal.version, to argument.

version: int(c_default="Py_MARSHAL_VERSION") = marshal.version -> not possible (unless I put "import marshal" in Tools/clinic/clinic.py).

So I gave it a raw value, 2.

version: int(c_default="Py_MARSHAL_VERSION") = 2

Someone else came up with this hack using a custom converter:

class PID_converter(int_converter):
  format_unit = '" _Py_PARSE_PID "'

Also, for marshal.version, soon you will be able to.  This syntax won't work now but it will soon:

version: int(c_default="Py_MARSHAL_VERSION") = version

Keep track of #20226, that's the issue where I'm adding improved support for simple expressions like "version".

With little modification:

class PID_converter(int_converter):
  type = 'pid_t'
  format_unit = '" _Py_PARSE_PID "'

It works. Thanks! But I got unnecessary empty string:

if (!PyArg_ParseTuple(args, "" _Py_PARSE_PID "i:prlimit", &pid, &resource))

It should be:

if (!PyArg_ParseTuple(args, _Py_PARSE_PID "i:prlimit", &pid, &resource))

Anyway, that is trivial problem. Now I hit a hard one. How do you convert this signature?

-    if (!PyArg_ParseTuple(args, _Py_PARSE_PID "i|(OO):prlimit",
-                          &pid, &resource, &curobj, &maxobj))

I create custom converters:

+class TupleOpen_converter(object_converter):
+  format_unit = '(O'
+class TupleClose_converter(object_converter):
+  format_unit = 'O)'

and

+/*[clinic input]
+resource.prlimit
+
+    pid: PID
+    resource: int
+    [
+    curobj: TupleOpen
+    maxobj: TupleClose
+    ]
+    /
+[clinic start generated code]*/

But I got invalid argument size in the generated code.

+        case 2:
+            if (!PyArg_ParseTuple(args, "" _Py_PARSE_PID "i:prlimit", &pid, &resource))
+                return NULL;
+            break;
+        case 4:   ========> should be "case 3:"
+            if (!PyArg_ParseTuple(args, "" _Py_PARSE_PID "i(OO):prlimit", &pid, &resource, &curobj, &maxobj))
+                return NULL;
+            group_right_1 = 1;
+            break;
+        default:
+            PyErr_SetString(PyExc_TypeError, "resource.prlimit requires 2 to 4 arguments"); =======> should be "2 to 3 arguments"
+            return NULL;

Any idea, Larry?

This is known bug in prlimit (issue20191).

Here is the patch for typeobject. I didn't convert the new method. It's so complicated. It counts whether how many arguments or keywords you pass. It asserts the args before parsing the args. I don't think clinic supports this.

This is the code:
    assert(args != NULL && PyTuple_Check(args));
    assert(kwds == NULL || PyDict_Check(kwds));

    /* Special case: type(x) should return x->ob_type */
    {
        const Py_ssize_t nargs = PyTuple_GET_SIZE(args);
        const Py_ssize_t nkwds = kwds == NULL ? 0 : PyDict_Size(kwds);

        if (PyType_CheckExact(metatype) && nargs == 1 && nkwds == 0) {
            PyObject *x = PyTuple_GET_ITEM(args, 0);
            Py_INCREF(Py_TYPE(x));
            return (PyObject *) Py_TYPE(x);
        }

        /* SF bug 475327 -- if that didn't trigger, we need 3
           arguments. but PyArg_ParseTupleAndKeywords below may give
           a msg saying type() needs exactly 3. */
        if (nargs + nkwds != 3) {
            PyErr_SetString(PyExc_TypeError,
                            "type() takes 1 or 3 arguments");
            return NULL;
        }
    }

    /* Check arguments: (name, bases, dict) */
    if (!PyArg_ParseTupleAndKeywords(args, kwds, "UO!O!:type", kwlist,
                                     &name,
                                     &PyTuple_Type, &bases,
                                     &PyDict_Type, &orig_dict))
        return NULL;

Here is the patch for longobject. There are two sites which I couldn't convert. The first is the constructor which is complicated. The other one is __round__ which clinic explicitly does not support.

Here is the patch for listobject. A couple of thoughts:

1. I can not convert this signature:

    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
                                _PyEval_SliceIndex, &start,
                                _PyEval_SliceIndex, &stop))
        return NULL;

It's bloody difficult. If I make start (and stop) as PyObject, I get a lot of pointer warning conversion. The closest I can get is making start (and stop) as Py_ssize_t but it changes the behaviour. a.index(0, -4*sys.maxsize, 4*sys.maxsize) will throw OverflowError.

2. for tp_init, it forces me to give wrong return signature.
"__init__([sequence=None])\n"

This is the clinic input:

list.__init__

  self: PyListObject
  [
  sequence: object(c_default="NULL") = None
  ]
  /

Without None, it will be an error in clinic process.

3. The init function returns integer but in error case, it returns NULL which will throws pointer warning conversion.

static int
list___init__(PyObject *self, PyObject *args, PyObject *kwargs)
{
    int return_value = -1;
    int group_right_1 = 0;
    PyObject *sequence = NULL;

    if (!_PyArg_NoKeywords("__init__", kwargs))
        goto exit;
    switch (PyTuple_GET_SIZE(args)) {
        case 0:
            break;
        case 1:
            if (!PyArg_ParseTuple(args, "O:__init__", &sequence))
                return NULL;
            group_right_1 = 1;
            break;
....

1. In list.index use custom converter.

/*[python input]

class slice_index_converter(CConverter):
    type = 'Py_ssize_t'
    converter = '_PyEval_SliceIndex'

[python start generated code]*/

2. In list.__init__ use either "unspecified" default value or optional group without default value.

3. This looks as Argument Clinic bug.

I'll fix the "return NULL" problem.  However, you are using optional groups in the list __init__.  The original doesn't use them.  Please stop using optional groups in functions that don't require them.

Thanks, Serhiy, for the pointer. Now, I am able to convert the method using _PyEval_SliceIndex function.

Sorry, Larry. I used optional groups in __init__ because I didn't know about unspecified.

Here is the updated patch for listobject. One thought, for list.index method, I use this:

stop: slice_index(c_default="Py_SIZE(self)") = unspecified

Then the signature will be:

"index(value, [start=0, [stop=unspecified]])\n"

Somehow I prefer more explanatory signature:

"index(value, [start=0, [stop=size of list]])\n"

Anyway, this is a trivial thing.

Here is the patch for floatobject. I did not convert 2 sites. The first is the round method which clinic explicitly does not support. The second one is the new method. This is the snippet of new method:

float_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    PyObject *x = Py_False; /* Integer zero */
    static char *kwlist[] = {"x", 0};

    if (type != &PyFloat_Type)
        return float_subtype_new(type, args, kwds); /* Wimp out */
    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O:float", kwlist, &x))
        return NULL;
    /* If it's a string, but not a string subclass, use
       PyFloat_FromString. */
    if (PyUnicode_CheckExact(x))
        return PyFloat_FromString(x);
....

If I clinic this method, I could not put custom code before parsing arguments code anymore. This could affect the performance or may not be correct at all.

This is updated patch for marshal incorporating the fixes from clinic. Now, I can use 'bytes' named argument and default value of marshal.version instead of hardcoded number.

Updated marshal patch. I just learned we don't need to release pybuffer (y*) manually.

Here is the updated patch for resource module based on Zachary's comment.

Here is the updated patch for type module based on Zachary's comment.

However, I can not convert this method.

{"__subclasshook__", object_subclasshook, METH_CLASS | METH_VARARGS,
     object_subclasshook_doc},

static PyObject *
object_subclasshook(PyObject *cls, PyObject *args)
{
    Py_RETURN_NOTIMPLEMENTED;
}

>>> type.__subclasshook__()
NotImplemented
>>> type.__subclasshook__('cutecat')
NotImplemented
>>> type.__subclasshook__('cutecat', 1, [])
NotImplemented

Here is the updated patch for list module based on Zachary and Serhiy's reviews.

Forgot to say that, in list module, anything is convertable except __getitem__.

Here is the updated patch for marshal module based on Zachary's review.

Here is the updated patch for float object based on Zachary and Serhiy's reviews. Some methods that can not be converted are:

__getnewargs__, __round__, float_new.


So these files are ready for Python 3.4: resource, typeobject, listobject, and floatobject.

These files are not ready yet: gc, longobject.

Attaching a conglomerate patch consisting of clinic_resource_v2.patch, clinic_typeobject_v2.patch, clinic_listobject_v3.patch, clinic_marshal_v4.patch, and clinic_floatobject_v2.patch, with updated (post-#20326) clinic output, for easier review.

Here is the updated patch addressing Zachary's reviews (Thanks!). However, there are some reviews that I could not implement.

1. "This is a good candidate for a custom return converter."

I can not synchronize struct rlimit and NULL return values.

2. "Should be 'class float "PyFloatObject *" "&PyFloat_Type"'.  Using PyFloatObject
* instead of PyObject * may require some casts to PyObject * in some places, but
it's better to use the real name."

I tried it but it was like opening pandora box. It's too much effort to surpress compile errors and warnings. And casting PyFloatObject to PyObject in many places, such as functions, macros, makes me nervous. I think this one deserves a dedicated ticket.

Vajrasky Kok wrote:
> However, there are some reviews that I could not implement.
>
> 1. "This is a good candidate for a custom return converter."
>
> I can not synchronize struct rlimit and NULL return values.

Looking again, that one is non-trivial, but still doable.  You just need a "this means an error happened" value to initialize rl to, and return that value instead of NULL.

> 2. "Should be 'class float "PyFloatObject *" "&PyFloat_Type"'.  Using
> PyFloatObject * instead of PyObject * may require some casts to
> PyObject * in some places, but it's better to use the real name."
>
> I tried it but it was like opening pandora box. It's too much effort
> to surpress compile errors and warnings. And casting PyFloatObject
> to PyObject in many places, such as functions, macros, makes me
> nervous. I think this one deserves a dedicated ticket.

It didn't look too bad to me.  There are already several places where a value is cast back and forth between PyObject and PyFloatObject.  Giving 'self' the right type allows a couple of casts to be removed, and the ones that have to be added are almost exclusively in calls to PyFloat_AsDouble or the CONVERT_TO_DOUBLE macro (which looks like it can just do the casts itself without much issue).  I would vote for making PyFloat_AsDouble expect PyFloatObject instead of PyObject, but since (I think?) it looks like it's part of the stable ABI, I'm not sure if that would fly.

See http://hg.python.org/sandbox/zware/rev/51473d8c23f8 for a patch on your patch that uses PyFloatObject, compiles cleanly (on win32, at least), and passes relevant tests (though I haven't run the full test suite on this yet; it takes forever on this PC).

I have a few more review comments that I hope to get posted later this evening.

The patch is looking pretty good overall, though!

Ok, so my changes to CONVERT_TO_DOUBLE don't fly with gcc, so that's out.  In the two cases where convert_to_double wants a PyObject *, we could declare "PyObject *objself = (PyObject *)self;", but that's not particularly appealing.  I'll leave the whole PyObject vs. PyFloatObject issue to someone with more C experience.

Here is the patch addressing Zachary's review (Thanks!). There are some Zachary's suggestions that I could not implement:

1. float_conjugate_impl,            /* nb_float */
I think this should still be the real function (the parser), not the impl.  The
impl function is really just an implementation detail.

It has to be that way. If I change it to float_conjugate, the compiler will complain. But on other places, we can use the real function.

2. v = list_sort_impl((PyListObject *)v, Py_None, 0);
Considering what I said about not using impl functions at the end of
floatobject.c, it would be nice to avoid it here, but I think that would be a
lot more trouble than it would be worth.

I can not use the real function here, otherwise the compiler will throw error.

Zachary, "Looking again, that one is non-trivial, but still doable.  You just need a "this means an error happened" value to initialize rl to, and return that value instead of NULL."

How do you give "this means an error happened" value to struct rlimit?

struct rlimit {
               rlim_t rlim_cur;  /* Soft limit */
               rlim_t rlim_max;  /* Hard limit (ceiling for rlim_cur) */
};

This is what prevents me to use custom converter.

Vajrasky: Note that you can reply to individual review comments within Rietveld; this way context is kept and replies are easier :).  The same people get the message either way.

Anyhow, for float point 1: you can use I believe you can use (unaryfunc) for nb_int and nb_float, just like nb_positive above.

list point 2: I agree, it's not worth it to try to not use the impl function.

rlimit: I'm not sure what value to give, and I'm not where I can play with it until my PC catches fire, either.  Is there some value that makes no sense as a legitimate value?  Is it legal for rlim_cur to be greater than rlim_max?  Or is there a value that is just exceedingly uncommon?  Or you could simply pick some random value; your converter should be written to render with self.err_occurred_if("_return_value == RLIMIT_ERROR_VALUE").

Ok, current status of this issue as I see it:

  Objects/typeobject.c:    part of conglomerate patch, LGTM
  Objects/longobject.c:    awaiting revision based on review
  Objects/listobject.c:    part of conglomerate patch, LGTM
  Objects/floatobject.c:   part of conglomerate patch, two minor issues, otherwise LGTM
  Modules/resource.c:      part of conglomerate patch, LGTM (could use return converter, not essential)
  Modules/_threadmodule.c: untouched, held til 3.5
  Modules/_bz2module.c:    untouched, held til 3.5
  Modules/_bisectmodule.c: untouched, held til 3.5
  Python/marshal.c:        part of conglomerate patch, LGTM
  Objects/exceptions.c:    untouched, held til 3.5
  Modules/nismodule.c:     untouched, held til 3.5
  Modules/gcmodule.c:      awaiting revision based on review
  Python/_warnings.c:      untouched, held til 3.5
  Modules/_cryptmodule.c:  done in another issue

I'd like someone with more C experience to have a look at typeobject, floatobject, listobject, resource, and marshal to make sure I haven't missed anything (or led Vajrasky astray).  If someone else can ok them, I can get them committed.

Vajrasky: What's the status on gcmodule and longobject?

Here is the updated patch. I missed the 'unaryfunc' part. :) Yeah, it's weird. I run clinic again and the dump buffer was getting smaller again.

Here is the updated patch for gc module.

Some functions that can not be converted:
- set_threshold(threshold0, [threshold1, threshold2])
Don't know what default value to give to threshold1, threshold2.

static struct gc_generation generations[NUM_GENERATIONS] = {
    /* PyGC_Head,                               threshold,      count */
    {{{GEN_HEAD(0), GEN_HEAD(0), 0}},           700,            0},
    {{{GEN_HEAD(1), GEN_HEAD(1), 0}},           10,             0},
    {{{GEN_HEAD(2), GEN_HEAD(2), 0}},           10,             0},
};

- collect([generation])
Don't know what default value to give to generation.

int genarg = NUM_GENERATIONS - 1;

I don't think we can use expression in signature.

- get_referrers and get_referents
Don't know what signature to use for functions that can accept any number of arguments.

About rlimit, I think we can use negative number for any member of the struct. But the thing is I am not really sure whether it's okay or not. So it's better on the safe side.

I'll finish longobject in one or two days.

Here is the updated patch for gc module based on Zachary's review.

Here is the updated patch for long object. A couple of functions that can not be converted:

- long_new
It has custom processing before parsing arguments part.

- long_round
Not supported by clinic.

_ __trunc__, __floor__, __ceil__ all are mapped to long_long. Not sure how to handle this case.

Here is the updated patch for gc module based on Zachary's review.

Here is the updated patch for long object based on Zachary's review.

All the Derby patches should only go into trunk at this point.

Look like this has never been applied, any reason for this?

Tip has probably highly diverged, but I may try to rebase it, should I?

New changeset c62352ec21bc by Victor Stinner in branch 'default':
Issue #20185: Convert _warnings.warn() to Argument Clinic
https://hg.python.org/cpython/rev/c62352ec21bc

Julien, to help push these changes forward, I suggest start by rebasing and reviewing the conglomerate patch, since it seems that was almost ready.

Reading through the thread, it seems the current status is:

1. _crypt module handled elsewhere by Antoine

2. type, list, and float objects, the resource module, and Python/marshal.c are done by Vajrasky, in part of the conglomerate v4 patch. Zach said they look good, but wanted a second opinion from someone with good C knowledge. Check if Zach’s minor comments for float object and resource module were dealt with.

3. long object and gc module were updated by Vajrasky since last review, so probably need to check if there are any outstanding problems with them.

4. _bisect module by Julien via Issue 28754

5. Victor made a change to Python/_warnings.c, but there may be more to do

6. Not yet handled: Objects/exceptions.c, _thread, _bz2, nis modules.

Zach’s comments on float seem to have been addressed (https://bugs.python.org/review/20185/diff2/10940:10949/Objects/floatobject.c). The comments on the resource module were about return converters: <http://bugs.python.org/review/20185/diff2/10817:10949/Modules/resource.c>. One has been addressed.

The other was about the rlimit2py() function, used in two places. One of these is resource.prlimit(), which is not converted to Arg Clinic in the patch. If we don’t convert it, I agree that leaving rlimit2py() as it is is simplest.

Given that the current prlimit() arg parsing is buggy (Issue 20191), it would be good to convert it to Arg Clinic in the process of solving the bug, but that could be done as a separate step.

Tangentially, the rlimit2py() function has special HAVE_LONG_LONG handling. This is no longer needed in 3.6; see Issue 27961. But again, I think it is better to review and commit the existing patch than to keep perfecting it :)

Victor, you perhaps could just use NULL as default:

+    source: object = NULL

I finished reviewing the most recent patches and left some comments. Perhaps it is worth splitting the conglomerate patch up. I don’t see any point holding back some modules while things are tweaked in unrelated files.

My biggest concern is casting function pointers, especially when the number of parameters is different than expected. It would be good to fix or discuss that more before going ahead with floatobject.c.

Martin, you are the one who looked at these patches for last three years. Do you want to take this issue to you and update Vajrasky's patches?

Now there is good performance argument for converting builtins to Argument Clinic.

Will keep this in mind, but my time is rather limited, so I may not get to it (and I wouldn’t want to discourage other people from working on it)

Updated patch for longobject.c addresses Martin's comments.

longobject_v5 looks good to me

New changeset aa7ac93d23b2 by Serhiy Storchaka in branch 'default':
Issue #20185: Converted the int class to Argument Clinic.
https://hg.python.org/cpython/rev/aa7ac93d23b2

Updated patch for the gc module is synchronized with current sources and addresses Martin's comments.

The only problem with this patch is that it exposes the default value for gc.collect() argument as constant 2 in Python instead of actual NUM_GENERATIONS-1. The latter value can't be exposed in Python.

On other side it is 2 in the documentation, and it is not so easy to change the value of NUM_GENERATIONS.

New changeset 50958e13c833 by Serhiy Storchaka in branch 'default':
Issue #20185: Converted the gc module to Argument Clinic.
https://hg.python.org/cpython/rev/50958e13c833

New changeset 725c112c941ca6ac7fb995449f85501ea100647e by Serhiy Storchaka in branch 'master':
Issue #20185: Converted the gc module to Argument Clinic.
https://github.com/python/cpython/commit/725c112c941ca6ac7fb995449f85501ea100647e

Here is a set of patches based on issue20185_conglomerate_v4.diff. They are synchronized with current sources. Addressed Martin's comments,converted list.index(), float.__round__() and resource.prlimit() and made other minor changes.

New changeset 5c643a028ee86c613d7168ca5bcb8fc94477a09e by Serhiy Storchaka in branch 'master':
bpo-20185: Convert typeobject.c to Argument Clinic. (#544)
https://github.com/python/cpython/commit/5c643a028ee86c613d7168ca5bcb8fc94477a09e

New changeset 1989763f0d0858ce6274f5e1d725b5b8da91a780 by Serhiy Storchaka in branch 'master':
bpo-20185: Convert the resource moduel to Argument Clinic. (#545)
https://github.com/python/cpython/commit/1989763f0d0858ce6274f5e1d725b5b8da91a780

New changeset 0767ad40bfe83525d2ba290cc6eb7c97ce01cdd6 by Serhiy Storchaka in branch 'master':
bpo-20185: Convert the marshal module to Argument Clinic. (#541)
https://github.com/python/cpython/commit/0767ad40bfe83525d2ba290cc6eb7c97ce01cdd6

New changeset fdd42c481edba4261f861fc1dfe24bbd79b5a17a by Serhiy Storchaka in branch 'master':
bpo-20185: Convert list object implementation to Argument Clinic. (#542)
https://github.com/python/cpython/commit/fdd42c481edba4261f861fc1dfe24bbd79b5a17a

New changeset b5c51d3dd95bbfde533655fb86ac0f96f771ba7b by Serhiy Storchaka in branch 'master':
bpo-20185: Convert float object implementation to Argument Clinic. (#543)
https://github.com/python/cpython/commit/b5c51d3dd95bbfde533655fb86ac0f96f771ba7b

New changeset 57d8de80313c536d409d6a104ae577af8ffc57fb by terryjreedy in branch '3.6':
[3.6]bpo-20185: Adjust IDLE test to 3.7 Clinic change [GH-542] (#2116)
https://github.com/python/cpython/commit/57d8de80313c536d409d6a104ae577af8ffc57fb

I rebased the conglomerate patch onto current master, and there's only  three methods left, in floatobject.c, it's three methods inside a #if 0, it does not looks interesting to merge it (30 insertions, 12 deletions), so I'm closing this issue.

New changeset 7d03c8be5af2f1559dbc35b775b3116dfd63cfb6 by Victor Stinner in branch 'main':
bpo-46852: Rename float.__set_format__() to float.__setformat__() (GH-31558)
https://github.com/python/cpython/commit/7d03c8be5af2f1559dbc35b775b3116dfd63cfb6

New changeset 0848da19ce8ea037ab1cfc569778e94bf8e3b24a by Victor Stinner in branch '3.10':
bpo-46852: Rename float.__set_format__() to float.__setformat__() (GH-31558) (GH-31578)
https://github.com/python/cpython/commit/0848da19ce8ea037ab1cfc569778e94bf8e3b24a

New changeset a549cd1fc55888e2e287714b25e2cb2251913909 by Victor Stinner in branch '3.9':
bpo-46852: Rename float.__set_format__() to float.__setformat__() (GH-31558) (GH-31581)
https://github.com/python/cpython/commit/a549cd1fc55888e2e287714b25e2cb2251913909