classification
Title: Verifying refcounts.dat
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: amaury.forgeotdarc, berker.peksag, brett.cannon, dmalcolm, docs@python, fdrake, miss-islington, python-dev, serhiy.storchaka, skip.montanaro
Priority: low Keywords: patch

Created on 2013-05-28 19:22 by serhiy.storchaka, last changed 2018-12-20 08:14 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
refcounts_const_char.patch serhiy.storchaka, 2013-05-31 20:20 Add `const` to `char*` arguments. review
Pull Requests
URL Status Linked Edit
PR 11247 merged serhiy.storchaka, 2018-12-19 20:01
PR 11258 merged miss-islington, 2018-12-20 07:34
Messages (17)
msg190236 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-28 19:22
I found one error in Doc/data/refcounts.dat (see issue9369, second argument of PyObject_CallMethodObjArgs is `PyObject*`, not `char*`). Perhaps there are other errors. How this file was generated first? Is it possible to write a tool which will automatically verify at least signatures of functions?
msg190238 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-28 19:48
New changeset 0889ab0d0da1 by Serhiy Storchaka in branch '3.3':
Issue #18085: Fix PyObject_CallMethodObjArgs()'s entry in refcounts.dat.
http://hg.python.org/cpython/rev/0889ab0d0da1

New changeset ef9d42b98a3d by Serhiy Storchaka in branch '2.7':
Issue #18085: Fix PyObject_CallMethodObjArgs()'s entry in refcounts.dat.
http://hg.python.org/cpython/rev/ef9d42b98a3d

New changeset 6d0fd113a2e4 by Serhiy Storchaka in branch 'default':
Issue #18085: Fix PyObject_CallMethodObjArgs()'s entry in refcounts.dat.
http://hg.python.org/cpython/rev/6d0fd113a2e4
msg190240 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2013-05-28 19:55
I'm a little surprised that still exists.

The first version was generated manually.
msg190245 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-05-28 20:23
Wow, I didn't even know that file existed. It could actually help with some static analysis of the C code to verify that the refcounts are as expected w/o actually having to do inter-procedural analysis to figure out all of the refcount values. I wonder if Dave's gcc work could help either verify the file or benefit from it?
msg190246 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2013-05-28 20:26
As far as I can tell that file is still manually maintained.  I see it mentioned in Doc/conf.py, but nowhere else.  It shouldn't be hard to deal with manually, as the C API doesn't change that often.

A verifier shouldn't be terribly difficult to write, as Python C source is pretty consistently structured.  I'm not volunteering for the task though. <wink>
msg190250 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2013-05-28 20:52
Were I adding that today, I'd use a more verbose (but more standard)
format, like configparser or JSON.  If any further use is going to be
made of it, that should be considered.  Colon-delimited is a pretty
fragile format.
msg190251 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2013-05-28 20:55
> Fred L. Drake, Jr. added the comment:
>
> Were I adding that today, I'd use a more verbose (but more standard)
> format, like configparser or JSON.  If any further use is going to be
> made of it, that should be considered.  Colon-delimited is a pretty
> fragile format.

Given the context, it should be okay until Python is rewritten in C++. :-)

I don't know where this is consumed.  Presumably by Sphinx somewhere.
msg190288 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-05-29 11:16
And is it necessary to list all functions there?
Many functions share the same behavior: they don't change the ownership of PyObject* passed as argument, and return a new reference.

Only document functions that don't conform to this rule, like PyTuple_SET_ITEM and PyImport_AddModule.
msg190320 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2013-05-29 16:46
> And is it necessary to list all functions there?

Probably not.  However, BITD, I was figuring this stuff out as I was
going along by examining the source code and writing down what I
found.  No regard for common or special cases.

S
msg190414 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-05-31 20:20
Since creating refcounts.dat many functions changed their argument's types from `char*` to `const char*`. Here is a patch which fixes mismatches (perhaps not all).

refcounts.dat in 3.x contains PyInt_* functions which don't exist in 3.x.
msg192885 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-07-11 19:01
New changeset ffe24e3e7a2a by Serhiy Storchaka in branch '3.3':
Issue #18085: Add missed const modifier for some entries in refcounts.dat.
http://hg.python.org/cpython/rev/ffe24e3e7a2a

New changeset 6587fd3d89ae by Serhiy Storchaka in branch 'default':
Issue #18085: Add missed const modifier for some entries in refcounts.dat.
http://hg.python.org/cpython/rev/6587fd3d89ae
msg266865 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-06-02 02:46
Can we now close this as a duplicate of issue 9755?
msg266868 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2016-06-02 03:07
I don't think this is a duplicate of issue 9755; this relates to verifying the data, and that revolves around possible process improvements.

Whether this issue should be closed is tied to whether the file has been verified, as the issue title suggests.

I don't know whether Serhiy verified everything when he made his changes or not; that's not explicit in the issue or commit comments.
msg266879 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-06-02 07:43
No, I hadn't verified all names. I'm sure there are API functions not mentioned in refcounts.dat.

See also issue23903. It would be nice to have a tool that parses Python source headers and produce lists of all public names in different formats for manual updating python3.def, refcounts.dat, docs, etc.
msg332164 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-19 20:24
PR 11247 adds more functions and fixes some errors.
msg332190 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-20 07:34
New changeset 83dd4e87a62311cfea5fdd37e8a945b6b07bccee by Serhiy Storchaka in branch 'master':
bpo-18085: Update refcounts.dat. (GH-11247)
https://github.com/python/cpython/commit/83dd4e87a62311cfea5fdd37e8a945b6b07bccee
msg332192 - (view) Author: miss-islington (miss-islington) Date: 2018-12-20 07:43
New changeset 73fc14d1f8441aef5ee03be627c63e74a6d915d6 by Miss Islington (bot) in branch '3.7':
bpo-18085: Update refcounts.dat. (GH-11247)
https://github.com/python/cpython/commit/73fc14d1f8441aef5ee03be627c63e74a6d915d6
History
Date User Action Args
2018-12-20 08:14:48serhiy.storchakasetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.7, Python 3.8, - Python 2.7, Python 3.3, Python 3.4
2018-12-20 07:43:29miss-islingtonsetnosy: + miss-islington
messages: + msg332192
2018-12-20 07:34:10miss-islingtonsetpull_requests: + pull_request10487
2018-12-20 07:34:00serhiy.storchakasetmessages: + msg332190
2018-12-19 20:24:57serhiy.storchakasetmessages: + msg332164
2018-12-19 20:23:19serhiy.storchakasetpull_requests: - pull_request10478
2018-12-19 20:23:05serhiy.storchakasetpull_requests: - pull_request10477
2018-12-19 20:01:49serhiy.storchakasetstage: patch review
pull_requests: + pull_request10478
2018-12-19 20:01:47serhiy.storchakasetstage: (no value)
pull_requests: + pull_request10477
2018-12-19 20:01:42serhiy.storchakasetstage: (no value)
pull_requests: + pull_request10476
2016-06-02 07:43:49serhiy.storchakasetmessages: + msg266879
2016-06-02 03:07:53fdrakesetmessages: + msg266868
2016-06-02 02:46:18berker.peksagsetnosy: + berker.peksag
messages: + msg266865
2013-07-17 07:36:13serhiy.storchakasetpriority: normal -> low
2013-07-11 19:01:30python-devsetmessages: + msg192885
2013-05-31 20:20:01serhiy.storchakasetfiles: + refcounts_const_char.patch
keywords: + patch
messages: + msg190414
2013-05-29 16:46:46skip.montanarosetmessages: + msg190320
2013-05-29 11:16:24amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg190288
2013-05-28 20:55:55skip.montanarosetmessages: + msg190251
2013-05-28 20:52:15fdrakesetmessages: + msg190250
2013-05-28 20:26:36skip.montanarosetmessages: + msg190246
2013-05-28 20:23:07brett.cannonsetnosy: + dmalcolm, brett.cannon
messages: + msg190245
2013-05-28 19:55:16fdrakesetnosy: + fdrake
messages: + msg190240
2013-05-28 19:48:07python-devsetnosy: + python-dev
messages: + msg190238
2013-05-28 19:22:38serhiy.storchakacreate