Recently it came to light that the classes in C and Python implementations of the io module have slightly different attributes (bpo-9858). I propose the addition of a helper function in Lib/test/support.py to verify that the classes in two different implementations define the same attributes. Then, we can add tests to use that function to verify that C and Python implementations define the same API (for the io module, but also for other modules where we have two implementations). The script I added to bpo-9858 could serve as a starting point for such a function.

Since CPython's standard library is the de facto reference implementation, it's important that it define one API and not two slightly different ones. :-)

Shouldn't the test suite catch such discrepancies by testing all of the API? So your script catching something would be the equivalent of "oops, we forgot a test" (or "oops, this name shouldn't be public"). Which is not a bad thing to have as part of the test suite.

Yes, exactly. :-)

(see also bpo-9731, which has a similar flavor)

+1
I presume you can use the experience with bpo-9858 to either refine the program to have fewer false positives or refine the program doc to explain the possibility where such are unavoidable. (And as a lint-like metatest, I think false positives are better than false negatives.)

@daniel do you intend putting forward a formal patch on this issue? I'm asking as I think bpo-9858 is effectively completed and I've just asked for a formal patch review on bpo-9731.

Created a patch to check whether classes define the same attributes.

This adds a test for RawIOBase, where Python and and C implementations are out of sync, but skips one of the tests as the issue (bpo-9858) is still outstanding.

With regret, I have not had time to work on patches and am unlikely to have time in the near future.

Thanks for the patch! I've reviewed it on Rietveld: http://bugs.python.org/review/9859/

Thanks Berker!

I've added some updates to the patch taking your suggestions into account. There is also a change from using "detect_module_matches" to "detect_api_mismatch", as it is a more general (and accurate) description.

Also, adding tests for the "detect_api_mismatch" helper function.

New changeset b42d1f2aa7a2 by Gregory P. Smith in branch 'default':
bpo-9859: Adds a test.support.detect_api_mismatch function useful to
https://hg.python.org/cpython/rev/b42d1f2aa7a2

New changeset 0b6c894c3c96 by Gregory P. Smith in branch 'default':
bpo-9859: Adds a CPyMatchTest test case to compare the exposed APIs
https://hg.python.org/cpython/rev/0b6c894c3c96

I've added a couple of comments about the test: http://bugs.python.org/review/9859/

Thanks! Patch applied. I reworded one doc string slightly and fixed up a few lines that were longer than 80 characters. Berker's most recent comments are good ones and can be addressed in another patch.

Laura, can you jump through the https://www.python.org/psf/contrib/contrib-form/ hoop? (it'll add the "*" next to your name in the bug tracker showing the contributor form has been signed)

Still TODO: fix the actual issue of the APIs being different. (bpo-9858)

New changeset 9903368b9d7b by Gregory P. Smith in branch 'default':
bpo-9859: rename CPyMatchTest to APIMismatchTest and add @support.cpython_only.
https://hg.python.org/cpython/rev/9903368b9d7b

New changeset cbdd56d07123 by Gregory P. Smith in branch 'default':
bpo-9859: Document test.support.detect_api_mismatch() and simplify its test.
https://hg.python.org/cpython/rev/cbdd56d07123

New changeset c6df85e1d42e by Gregory P. Smith in branch 'default':
bpo-9859: Use an expected failure rather than a skip.
https://hg.python.org/cpython/rev/c6df85e1d42e

Thanks :)

New changeset 0316811f33b2 by Gregory P. Smith in branch 'default':
bpo-9859: add the missing versionadded tag to the documentation.
https://hg.python.org/cpython/rev/0316811f33b2