classification
Title: PyCompilerFlags got a new cf_feature_version field
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, miss-islington, vstinner
Priority: normal Keywords: patch

Created on 2019-06-12 15:15 by vstinner, last changed 2019-06-13 12:17 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 14018 merged vstinner, 2019-06-12 15:16
PR 14019 merged vstinner, 2019-06-12 15:24
PR 14020 merged vstinner, 2019-06-12 15:32
PR 14035 closed miss-islington, 2019-06-13 00:01
PR 14036 merged vstinner, 2019-06-13 00:11
PR 14037 merged miss-islington, 2019-06-13 00:16
PR 14038 merged vstinner, 2019-06-13 00:19
PR 14044 merged miss-islington, 2019-06-13 07:18
Messages (12)
msg345368 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 15:15
The commit dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c of bpo-35766 added a new cf_feature_version field to PyCompilerFlags. Each PyCompilerFlags variable must properly initialize this new field to PY_MINOR_VERSION. I propose to add a new _PyCompilerFlags_INIT macro to statically initialize such variable. I'm not sure if it should be public or not.

The PyCompilerFlags structure is excluded from the stable ABI (PEP 384), but it's documented in the "The Very High Level Layer" C API documentation:
https://docs.python.org/dev/c-api/veryhigh.html#c.PyCompilerFlags

Structure fields are documented there:

struct PyCompilerFlags {
    int cf_flags;
}

The doc is outdated. I'm not sure if it's on purpose or not.

Moreover, the new PyCompilerFlags.cf_feature_version field is not documented in https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api whereas C extensions using PyCompilerFlags should initialize cf_feature_version to PY_MINOR_VERSION?

I'm not sure if C extensions really *must* initialize cf_feature_version, since the field is only used if PyCF_ONLY_AST flag is set in the cf_flags field.

Something else, ast.parse() has been modified to use a (major, minor) version tuple rather an integer to specify the Python version in feature_version, but PyCompilerFlags still only uses the minor major. This API will be broken once the Python major version will be increased to 4, no? Would it make sense to use PY_VERSION_HEX format which includes the major version instead?

Or cf_feature_version could be a structure with major and minor fields, but that might be overkill?
msg345371 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 15:34
> Something else, ast.parse() has been modified to use a (major, minor) version tuple rather an integer to specify the Python version in feature_version, but PyCompilerFlags still only uses the minor major. This API will be broken once the Python major version will be increased to 4, no? Would it make sense to use PY_VERSION_HEX format which includes the major version instead?

I can work on a PR to change cf_feature_version format, but I would prefer to agree here on what is the best format ;-)

Note: compile() has a private keyword-only _feature_version which is also the Python minor version (int): don't include the major version. If we change cf_feature_version, we may also change compile(_feature_version=N) format.
msg345398 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-06-12 18:14
It's fine to document the current state. I don't think you should spend any time *changing* the API to "future-proof" it.

I am hoping that larger changes to the compiler implementation will happen before Python 4, which will make the whole API moot (including the "parser" module, which should be deprecated ASAP). The compiler is excluded from the ABI for a reason.
msg345431 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 00:01
New changeset 2c9b498759f4fc74da82a0a96d059d666fa73f16 by Victor Stinner in branch 'master':
bpo-37253: Document PyCompilerFlags.cf_feature_version (GH-14019)
https://github.com/python/cpython/commit/2c9b498759f4fc74da82a0a96d059d666fa73f16
msg345434 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 00:16
New changeset 37d66d7d4bc7dbac9809d69966a774ebb32563be by Victor Stinner in branch 'master':
bpo-37253: Add _PyCompilerFlags_INIT macro (GH-14018)
https://github.com/python/cpython/commit/37d66d7d4bc7dbac9809d69966a774ebb32563be
msg345435 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 00:17
New changeset a04ea4f92ccbe20ffdbb5fa9b6bb93ee8da50f5d by Victor Stinner in branch 'master':
bpo-37253: Fix typo in PyCompilerFlags doc (GH-14036)
https://github.com/python/cpython/commit/a04ea4f92ccbe20ffdbb5fa9b6bb93ee8da50f5d
msg345436 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 00:21
Many PyRun_xxx() functions of the public C API accept a "PyCompilerFlags* flags" parameter. So yeah, I documented the new cf_feature_version. PyCompilerFlags structure is kind of public ;-)
msg345439 - (view) Author: miss-islington (miss-islington) Date: 2019-06-13 00:36
New changeset 92e836c7dcaf74f7b8617250414224d24d1eb1f2 by Miss Islington (bot) in branch '3.8':
bpo-37253: Add _PyCompilerFlags_INIT macro (GH-14018)
https://github.com/python/cpython/commit/92e836c7dcaf74f7b8617250414224d24d1eb1f2
msg345440 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 00:40
New changeset f9445a391e6a91103fbe88bff4d3adc07f526c73 by Victor Stinner in branch '3.8':
[3.8] bpo-37253: Document PyCompilerFlags.cf_feature_version (GH-14019) (GH-14038)
https://github.com/python/cpython/commit/f9445a391e6a91103fbe88bff4d3adc07f526c73
msg345465 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 07:18
New changeset 022ac0a497b668d8b15e34e582a6396ead1a35e1 by Victor Stinner in branch 'master':
bpo-37253: Remove PyAST_obj2mod_ex() function (GH-14020)
https://github.com/python/cpython/commit/022ac0a497b668d8b15e34e582a6396ead1a35e1
msg345469 - (view) Author: miss-islington (miss-islington) Date: 2019-06-13 07:46
New changeset 032bf30643fff49b5595b53f9c1ce5b2cd2df504 by Miss Islington (bot) in branch '3.8':
bpo-37253: Remove PyAST_obj2mod_ex() function (GH-14020)
https://github.com/python/cpython/commit/032bf30643fff49b5595b53f9c1ce5b2cd2df504
msg345505 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 12:12
> It's fine to document the current state. I don't think you should spend any time *changing* the API to "future-proof" it.

Ok.

> I am hoping that larger changes to the compiler implementation will happen before Python 4, which will make the whole API moot (including the "parser" module, which should be deprecated ASAP). The compiler is excluded from the ABI for a reason.

Aha, that sounds exciting :-)

I created bpo-37268 to propose to deprecate the parser module in Python 3.8.

--

PyCompilerFlags changes are now documented. I made the small code changes that I wanted to do in 3.8 and master. I close the issue.
History
Date User Action Args
2019-06-13 12:17:00vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-06-13 12:12:39vstinnersetmessages: + msg345505
2019-06-13 07:46:15miss-islingtonsetmessages: + msg345469
2019-06-13 07:18:56miss-islingtonsetpull_requests: + pull_request13906
2019-06-13 07:18:48vstinnersetmessages: + msg345465
2019-06-13 00:40:44vstinnersetmessages: + msg345440
2019-06-13 00:36:05miss-islingtonsetnosy: + miss-islington
messages: + msg345439
2019-06-13 00:21:04vstinnersetmessages: + msg345436
2019-06-13 00:19:40vstinnersetpull_requests: + pull_request13902
2019-06-13 00:17:17vstinnersetmessages: + msg345435
2019-06-13 00:16:59miss-islingtonsetpull_requests: + pull_request13901
2019-06-13 00:16:44vstinnersetmessages: + msg345434
2019-06-13 00:11:33vstinnersetpull_requests: + pull_request13900
2019-06-13 00:01:40miss-islingtonsetpull_requests: + pull_request13899
2019-06-13 00:01:32vstinnersetmessages: + msg345431
2019-06-12 18:14:43gvanrossumsetnosy: + gvanrossum
messages: + msg345398
2019-06-12 15:34:00vstinnersetmessages: + msg345371
2019-06-12 15:32:16vstinnersetpull_requests: + pull_request13885
2019-06-12 15:24:09vstinnersetpull_requests: + pull_request13884
2019-06-12 15:16:37vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request13883
2019-06-12 15:15:01vstinnercreate