classification
Title: [WIP] PEP 511: Add sys.set_code_transformers()
Type: enhancement Stage:
Components: Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, haypo, yselivanov
Priority: normal Keywords: patch

Created on 2016-01-18 09:10 by haypo, last changed 2017-06-28 01:12 by haypo.

Files
File name Uploaded Description Edit
transformers.patch haypo, 2016-01-18 09:10 review
test_sys.patch haypo, 2016-01-18 09:15 review
transformers-2.patch haypo, 2016-01-19 10:49 review
transformers-3.patch haypo, 2016-01-22 16:33 review
transformers-4.patch haypo, 2016-01-22 16:48 review
pycf_transformed_ast.patch haypo, 2016-01-23 12:11
transformers-5.patch haypo, 2016-02-05 09:15 review
transformers-6.patch haypo, 2016-02-13 00:30 review
Pull Requests
URL Status Linked Edit
PR 2355 open haypo, 2017-06-23 12:26
Messages (14)
msg258509 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-18 09:10
Attached patch implements the largest part of the PEP 511:

* add sys.get_code_transformers() and sys.set_code_transformers()
* add sys.implementation.optim_tag
* add -o command line option setting sys.implementation.optim_tag
* modify importlib to use sys.implementation.optim_tag to build .pyc filenames
* modify importlib to raise an ImportError when the .pyc file is missing, the .py is available, and the optimizer tag created from code transformers is different than the current sys.implementation.optim_tag
* add Lib/test/test_pep511.py: test changes especially bytecode and AST transformers
* add a PeepholeOptimizer object created by default for sys.set_code_transformers()
* skip the peephole optimizer when -o noopt is used
* add sys field to the PyInterpreterState structure: used by _PySys_GetCodeTransformers() to get code transformers
* update unit tests to use the optimizer to create the .pyc filename

An optimizer tag must contain characters: '.', '-', directory separators or NUL character.

Brett Canon proposed to not include the optimizer tag in .pyc filenames if it is "opt" to limit backward incompatible changes.

Note: The PEP 511 is under discussion on python-ideas. The patch implements the first version of the PEP, sent to python-ideas, so it doesn't include proposed changes like changing code_transformer() prototype to "def code_transformer(self, code, context)".

See also the issue #26100: add test.support.optim_args_from_interpreter_flags().
msg258511 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-18 09:15
Missing patch for test_sys testing the new sys.get/set_code_transformers() functions.
msg258560 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-18 23:40
main.c:


+        case 'o':
+            if (wcslen(_PyOS_optarg) == 0
+               || wcschr(_PyOS_optarg, L'.')
+               || wcschr(_PyOS_optarg, L'-')
...

'-' character must be valid for -o argument.
msg258573 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-19 07:44
sys_init_code_transformer: Py_DECREF(name); must be Py_XDECREF(name); at exit, it can NULL. An unit test on an invalid transformer (with no name) is missing?
msg258580 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-19 10:49
Patch version 2:

* allow "-" character in the argument of the -o command line option
* add an unit test on invalid type for code transformer
* fix a crash in sys.set_code_transformers(): replace Py_DECREF() with Py_XDECREF()
msg258581 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-19 10:51
(oops, i posted the same message with patch twice)
msg258678 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-20 12:41
Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.
msg258822 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-22 16:33
Rebased patch.
msg258829 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-22 16:48
Patch version 4: Fix sys_set_code_transformers(), initialize seq to NULL to fix a crash on error handling.
msg258866 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-23 12:11
> Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.

Additionnal pycf_transformed_ast.patch implements it. The patch should be applied on top of transformers-4.patch.

Note: PyCF_TRANSFORMED_AST has the same value (0x1000) than the old constant CO_GENERATOR_ALLOWED. This constant was removed in Python 2.5.0 by the changeset 6b42920accc9 in 2006. Is it an issue?
msg259648 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-02-05 09:15
Rebased patch combining also pycf_transformed_ast.patch.
msg260217 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-02-13 00:30
Patch version 6: updated to the new code_transformer() API, it now takes a code object as input and must return a new code object.

Sadly, it looks like there are reference leaks (try: ./python -m test -R 3:3 test_import).
msg260222 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-02-13 01:07
TODO: implement a fast-path for PyCode_Optimize() avoiding completly the "expensive" code transformer API, before the first call to sys.set_code_transformer().

It would avoid list <=> tuple conversions for code constants, and the need of creating two code objects.

In practice, this fast-path should be taken in most cases.
msg296704 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 12:29
Recently, some people asked me for an update for my FAT Python project. So I rebased this change I wrote 1 year 1/2 and adapted it for the new code base:

* I renamed test_pep511.py to test_code_transformer.py
* I removed the sys module from the PyInterpreterState structure (added in my previous patch), since Eric Snow and Nick Coghlan removed a variant of the sys module from this structure for their work on subinterpreters and reworked Python initialization

The PEP 511 is not accepted, so the implementation is still a work-in-progress (WIP) and must not be merged.
History
Date User Action Args
2017-06-28 01:12:15hayposettitle: PEP 511: Add sys.set_code_transformers() -> [WIP] PEP 511: Add sys.set_code_transformers()
2017-06-23 12:29:20hayposetmessages: + msg296704
2017-06-23 12:26:39hayposetpull_requests: + pull_request2401
2016-02-13 01:07:16hayposetmessages: + msg260222
2016-02-13 00:30:35hayposetfiles: + transformers-6.patch

messages: + msg260217
2016-02-05 09:16:08hayposetfiles: + transformers-5.patch

messages: + msg259648
2016-01-23 12:11:43hayposetfiles: + pycf_transformed_ast.patch

messages: + msg258866
2016-01-22 16:48:10hayposetfiles: + transformers-4.patch

messages: + msg258829
2016-01-22 16:33:22hayposetfiles: + transformers-3.patch

messages: + msg258822
2016-01-20 12:41:38hayposetmessages: + msg258678
2016-01-19 10:51:20hayposetmessages: + msg258581
2016-01-19 10:50:23hayposetmessages: - msg258579
2016-01-19 10:50:14hayposetfiles: - transformers-2.patch
2016-01-19 10:49:56hayposetfiles: + transformers-2.patch

messages: + msg258580
2016-01-19 10:49:38hayposetfiles: + transformers-2.patch

messages: + msg258579
2016-01-19 07:44:34hayposetmessages: + msg258573
2016-01-18 23:40:10hayposetmessages: + msg258560
2016-01-18 18:55:22yselivanovsetnosy: + yselivanov
2016-01-18 09:15:16hayposetfiles: + test_sys.patch

messages: + msg258511
2016-01-18 09:10:57haypocreate