This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add sys.setasthook() to allow to use a custom AST optimizer
Type: Stage:
Components: Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, vstinner
Priority: normal Keywords: patch

Created on 2013-03-22 00:58 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
setasthook.patch vstinner, 2013-03-22 00:58 review
enable_astoptimizer.patch vstinner, 2013-03-22 01:09 review
Messages (7)
msg184932 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-03-22 00:58
I wrote an optimizer rewriting the AST to implement various optimizatoions:
https://bitbucket.org/haypo/astoptimizer

To be able to use it, or use any other kind of AST transformation, I propose to add a new sys.setasthook() function to set a custom AST hook.

Attached patch is a proof-of-concept. The patch should be improved: document the function and add write unit tests.

Prototype of the hook:

def asthook(ast, filename):
   # ...
   return new_ast

I don't know if we need to add a compiler option (like _ast.PyCF_ONLY_AST) to skip the hook.
msg184936 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-03-22 01:09
enable_astoptimizer.patch: example of usage of the new function, you can use it to try astoptimizer.
msg184971 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-22 15:17
This is unnecessary. I added the source_to_code() method (http://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader.source_to_code) to loaders to explicitly handle this case.
msg184978 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-03-22 15:53
> This is unnecessary.

Your change only concerns imports. eval() and compile() cannot be
hooked using your method.

2013/3/22 Brett Cannon <report@bugs.python.org>:
>
> Brett Cannon added the comment:
>
> This is unnecessary. I added the source_to_code() method (http://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader.source_to_code) to loaders to explicitly handle this case.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17515>
> _______________________________________
msg184986 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-22 17:09
And that's fine as it allows for more explicit control over what gets optimized. Basically if there is a global hook directly into the compiler there needs to be more control over detecting it's being optimized, what to do about .pyo files, etc. At least at the loader level it's much more modular and controlled as to what will and will not get the optimizations.

So if you want to pursue this I think there needs to be more thought than just a global sys hook for this sort of thing.
msg238416 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 11:05
I'm not more interested to work on my astoptimizer project, and nobody looks to need sys.setasthook(), so I prefer to close the issue.
msg239459 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-03-28 14:41
The closing of this issue inspired me to think of a better way to do this, and I think providing something in importlib.util that accepted a NodeTransformer as an argument to a function that compiled source files to bytecode files would fill this issue thanks to PEP 488. It would mean that nothing is done dynamically and everything is done offline, but that also means startup costs are gone in regards to bytecode generation and it gets people to actually think about optimizations of Python instead of it just being an afterthought.

importlib.util.compile(path: Union[str, pathlib.Path], transformer: ast.NodeTransformer = None, *, optimization: Union[str, int] = None) -> pathlib.Path
History
Date User Action Args
2022-04-11 14:57:43adminsetgithub: 61717
2015-03-28 14:41:42brett.cannonsetmessages: + msg239459
2015-03-18 11:05:53vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg238416
2013-03-22 17:09:41brett.cannonsetmessages: + msg184986
2013-03-22 15:53:19vstinnersetmessages: + msg184978
2013-03-22 15:17:42brett.cannonsetmessages: + msg184971
2013-03-22 01:09:12vstinnersetfiles: + enable_astoptimizer.patch

messages: + msg184936
2013-03-22 00:58:11vstinnercreate