classification
Title: lib2to3 requires source files for fixes
Type: behavior Stage:
Components: 2to3 (2.x to 3.x conversion tool) Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, r.david.murray, steve.dower, tschiller, veky
Priority: normal Keywords:

Created on 2017-08-08 13:43 by tschiller, last changed 2017-08-12 03:41 by steve.dower.

Messages (9)
msg299920 - (view) Author: Todd Schiller (tschiller) Date: 2017-08-08 13:43
The lib2to3 library doesn't detect/apply any fixes if the source files aren't included with the distribution. See get_all_fix_names at https://github.com/python/cpython/blob/master/Lib/lib2to3/refactor.py#L30

This affects Azure's official Python site extensions [1], which only include bytecode pyc's for the fixes [2].

This results in bad installs when using setuptools to install packages that leverage the use_2to3 flag during the install [3]

[1] https://github.com/Azure/azure-python-siteextensions
[2] https://github.com/Azure/azure-python-siteextensions/issues/14
[3] https://github.com/pypa/setuptools/issues/1120#issuecomment-320769664
msg299924 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-08-08 14:22
What are you reporting as the bug here?  2to3 obviously can't work without the source, so based just on what you have written here this sounds like an Azure bug.
msg299926 - (view) Author: Todd Schiller (tschiller) Date: 2017-08-08 14:28
I'm reporting that lib2to3 doesn't work without having source .py files for the fixes that are packaged with the lib2to3 library. Perhaps the get_all_fix_names method in lib2to3/refactor.py should check for either .py or .pyc files?

The source files are included with the package that I'm installing (anyjson 0.3.3 in this case).
msg299927 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-08-08 14:37
Ah, I see.  We don't really support .pyc-only distribution, though we try not to break it.

Do you want to propose a fix?
msg300100 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-08-10 15:04
It also breaks .zip file distribution, which I'm fairly sure we do explicitly support by virtue of having "python36.zip" in sys.path by default. And the ".pyc-only in a zip file" distribution is *totally* busted :) (and also officially released for Windows...)

I'm dragging in an importlib expert to the discussion, probably against his will, since this is really a problem of "how do I programmatically discover what I can import without using os.listdir()". Hopefully Brett can answer that question and we can come up with a patch.
msg300112 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-08-10 19:17
Simplest way is to do https://docs.python.org/3/library/importlib.html#importlib.util.find_spec and see if a spec can be found for the module in question. That will do the search for the module but it won't load it. This does dictate that you know the name of the module upfront, though. If you want more of a "list all modules since I don't know what I want" then you're out of luck until I develop the API (maybe 3.7?).
msg300150 - (view) Author: Vedran Čačić (veky) * Date: 2017-08-11 04:04
I'm sorry about entering the discussion without knowing any connection between the original problem and Brett's question, but isn't the latter answered by pkgutil.iter_modules (and pkgutil.walk_packages if needed)?
msg300178 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-08-11 18:03
You're technically right, Verdan, but pkgutils is kind of a hack as it isn't always using standard APIs to achieve its goals, so I personally never recommend using the module unless absolutely necessary and you know exactly how brittle it can be.
msg300187 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-08-12 03:41
Perhaps the more important question is whether 2to3 really needs to dynamically generate a list of fixers or if that should have been stopped due to YAGNI. There are only a few modules in the stdlib, and it doesn't seem to support proper namespace packages, so unless there are plans to suddenly add many more we could just make it a hard coded list.
History
Date User Action Args
2017-08-12 03:41:09steve.dowersetmessages: + msg300187
2017-08-11 18:03:23brett.cannonsetmessages: + msg300178
2017-08-11 04:04:27vekysetnosy: + veky
messages: + msg300150
2017-08-10 19:17:50brett.cannonsetmessages: + msg300112
2017-08-10 15:04:54steve.dowersetnosy: + brett.cannon, steve.dower
messages: + msg300100
2017-08-08 14:37:59r.david.murraysetmessages: + msg299927
2017-08-08 14:28:32tschillersetmessages: + msg299926
2017-08-08 14:22:23r.david.murraysetnosy: + r.david.murray
messages: + msg299924
2017-08-08 13:43:40tschillercreate