classification
Title: Can't use lib2to3 with embeddable zip file.
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Florian Hassanen, Lukasz00500, Marcus.Smith, Oleg N, Sébastien Taylor, benjamin.peterson, dstufft, fourplusone, gregory.p.smith, kirk.d.sayre, ncoghlan, paul.moore, ppinard
Priority: normal Keywords: patch

Created on 2015-08-29 17:36 by Oleg N, last changed 2018-04-03 21:10 by Lukasz00500. This issue is now closed.

Files
File name Uploaded Description Edit
pgen.py kirk.d.sayre, 2017-05-16 20:37
pgen.py kirk.d.sayre, 2017-05-17 14:04
Pull Requests
URL Status Linked Edit
PR 4977 merged benjamin.peterson, 2017-12-22 17:28
PR 4979 merged python-dev, 2017-12-22 20:18
PR 4980 merged benjamin.peterson, 2017-12-22 20:22
PR 5034 merged benjamin.peterson, 2017-12-29 01:30
PR 5036 merged python-dev, 2017-12-29 01:54
PR 5037 merged python-dev, 2017-12-29 01:55
Messages (18)
msg249329 - (view) Author: Oleg N (Oleg N) Date: 2015-08-29 17:36
Error: [Errno 2] No such file or directory: '...\\python35.zip\\lib2to3\\Grammar.txt'. Python35.zip from this archive: https://www.python.org/ftp/python/3.5.0/python-3.5.0rc2-embed-amd64.zip. Path is correct.
msg249804 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2015-09-04 18:30
This looks more like lib2to3 doesn't support running from a .zip archive.
msg255441 - (view) Author: Florian Hassanen (Florian Hassanen) Date: 2015-11-26 23:20
got this traceback:

Traceback (most recent call last):
  File "setup.py", line 169, in <module>
    dist = setuptools.setup(**setup_params)
  File "distutils\core.py", line 134, in setup
  File "...\setuptools\setuptools\dist.py", line 299, in parse_command_line
    result = _Distribution.parse_command_line(self)
  File "distutils\dist.py", line 469, in parse_command_line
  File "...\setuptools\setuptools\dist.py", line 679, in handle_display_options
    return _Distribution.handle_display_options(self, option_order)
  File "distutils\dist.py", line 672, in handle_display_options
  File "...\setuptools\setuptools\dist.py", line 439, in print_commands
    cmdclass = ep.resolve()
  File "...\setuptools\pkg_resources\__init__.py", line 2383, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "...\setuptools\setuptools\command\build_py.py", line 10, in <module>
    from setuptools.lib2to3_ex import Mixin2to3
  File "...\setuptools\setuptools\lib2to3_ex.py", line 12, in <module>
    from lib2to3.refactor import RefactoringTool, get_fixers_from_package
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\refactor.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\fixer_util.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\pygram.py", line 32, in <module>
  File "lib2to3\pgen2\driver.py", line 121, in load_grammar
  File "lib2to3\pgen2\pgen.py", line 385, in generate_grammar
  File "lib2to3\pgen2\pgen.py", line 15, in __init__
FileNotFoundError: [Errno 2] No such file or directory: '...\\python-3.5.0-embed-amd64\\python35.zip\\lib2to3\\Grammar.txt'


triggered e.g. by "python setup.py --help-commands" (when building setuptools)

workaround is to extract "...\python-3.5.0-embed-amd64\python35.zip" into a __directory__ named exactly the same (hence a directory named "python35.zip" which replaces the original zip file)
msg269765 - (view) Author: (fourplusone) Date: 2016-07-03 12:28
The problem is indeed that lib2to3 fails to load the Grammar.txt file if it is located inside a .zip archive.

I think one way to fix it is using of the "get_data" function of the loader  https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader.get_data

However, this would make caching the Grammar more difficult, as one cannot write back the pickled dict to a zip file

Alternatively one could generate an importable *.pyc file from the grammar during the build process
msg269779 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-04 04:54
(issue title updated to indicate this appears to be lib2to3 related and added Benjamin Peterson to the nosy list)

The build process already generates a _sysconfigdata.py file in order to pass along build time details to the sysconfig module in a way that isn't dependent on having otherwise build-time only files accessible at runtime.

Doing something similar with a _lib2to3grammar.py file seems like it could be a reasonable approach to making lib2to3 zip archive friendly, but I'm not familiar enough with pgen2's internals to say how feasible that would be.
msg285753 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-01-18 20:55
I'm not sure I see why you'd want to run lib2to3 with the embedded distribution anyway. Surely you'd do any 2to3 conversion in a normal development installation, and then simply bundle the resulting converted scripts with your application and the embedded interpreter?
msg285756 - (view) Author: Sébastien Taylor (Sébastien Taylor) Date: 2017-01-18 21:14
Are there existing tools for bundling the python35.zip (and adding additional packages)?  I personally ran into this issue by using pip to add additional modules to my embeddable python but if there's a better way to migrate from devel to embeddable python I'd rather not have to use pip for this.
msg292529 - (view) Author: Philippe Pinard (ppinard) Date: 2017-04-28 12:08
As Sébastien Taylor, I ran into the same problem. The workaround I found was to unzip the content of python35.zip and put it in the Lib/ folder.
msg293780 - (view) Author: Kirk Sayre (kirk.d.sayre) Date: 2017-05-16 20:37
I've been fighting with this issue today trying to import networkx in the embedded Python shipped with IDA-Pro. I have implemented a local fix to pgen.py that allows networkx to import and be used in IDAPython scripts. I added a check to see if a .zip archive appears in the path of the file loaded in the constructor and if so I use the zipfile package to open the archived file rather than the standard file open() function.

I've uploaded the modified pgen.py in case any one is interested.
msg293855 - (view) Author: Kirk Sayre (kirk.d.sayre) Date: 2017-05-17 14:04
Here is a more robust version of my local fix for this problem.
msg294105 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-05-21 18:03
re: Paul's "I'm not sure I see why you'd want to run lib2to3 with the embedded distribution anyway. Surely you'd do any 2to3 conversion in a normal development installation"

lib2to3 is a library useful for doing things with Python code.  Not just a stand alone tool.

Kirk: Could you please post your change as a pull request on https://github.com/python/cpython/ rather than attaching individual files here.  a full file isn't a reviewable change.

Also, Grammar.txt is loaded and saved as a cached pickle file for speed.  When packaging up an embedded stdlib bundle I recommend shipping that pickle.  (loaded in pgen2/grammar.py).

both probably need to use https://docs.python.org/3.6/library/pkgutil.html#pkgutil.get_data to read the file out of the package rather than assuming it lives on a filesystem.

using the zipfile module within lib2to3 to try and figure out when it should look somewhere other than a filesystem is the wrong approach.
msg308940 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-12-22 20:18
New changeset 8a5877165e993afb2633cd48da5222326d3f6e0e by Benjamin Peterson in branch 'master':
bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (#4977)
https://github.com/python/cpython/commit/8a5877165e993afb2633cd48da5222326d3f6e0e
msg308941 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-12-22 20:51
New changeset c1b8eb8006ed79bbf91dc7d89918f39cc10b4fe6 by Benjamin Peterson (Miss Islington (bot)) in branch '3.6':
bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) (#4979)
https://github.com/python/cpython/commit/c1b8eb8006ed79bbf91dc7d89918f39cc10b4fe6
msg308942 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-12-22 20:52
New changeset 770a802df7057bcf0d72bbc01026fac8d396e1b3 by Benjamin Peterson in branch '2.7':
[2.7] bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) (#4980)
https://github.com/python/cpython/commit/770a802df7057bcf0d72bbc01026fac8d396e1b3
msg309153 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-12-29 01:54
New changeset e5f7dccefaa8d97ab53b3051acbb4a4d49379dc4 by Benjamin Peterson in branch 'master':
make PatternCompiler use the packaged grammar if possible (more bpo-24960) (#5034)
https://github.com/python/cpython/commit/e5f7dccefaa8d97ab53b3051acbb4a4d49379dc4
msg309154 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-12-29 02:05
New changeset 417f76a20e93044e422fb328a2d8e200fc6df295 by Benjamin Peterson (Miss Islington (bot)) in branch '2.7':
make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) (#5037)
https://github.com/python/cpython/commit/417f76a20e93044e422fb328a2d8e200fc6df295
msg309155 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-12-29 02:12
New changeset 85f71aa9d6f834c7d64e979009c8fda0f19b585d by Benjamin Peterson (Miss Islington (bot)) in branch '3.6':
make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) (#5036)
https://github.com/python/cpython/commit/85f71aa9d6f834c7d64e979009c8fda0f19b585d
msg314896 - (view) Author: Lukasz (Lukasz00500) Date: 2018-04-03 21:10
I have simmilar problem but im using python 2.7 on android. So I think I cant use this solution 
My problem:
https://bugs.python.org/issue33208
History
Date User Action Args
2018-04-05 04:47:10benjamin.petersonlinkissue33208 superseder
2018-04-03 21:10:50Lukasz00500setnosy: + Lukasz00500
messages: + msg314896
2017-12-29 02:12:43benjamin.petersonsetmessages: + msg309155
2017-12-29 02:05:09benjamin.petersonsetmessages: + msg309154
2017-12-29 01:55:23python-devsetpull_requests: + pull_request4922
2017-12-29 01:54:26python-devsetpull_requests: + pull_request4921
2017-12-29 01:54:14benjamin.petersonsetmessages: + msg309153
2017-12-29 01:30:32benjamin.petersonsetpull_requests: + pull_request4920
2017-12-22 20:52:37benjamin.petersonsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-12-22 20:52:12benjamin.petersonsetmessages: + msg308942
2017-12-22 20:51:48benjamin.petersonsetmessages: + msg308941
2017-12-22 20:22:31benjamin.petersonsetpull_requests: + pull_request4869
2017-12-22 20:18:51python-devsetpull_requests: + pull_request4868
2017-12-22 20:18:40benjamin.petersonsetmessages: + msg308940
2017-12-22 17:28:47benjamin.petersonsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request4867
2017-12-22 13:54:56r.david.murraylinkissue32407 superseder
2017-05-21 18:03:21gregory.p.smithsetversions: + Python 3.6, Python 3.7
nosy: + gregory.p.smith

messages: + msg294105

stage: needs patch
2017-05-17 14:04:06kirk.d.sayresetfiles: + pgen.py

messages: + msg293855
2017-05-16 20:37:35kirk.d.sayresetfiles: + pgen.py
versions: + Python 2.7, - Python 3.5
nosy: + kirk.d.sayre

messages: + msg293780
2017-04-28 12:08:16ppinardsetnosy: + ppinard
messages: + msg292529
2017-01-18 21:14:35Sébastien Taylorsetmessages: + msg285756
2017-01-18 20:55:26paul.mooresetmessages: + msg285753
2017-01-18 20:44:41Sébastien Taylorsetnosy: + Sébastien Taylor
2016-07-04 04:54:10ncoghlansetnosy: + benjamin.peterson

messages: + msg269779
title: Can't use pip or easy_install with embeddable zip file. -> Can't use lib2to3 with embeddable zip file.
2016-07-03 12:28:32fourplusonesetnosy: + fourplusone
messages: + msg269765
2015-11-26 23:20:50Florian Hassanensetnosy: + Florian Hassanen
messages: + msg255441
2015-09-04 18:30:03dstufftsetmessages: + msg249804
2015-09-04 18:22:33terry.reedysetnosy: + paul.moore, ncoghlan, dstufft, Marcus.Smith
2015-08-29 17:36:22Oleg Ncreate