New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
importlib/__init__.py can not be loaded without __file__ - breaks cxFreeze #65083
Comments
Python 3.4 introduced a change to Lib/importlib/init.py that added the following code to it:
When attempting to use cxFreeze on a project, using Python 3.4. we ran into a problem with '__file__' identifier not being defined. I believe this is a bug since the code expects the module loaded from importlib/init.py to always have a __file__ identifier set. Python 3.4.0rc3 documentation explicitly states:
and:
A possible fix would be to skip setting the _bootstrap.__file__ attribute if the current module does not have a __file__ attribute set. I hope this can be resolved before the 3.4 final release or it will not be possible to use cxFreeze with Python 3.4 without additional workarounds in cxFreeze. Best regards, |
So you aren't running into problems with bpo-20778? (That issue exists in 3.3 as well). |
Not having __file__ defined is an odd issue as importlib.__init__ always comes from a file in CPython. I have no issue making the setting of the attribute conditional, though. |
David: Nope, the issue does not exist with 3.3. |
"I hope this can be resolved before the 3.4 final release or it will not be possible to use cxFreeze with Python 3.4 without additional workarounds in cxFreeze." It's too late for this to go into 3.4.0 unless someone can make a really good case to the release manager that this is critical enough to be a "release blocker". My guess is that it isn't but I don't use cxFreeze. Anyone? If not, then one should ask if this should be a "release blocker" for 3.4.1. If so, the priority should be set to "deferred blocker". |
I'll try to set up and post a reproducible use case tomorrow. Then you can decide. It could turn out that the use case we ran into has an easy workaround. |
It's at least a deferred blocker, and that's my instinct as to the right setting (especially since we're assuming that 3.4.1 will happen a month or so after the PyCon US sprints). I'll dig into a bit more to see if I can provoke the misbehaviour (e.g. we may be avoiding it in the test suite by always unloading the frozen one before loading from source, so cx_freeze may be able to just skip loading the bootstrap module from source entirely in both 3.3 and 3.4 with no ill effects). |
Reading https://bitbucket.org/anthony_tuininga/cx_freeze/src/default/cx_Freeze/finder.py I have a suspicion that the import system reimplementation in there is going to need updates to handle PEP-451 (I suspect there are even latent defects in relation to Python 3.3 changes). Setting this to pending for the moment - it needs someone that is more familiar than I am with cx-freeze internals to go over: The import related changes in http://docs.python.org/3/whatsnew/3.3.html#porting-to-python-3-3 In particular, bpo-18065 (path now being empty in frozen packages) seems like it might have the potential to cause trouble. |
Another relevant link: http://docs.python.org/3.4/reference/import.html Just to clarify the reason for the pending status: At the moment, I suspect this is a matter of either 3.4 revealing a latent defect in the cx-freeze import emulation (by being less tolerant of deviations from the language spec), or else that emulation needs to be updated to address one or more of the existing porting notes. In the first case, we'll just need a new porting note, in the latter, no change at all. However, an inadvertent backwards incompatible change as a result of the PEP-451 changes is also a possibility. cx-freeze digs much deeper into the import system than most tools, so it's entirely plausible that it might hit an edge case that isn't covered by our test suite. Note that Python source and bytecode files, whether imported from the filesystem or from inside a zipfile *should* have __file__ set (as should extension modules). Frozen modules won't have it set, but that shouldn't apply in the case of importlib/init.py. |
Here's a recipe I can use to reproduce the problem on my PC. Environment:
-- 1. prepare to build cx_Freeze -- Around line 649 replace the line: This is an unrelated issue that still needs to be researched. -- 2. build & install cx_Freeze -- -- 3. prepare a test project -- hello_world.py: setup.py: Note that only the 'import asyncio' line is necessary to -- 4. prepare a 'frozen' test project executable -- py34 setup.py build_exe This should create a new executable under: build\exe.win-amd64-3.4\hello_world.exe -- 5. run the prepared executable -- You should get output looking something like: Hello world
Traceback (most recent call last):
File "C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\initscripts\Cons
exec(code, m.__dict__)
File "hello_world.py", line 8, in <module>
import asyncio
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
module = _SpecMethods(spec)._load_unlocked()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
return self._load_backward_compatible()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
spec.loader.load_module(spec.name)
File "C:\Program Files\Python\Python34rc3\lib\asyncio\__init__.py", line 23, in <module>
from .locks import *
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
module = _SpecMethods(spec)._load_unlocked()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
return self._load_backward_compatible()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
spec.loader.load_module(spec.name)
File "C:\Program Files\Python\Python34rc3\lib\asyncio\locks.py", line 9, in <module>
from . import tasks
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2261, in _handle_fromlist
_call_with_frames_removed(import_, from_name)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 321, in _call_with_frames_removed
return f(*args, **kwds)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
module = _SpecMethods(spec)._load_unlocked()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
return self._load_backward_compatible()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
spec.loader.load_module(spec.name)
File "C:\Program Files\Python\Python34rc3\lib\asyncio\tasks.py", line 12, in <module>
import inspect
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
module = _SpecMethods(spec)._load_unlocked()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
return self._load_backward_compatible()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
spec.loader.load_module(spec.name)
File "C:\Program Files\Python\Python34rc3\lib\inspect.py", line 35, in <module>
import importlib.machinery
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2189, in _find_and_load_unlocked
_call_with_frames_removed(import_, parent)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 321, in _call_with_frames_removed
return f(*args, **kwds)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
module = _SpecMethods(spec)._load_unlocked()
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1200, in _load_unlocked
self._exec(module)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1129, in _exec
self.spec.loader.exec_module(module)
File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1336, in exec_module
exec(code, module.__dict__)
File "C:\Program Files\Python\Python34rc3\lib\importlib\__init__.py", line 26, in <module>
_bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py')
NameError: name '__file__' is not defined Hope this helps. Best regards, |
As mentioned by Anthony Tuininga at: The value of __file__ in the problematic importlib/init.py And this '<frozen>' string seems to be something coming from Python Hope this helps. Best regards, |
Hmm, I think we still have something weird going on: $ python3
Python 3.3.2 (default, Nov 8 2013, 13:38:57)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _frozen_importlib
>>> _frozen_importlib.__file__
'/home/ncoghlan/<frozen>'
>>>
$ ./python
Python 3.4.0rc1+ (default, Mar 11 2014, 19:49:01)
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _frozen_importlib
>>> _frozen_importlib.__file__
'/home/ncoghlan/devel/py3k/<frozen>' Perhaps the problem is specifically with frozen *packages*? I don't currently have a handy one of those to test against, so I added a comment to the BitBucket issue suggest a possible simplification of the reproducer that would confirm the theory. |
Ask per Nick's instructions I tweaked cx_Freeze to import a dummy gugu/ and its __init__.py file contains only the 8 bytes '__file__' Then running a frozen executable based on a script with the following content: import gugu results in a NameError due to the name '__file__' not being You can see more detailed information at: Hope this helps. Best regards, |
Depending on what cx_Freeze is doing with packages, I wnder if this What's New 3.4 porting note is relevant:
|
That 'what's new' item seems relevant, except that the issue here The bug seems to be caused by importlib\init.py expecting its Based on my rather slim understanding of how module 'freezing' Best regards, |
This could possibly also have been caused by a resolution to bpo-18088 (http://bugs.python.org/issue18088). See commit e873f2e67353 (http://hg.python.org/cpython/rev/e873f2e67353). |
I think msg213138 has the key: importlib is actually getting frozen in the Python sense of the module's bytecode being included in a C file and then compiled, not just copied into a zip file. When we freeze importlib._bootstrap as _frozen_importlib, importlib is brought along for the ride as well. So when Python code imports it, it's loading from the frozen copy, and __file__ is not defined. I think I can see how to fix this in cx_Freeze. |
I found the problem. _frozen_importlib is being loaded by C code which is using the equivalent of _imp.init_frozen(). importlib.machinery.FrozenImporter is doing a more controlled, manual approach of using _imp.get_frozen_object() and then initializing the module itself. Problem is that the former sets __file__ to '<frozen>' and the latter doesn't; this was an unintended side-effect of updating FrozenImporter for PEP-451 as this was the only way to make exec_module() work w/o serious C hacking. You can see the difference this way: >>> import _imp
>>> __hello__again = _imp.init_frozen('__hello__')
Hello world!
>>> __hello__again.__spec__.has_location
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'has_location'
>>> __hello__again.__file__
'<frozen>' compared to: >>> import __hello__
Hello world!
>>> __hello__.__spec__.has_location
False
>>> __hello__.__spec__.origin
'frozen'
>>> __hello__.__file__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__file__' The clean solution is to fix PyImport_ImportFrozenModuleObject() to stop setting __file__ and then fix importlib/init.py to deal with the lack of file path (this should implicitly fix _frozen_importlib when it's __spec__ is set in _setup). The more backwards-compatible would be to toss '<frozen>' back in for __spec__.origin and set __spec__.has_location to True which is all semantically wrong. I vote for the clean solution along with a backported note into What's New. |
+1 for the clean solution :-) |
In the clean solution, it sounds like making importlib/init.py deal with the lack of a __file__ attribute would fix the problem in cx_Freeze. We'd still need to work out whether freezing importlib into the base executable is the right thing to do, but freezing would work on Python 3.4. So that sounds good to me. |
I'm not sure Brett's plan is such a great idea. If you look at the file So I don't think we should decide what to do without first comparing the |
On Mar 12, 2014 6:04 PM, "Nick Coghlan" <report@bugs.python.org> wrote:
But the fact that has any meaning is dumb luck. Why would anyone but
I'm not seeing a use-case here. If you're freezing your code you are And that path being absolute is a 3.4 thing so there is not
|
OK, that sounds reasonable. That means I think we need to:
|
I assume you mean for 3. to fix PyImport_ImportFrozenModuleObject() and not stop resetting __file__ for _frozen_importlib in importlib.__init__ like we are currently doing. I think you're right that it could wait until 3.5 as it will only be apparent to people who are poking around with importlib._bootstrap -- which people should not be doing -- or are using imp.init_frozen() which isn't even documented anymore. |
New changeset b626f4978a28 by Brett Cannon in branch 'default': |
Step 1 was just checked in. Step 2: Jurko can you see if the uploaded patch fixes things for you? |
Yup. That's exactly how we were working around the issue before |
So is this fixed? Can we close the issue? I'm tagging 3.4.0 final soon. |
Status update:
|
bpo-20942 now covers the spurious value in _frozen_importlib.__file__, leaving this free to cover the fact that importlib/init.py doesn't support freezing in 3.4.0. |
New changeset b5b81a3eb6e6 by Brett Cannon in branch '3.4': New changeset 42ae7b2524a2 by Brett Cannon in branch 'default': |
Thanks Brett! :-) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: