Python 3.4 introduced a change to Lib/importlib/__init__.py that added the following code to it:

> else:
>     # importlib._bootstrap is the built-in import, ensure we don't create
>     # a second copy of the module.
>     _bootstrap.__name__ = 'importlib._bootstrap'
>     _bootstrap.__package__ = 'importlib'
>     _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py')
>     sys.modules['importlib._bootstrap'] = _bootstrap

When attempting to use cxFreeze on a project, using Python 3.4. we ran into a problem with '__file__' identifier not being defined.

I believe this is a bug since the code expects the module loaded from importlib/__init__.py to always have a __file__ identifier set.


Python 3.4.0rc3 documentation explicitly states:

> __file__ is optional. If set, this attribute’s value must be a
> string. The import system may opt to leave __file__ unset if it
> has no semantic meaning (e.g. a module loaded from a database).

  and:

> Ultimately, the loader is what makes use of __file__ and/or
> __cached__.


A possible fix would be to skip setting the _bootstrap.__file__ attribute if the current module does not have a __file__ attribute set.


I hope this can be resolved before the 3.4 final release or it will not be possible to use cxFreeze with Python 3.4 without additional workarounds in cxFreeze.

Best regards,
  Jurko Gospodnetić

So you aren't running into problems with issue 20778?  (That issue exists in 3.3 as well).

Not having __file__ defined is an odd issue as importlib.__init__ always comes from a file in CPython. I have no issue making the setting of the attribute conditional, though.

David: Nope, the issue does not exist with 3.3.

"I hope this can be resolved before the 3.4 final release or it will not be possible to use cxFreeze with Python 3.4 without additional workarounds in cxFreeze."

It's too late for this to go into 3.4.0 unless someone can make a really good case to the release manager that this is critical enough to be a "release blocker".  My guess is that it isn't but I don't use cxFreeze.  Anyone?  If not, then one should ask if this should be a "release blocker" for 3.4.1.  If so, the priority should be set to "deferred blocker".

I'll try to set up and post a reproducible use case tomorrow. Then you can decide. It could turn out that the use case we ran into has an easy workaround.

It's at least a deferred blocker, and that's my instinct as to the right setting (especially since we're assuming that 3.4.1 will happen a month or so after the PyCon US sprints).

I'll dig into a bit more to see if I can provoke the misbehaviour (e.g. we may be avoiding it in the test suite by always unloading the frozen one before loading from source, so cx_freeze may be able to just skip loading the bootstrap module from source entirely in both 3.3 and 3.4 with no ill effects).

Reading https://bitbucket.org/anthony_tuininga/cx_freeze/src/default/cx_Freeze/finder.py I have a suspicion that the import system reimplementation in there is going to need updates to handle PEP 451 (I suspect there are even latent defects in relation to Python 3.3 changes).

Setting this to pending for the moment - it needs someone that is more familiar than I am with cx-freeze internals to go over:

The import related changes in http://docs.python.org/3/whatsnew/3.3.html#porting-to-python-3-3
The import related changes in http://docs.python.org/dev/whatsnew/3.4.html#changes-in-the-python-api
The import system update in http://www.python.org/dev/peps/pep-0451/

In particular, issue 18065 (__path__ now being empty in frozen packages) seems like it might have the potential to cause trouble.

Another relevant link: http://docs.python.org/3.4/reference/import.html

Just to clarify the reason for the pending status:

At the moment, I suspect this is a matter of either 3.4 revealing a latent defect in the cx-freeze import emulation (by being less tolerant of deviations from the language spec), or else that emulation needs to be updated to address one or more of the existing porting notes.

In the first case, we'll just need a new porting note, in the latter, no change at all.

However, an inadvertent backwards incompatible change as a result of the PEP 451 changes is also a possibility. cx-freeze digs much deeper into the import system than most tools, so it's entirely plausible that it might hit an edge case that isn't covered by our test suite.

Note that Python source and bytecode files, whether imported from the filesystem or from inside a zipfile *should* have __file__ set (as should extension modules). Frozen modules won't have it set, but that shouldn't apply in the case of importlib/__init__.py.

Here's a recipe I can use to reproduce the problem on my PC.

Environment:
  * Windows 7 SP1 x64
  * Python 3.4.0rc3
  * cx_Freeze checkout from its current HEAD
    * repository: https://bitbucket.org/anthony_tuininga/cx_freeze
    * HEAD commit: 52b63b3296843cd612cfbe047a9f6529df4c0444
    * I'm attaching a compressed checkout.
  * TortoiseHg 2.11 - just in case you're having problems using
    the cx_Freeze Hg sandbox I attached to this comment and want
    to know if it is Hg version related.
  * Microsoft Visual Studio 2010 C/C++ compiler.
    * Exact cl.exe version information: "Microsoft (R) 32-bit
      C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86"

-- 1. prepare to build cx_Freeze --
To build the given cx_Freeze project using Python 3.4 the
following tweak must first be done in Python's
Lib\distutils\msvc9compiler.py module:

Around line 649 replace the line:
  if mfinfo is not None:
with:
  if mfinfo is not None and target_desc != "executable":

This is an unrelated issue that still needs to be researched.

-- 2. build & install cx_Freeze --
Standard Python package installation - go to your cx_Freeze
sandbox project folder and run "py34 setup.py install".
This should result in "cx_Freeze-4.3.2-py3.4-win-amd64.egg"
folder being added under your Python 3.4 installation's
site-packages folder.

-- 3. prepare a test project --
Create a new folder and add the following files to it:

hello_world.py:
  print("Hello world")
  import asyncio
  import sys
  print(sys.version_info)

setup.py:
  from cx_Freeze import setup, Executable
  setup(name="HelloWorld",
      version="0.1",
      description="HelloWorld",
      executables=[Executable("hello_world.py", base="Console")])

Note that only the 'import asyncio' line is necessary to
reproduce the issue. The rest is just useful output for the
user/tester.

-- 4. prepare a 'frozen' test project executable --
Go to the test project folder and run the following command:

  py34 setup.py build_exe

This should create a new executable under:

  build\exe.win-amd64-3.4\hello_world.exe

-- 5. run the prepared executable --

You should get output looking something like:

Hello world
Traceback (most recent call last):
  File "C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\initscripts\Cons
    exec(code, m.__dict__)
  File "hello_world.py", line 8, in <module>
    import asyncio
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\asyncio\__init__.py", line 23, in <module>
    from .locks import *
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\asyncio\locks.py", line 9, in <module>
    from . import tasks
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2261, in _handle_fromlist
    _call_with_frames_removed(import_, from_name)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 321, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\asyncio\tasks.py", line 12, in <module>
    import inspect
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\inspect.py", line 35, in <module>
    import importlib.machinery
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2189, in _find_and_load_unlocked
    _call_with_frames_removed(import_, parent)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 321, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1200, in _load_unlocked
    self._exec(module)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1129, in _exec
    self.spec.loader.exec_module(module)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1336, in exec_module
    exec(code, module.__dict__)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\__init__.py", line 26, in <module>
    _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py')
NameError: name '__file__' is not defined

  Hope this helps.

  Best regards,
    Jurko Gospodnetić

As mentioned by Anthony Tuininga at:

  https://bitbucket.org/anthony_tuininga/cx_freeze/issue/61/python-340-nameerror-name-__file__-is-not#comment-9077661

The value of __file__ in the problematic importlib/__init__.py
module with Python 3.3. is '<frozen>' while with Python 3.4 it is
not defined.

And this '<frozen>' string seems to be something coming from Python
and not cx_Freeze (I see it in 'Python/import.c' in both Python
3.3.5 & 3.4).

Hope this helps.

Best regards,
  Jurko Gospodnetić

Hmm, I think we still have something weird going on:


$ python3
Python 3.3.2 (default, Nov  8 2013, 13:38:57) 
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _frozen_importlib
>>> _frozen_importlib.__file__
'/home/ncoghlan/<frozen>'
>>> 

$ ./python
Python 3.4.0rc1+ (default, Mar 11 2014, 19:49:01) 
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _frozen_importlib
>>> _frozen_importlib.__file__
'/home/ncoghlan/devel/py3k/<frozen>'

Perhaps the problem is specifically with frozen *packages*? I don't currently have a handy one of those to test against, so I added a comment to the BitBucket issue suggest a possible simplification of the reproducer that would confirm the theory.

Ask per Nick's instructions I tweaked cx_Freeze to import a dummy
package as frozen. The dummy package has the following structure:

  gugu/
    __init__.py

and its __init__.py file contains only the 8 bytes '__file__'
(does not include the quotes).

Then running a frozen executable based on a script with the following content:

  import gugu

results in a NameError due to the name '__file__' not being
recognized.

You can see more detailed information at:

  https://bitbucket.org/anthony_tuininga/cx_freeze/issue/61/python-340-nameerror-name-__file__-is-not#comment-9084842

Hope this helps.

Best regards,
  Jurko Gospodnetić

Depending on what cx_Freeze is doing with packages, I wnder if this What's New 3.4 porting note is relevant:

  * Frozen packages no longer set ``__path__`` to a list containing the package
  name, they now set it to an empty list.  The previous behavior could cause
  the import system to do the wrong thing on submodule imports if there was
  also a directory with the same name as the frozen package.  The correct way
  to determine if a module is a package or not is to use``hasattr(module,
  '__path__')`` (:issue:`18065`).

That 'what's new' item seems relevant, except that the issue here
is related to the __file__ attribute instead of __path__. Could it
be that the same fix actually updated how the __file__ attribute is
defined as well?

The bug seems to be caused by importlib\__init__.py expecting its
__file__ attribute to be defined, thus not allowing it to be frozen
at all. :-(

Based on my rather slim understanding of how module 'freezing'
works I guess executables wanting to freeze the
importlib\__init__.py module can work around the problem by adding
code to the front of that module to hardcode its __file__ attribute
value, before freezing it. However I'd be happier with Python not
requiring that the __file__attribute be defined internally at all.
Imposing a workaround such as this on anyone wanting to freeze the
importlib package seems rather nasty.

Best regards,
  Jurko Gospodnetić

This could possibly also have been caused by a resolution to issue 18088 (http://bugs.python.org/issue18088).

See commit e873f2e67353 (http://hg.python.org/cpython/rev/e873f2e67353).

I think msg213138 has the key: importlib is actually getting frozen in the Python sense of the module's bytecode being included in a C file and then compiled, not just copied into a zip file. When we freeze importlib._bootstrap as _frozen_importlib, importlib is brought along for the ride as well. So when Python code imports it, it's loading from the frozen copy, and __file__ is not defined.

I think I can see how to fix this in cx_Freeze.

I found the problem. _frozen_importlib is being loaded by C code which is using the equivalent of _imp.init_frozen(). importlib.machinery.FrozenImporter is doing a more controlled, manual approach of using _imp.get_frozen_object() and then initializing the module itself. Problem is that the former sets __file__ to '<frozen>' and the latter doesn't; this was an unintended side-effect of updating FrozenImporter for PEP 451 as this was the only way to make exec_module() work w/o serious C hacking. You can see the difference this way:

  >>> import _imp
  >>> __hello__again = _imp.init_frozen('__hello__')
  Hello world!
  >>> __hello__again.__spec__.has_location
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  AttributeError: 'NoneType' object has no attribute 'has_location'
  >>> __hello__again.__file__
  '<frozen>'

compared to:

  >>> import __hello__
  Hello world!
  >>> __hello__.__spec__.has_location
  False
  >>> __hello__.__spec__.origin
  'frozen'
  >>> __hello__.__file__
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  AttributeError: 'module' object has no attribute '__file__'

The clean solution is to fix PyImport_ImportFrozenModuleObject() to stop setting __file__ and then fix importlib/__init__.py to deal with the lack of file path (this should implicitly fix _frozen_importlib when it's __spec__ is set in _setup). The more backwards-compatible would be to toss '<frozen>' back in for __spec__.origin and set __spec__.has_location to True which is all semantically wrong.

I vote for the clean solution along with a backported note into What's New.

+1 for the clean solution :-)

In the clean solution, it sounds like making importlib/__init__.py deal with the lack of a __file__ attribute would fix the problem in cx_Freeze. We'd still need to work out whether freezing importlib into the base executable is the right thing to do, but freezing would work on Python 3.4. So that sounds good to me.

I'm not sure Brett's plan is such a great idea. If you look at the file
attributes in my demo above, they include a leading path element (based on
the executable location), so it *is* possible to use the pkgutil data
access APIs sensibly from a frozen module (and even when there's no path
component, I believe the recommended way of invoking pkgutil would fall
back to the current directory).

So I don't think we should decide what to do without first comparing the
behaviour of frozen pkgutil.get_data dependent code between 3.3 and 3.4.

On Mar 12, 2014 6:04 PM, "Nick Coghlan" <report@bugs.python.org> wrote:
>
>
> Nick Coghlan added the comment:
>
> I'm not sure Brett's plan is such a great idea. If you look at the file
> attributes in my demo above, they include a leading path element (based on
> the executable location), so it *is* possible to use the pkgutil data
> access APIs sensibly from a frozen module (and even when there's no path
> component, I believe the recommended way of invoking pkgutil would fall
> back to the current directory).

But the fact that has any meaning is dumb luck. Why would anyone but
CPython partially freeze a package? If you go to the trouble to freeze a
module in a package you are probably going to freeze it all, not just a
subset like we do.

>
> So I don't think we should decide what to do without first comparing the
> behaviour of frozen pkgutil.get_data dependent code between 3.3 and 3.4.

I'm not seeing a use-case here. If you're freezing your code you are
typically doing it to avoid shipping files or protecting code. If you're
doing it to avoid files then you won't want to have data files lying around
either. And if you are doing it to hide code then you are probably not
going to want your data files exposed as that could also be valuable IP.

And that path being absolute is a 3.4 thing so there is not
backwards-compatibility there.

>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue20884>
> _______________________________________

OK, that sounds reasonable. That means I think we need to:

1. add an explicit porting note to the What's New for 3.4.0 final about frozen modules no longer defining __file__

2. fix importlib.__init__ to tolerate freezing again in 3.4.1

3. remove _frozen_importlib.__file__ in 3.4.1 (or 3.5?)

I assume you mean for 3. to fix PyImport_ImportFrozenModuleObject() and not stop resetting __file__ for _frozen_importlib in importlib.__init__ like we are currently doing. I think you're right that it could wait until 3.5 as it will only be apparent to people who are poking around with importlib._bootstrap -- which people should not be doing -- or are using imp.init_frozen() which isn't even documented anymore.

New changeset b626f4978a28 by Brett Cannon in branch 'default':
Issue #20884: whatsnew: Frozen modules don't set __file__ anymore.
http://hg.python.org/cpython/rev/b626f4978a28

Step 1 was just checked in.

Step 2: Jurko can you see if the uploaded patch fixes things for you?

> Step 2: Jurko can you see if the uploaded patch fixes things for you?

Yup. That's exactly how we were working around the issue before
reporting it here. :-)

So is this fixed?  Can we close the issue?  I'm tagging 3.4.0 final soon.

Status update:

- Brett added the porting note to What's New for 3.4.0 final.

- the fact importlib.__init__ can't currently be frozen due to the incidental use of __file__ is now just a bug to be fixed in 0.4.1, and fixing that will be the trigger for closing this issue.

- I'll file a separate bug about _frozen_importlib.__file__ being set erroneously.

Issue 20942 now covers the spurious value in _frozen_importlib.__file__, leaving this free to cover the fact that importlib/__init__.py doesn't support freezing in 3.4.0.

New changeset b5b81a3eb6e6 by Brett Cannon in branch '3.4':
Issue #20884: Don't assume in importlib.__init__ that __file__ is
http://hg.python.org/cpython/rev/b5b81a3eb6e6

New changeset 42ae7b2524a2 by Brett Cannon in branch 'default':
Merge for issue #20884
http://hg.python.org/cpython/rev/42ae7b2524a2

Thanks Brett! :-)