classification
Title: Tools/freeze no longer works in Python 3
Type: behavior Stage: patch review
Components: Demos and Tools Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: meador.inge Nosy List: Arfrever, Trundle, barry, brett.cannon, chba, eric.smith, eric.snow, jcea, jkloth, lemburg, loewis, meador.inge, python-dev, r.david.murray
Priority: normal Keywords: 3.3regression, patch

Created on 2012-09-25 16:27 by lemburg, last changed 2014-03-31 17:46 by meador.inge. This issue is now closed.

Files
File name Uploaded Description Edit
issue16047-0.patch meador.inge, 2013-01-02 16:06
issue16047-1.patch meador.inge, 2013-01-03 01:28
site.py chba, 2014-03-21 12:36 manually patches version from Python 3.4.0
Messages (37)
msg171295 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2012-09-25 16:27
The freeze tool used for compiling Python binaries with frozen modules no longer works with Python 3.x.

It looks like it was never updated to the various path and symbols changes introduced with PEP 3149 (ABI tags) in Python 3.2.

Even with lots of symlinks to restore the non-ABI flagged names, freezing fails with a linker error in Python 3.3:

Tools/freeze> python3 freeze.py hello.py
Tools/freeze> make
config.o:(.data+0x38): undefined reference to `PyInit__imp'
collect2: ld returned 1 exit status
make: *** [hello] Error 1
msg175492 - (view) Author: Andreas Stührk (Trundle) Date: 2012-11-13 09:22
See also issue #11824 for the ABI tags changes.
msg178771 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-01-01 22:42
As mentioned, the ABI issues are being handled in issue11824.  I believe the linking problems in this issue have to do with the changes that were made in 3.3 to bootstrap importlib into Python.  I will look into it more.
msg178819 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-01-02 16:06
After applying the fix for issue11824 to tip there ended up being three more
issues:

   1. makeconfig.py should have been updated in d777f854a66e when changing
      imp to _imp.  Without this change a reference to 'PyInit__imp'
      was being generated, but that function does not actually exist.

   2. After fixing (1) I hit an issue where '_frozen_importlib' can
      not be found while bringing up the interpreter.

   3. After fixing (2) I hit the following error when running the
      simple frozen hello app:

         $ ./hello 
         Traceback (most recent call last):
      
         ...

           File "/Users/meadori/Code/src/cpython/Lib/site.py", line 578, in main
             setcopyright()
           File "/Users/meadori/Code/src/cpython/Lib/site.py", line 438, in setcopyright
             here = os.path.dirname(os.__file__)
         AttributeError: 'module' object has no attribute '__file__'

      This happens during 'initsite' and is because of a change made
      to remove the __file__ attribute from frozen modules in 702009f3c0b1.

The attached patch fixes (1) by filtering '_imp' instead of 'imp' in
makeconfig.py, (2) by making sure '_frozen_importlib' is in the frozen
module table created by freeze.py, and (3) by removing the code to delete
__file__ from frozen modules.

I am a little unsure about (3) since I am not sure why the __file__ attribute
is being removed to begin with.  eric.smith?
msg178821 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-01-02 16:22
On Jan 02, 2013, at 04:06 PM, Meador Inge wrote:

>I am a little unsure about (3) since I am not sure why the __file__ attribute
>is being removed to begin with.  eric.smith?

This is related to PEP 420, which relaxes the requirement that all modules
have a __file__ attribute.  Now they only need it if it makes sense.  At the
time, we didn't think it made sense for frozen modules to have an __file__.
FrozenImporter.module_repr() provides a reasonable repr in the absence of
__file__.

I'm not sure what to do.  I still think it doesn't make a lot of sense for
frozen modules to have an __file__, but perhaps practicality beats purity
here.
msg178823 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2013-01-02 16:29
What Barry said. :)

I haven't had time to check yet: but why does site.py need the __file__ attribute? Maybe that's the actual problem.
msg178824 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-01-02 16:34
On Jan 02, 2013, at 04:29 PM, Eric V. Smith wrote:

>I haven't had time to check yet: but why does site.py need the __file__
>attribute? Maybe that's the actual problem.

It uses it to find the license text when you type license() at the interactive
prompt.  I think setcopyright() should just ignore that if os.__file__ isn't
defined.  Why would you be printing the license at a prompt in a frozen
Python?
msg178825 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-01-02 16:53
I agree that it makes no sense to define __file__ for frozen modules. Originally they did because that was the only way to tell if a module was a builtin module or not, but with the imp module's API on top of sys.builtin_module_names, there is no need to maintain this invariant.

As for site.py requiring the module to have a __file__ on os, I think that is somewhat bogus as well and should be optional so issue (3) for Meador should be to patch site.py to not flat-out require os.__file__ exist.
msg178877 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-01-03 01:28
Thanks for the feedback everyone -- that helped.  Here is a new patch
which addresses issue (3) by not requiring os.__file__ to exist.

The patch from issue11824 fixes freeze for 3.2 completely.  The patch
from issue11824 plus this patch fixes freeze for 3.3 and 3.4.  I tested
both `make install` and Python source tree versions (i.e. freeze.py -p).
A frozen "Hello, world" app works fine in all cases.

I also verified that 'license' works from a frozen app.  You will see
something like the following instead of the full licensing text on stdout:

   See http://www.python.org/3.4/license.html
msg214352 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-21 12:36
Thanks for the patches. After applying issue11824-0.patch and then isse16047-1.patch I am successfully able to freeze a hello world python script under ubuntu 14.04 with python 3.4 rc3. I have attached my new site.py file, since the automatical patch application did not run through.

However, in my productive project I need the psycopg2 library for database access. Whenever using 'import psycopg2' in my code the binary shows a runtime error that it cannot find the module psycopg2._psycopg. A one line script suffices for reproduction. Executing the script without freezing it works like a charm. In Python 3.2 it helped to create  in the working dir a psycopg2 subdir with a symlink  _psycopg.so -> /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython-32mu.so . This seems no longer help. Is this a bug/feature of freeze or psycopg2? Thanks for your interest and help.
msg214400 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-03-21 18:40
Did you want to update your patch for Python 3.4 and 3.5, Meador?
msg214427 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2014-03-22 01:28
Sure.  I will refresh it tonight or sometime tomorrow.
msg214794 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 08:58
Same issue with external libraries under (pachted) Python 3.4.0 final on Ubuntu 14.04 LTS or Debian Wheezy/Sid.

Meader, is there any option/possibility zu use an external library (like psycopg2.psycopg.so)from the freezed binary? I alreday tried to manually add various -L -l options to the linker command of the generated make file or adjust LD_LIBRARY_PATH and so on ... always without any success. Unfortunately, I need this for a large productive system Praktomat. Please help.

Thanks again.

Chris
msg214797 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 09:16
I also issued a ticket at the bugtracker of the psycopg2 project:
http://psycopg.lighthouseapp.com/projects/62710/tickets/201
msg214802 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-03-25 10:07
Christian: please don't use this bug tracker to get help. Please use e.g. python-list to ask questions on how to use Python. To answer your question: in theory, you have the choice to either continue to use dynamic loading from the frozen interpreter, or to make psycopg.so a builtin module. Freeze won't help here, as it only deals with modules written in Python.
msg214805 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 10:36
Will these patches still make it into the Python 3.4 branch ?
msg214806 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 10:37
Martin: this is clearly a bug, as it is now (Python 3.3 onwards) impossible to use an external module (in a .so) from a frozen binary. 
The phrase "please help" was intended to fix the bug. If there is a new option then this would also result in some kind of a bug, since nowhere documented. It turns freeze.py unusable in many situations. At least it cannot be done as in Python 3.2 (described above) and the many other methods I tried. The mailing list can only be a last option, since Meador's patches are nowhere officially included and thus not very widespread. Unfortunately, both of your advices only work in theory for me. However, I am glad to have found in you a person who gives at least some advices.

Meanwhile, the psycopg guys are do (also) not see a bug on their side:
http://psycopg.lighthouseapp.com/projects/62710/tickets/201
msg214808 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 10:45
On 25.03.2014 11:37, Christian Bachmaier wrote:
> 
> Martin: this is clearly a bug, as it is now (Python 3.3 onwards) impossible to use an external module (in a .so) from a frozen binary. 

Are you sure about this ?

If you freeze an application which imports just the _psycopg*.so file
and make sure the linker can find it, does the import still fail from
the frozen app ?
msg214817 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 12:02
# ldd hello
        linux-vdso.so.1 =>  (0x00007fffd677e000)
        libpython3.4m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0 (0x00007f968c6c2000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f968c4a4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f968c0dd000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f968beb3000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f968bc9a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f968ba95000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f968b892000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f968b58c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f968ccfb000)

and (even strace) does not show an attempt to acces the so-file. Also tried to add

-L/usr/lib/python3/dist-packages/psycopg2 -l_psycopg.cpython-34m-x86_64-linux-gnu.so

and similar without success.

Can anyone try to reproduce it, a freeze (with patches) of hello.py containing only

import _psycopg2

or maybe another external library? In the first case apt-get install python3-psycopg2 is necessary.
msg214824 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2014-03-25 13:33
Apologies for not replying over the weekend.  I am still looking into this one.
msg214826 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 14:28
On 25.03.2014 13:02, Christian Bachmaier wrote:
> 
> Christian Bachmaier added the comment:
> 
> # ldd hello
>         linux-vdso.so.1 =>  (0x00007fffd677e000)
>         libpython3.4m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0 (0x00007f968c6c2000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f968c4a4000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f968c0dd000)
>         libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f968beb3000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f968bc9a000)
>         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f968ba95000)
>         libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f968b892000)
>         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f968b58c000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f968ccfb000)
> 
> and (even strace) does not show an attempt to acces the so-file. Also tried to add
> 
> -L/usr/lib/python3/dist-packages/psycopg2 -l_psycopg.cpython-34m-x86_64-linux-gnu.so
> 
> and similar without success.
> 
> Can anyone try to reproduce it, a freeze (with patches) of hello.py containing only
> 
> import _psycopg2
> 
> or maybe another external library? In the first case apt-get install python3-psycopg2 is necessary.

Christian, what you're expecting is not what freeze can offer.

The shared library is not linked into the resulting binary
by simply having an import in the Python file. freeze does
support adding the external library statically, but it's not
easy.

See the eGenix PyRun source archive for how it's done:

    http://www.egenix.com/products/python/PyRun/

The question I raised was whether running "hello" will
fail to import the shared library _pyscopg2*.so or not.

If that's the case (and only then), you have found a bug that
needs fixing. If not, please ask your usage questions on other
lists. This ticket is about getting freeze working again for
Python 3.x, not about its usage.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com
msg214829 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 15:14
The shared library is not linked into the resulting binary
by simply having an import in the Python file.

Yes. This is why (at least in Python 3.2) it must be in the right path (subdirectory), see above.

> freeze does
> support adding the external library statically, but it's not
> easy.

Unfortunately, Debian/Ubuntu does not deliver a static version of psycopg2. So I'd like to use the dynamic version. This is definitively possible with Python 3.2 x86, again, see above.

> The question I raised was whether running "hello" will
> fail to import the shared library _pyscopg2*.so or not.

That's a good question. I think so, but how can I test that? At least the (only) way in Python 3.2 does not work any more. Even with Python 3.2 there must be a link in the subdirectory as shown above. It is not enough to have it only in the usual installation directory /usr/lib/python3/dist-packages/psycopg2/_psycopg.xxx.so
Even any set LD_LIBRARY_PATH is ignored, like also putting it in /usr/lib as far as I can see.

> This ticket is about getting freeze working again for
> Python 3.x

Right. So we should test the library feature which worked somehow magically in Python 3.2. Then we will see if it is a bug. My statement is that it is a bug.

Thanks again to all.
msg214831 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 15:21
Sorry I forgot: PyRun seems only support Python 2.x.

The only other freeze tool I know for Pyhton3 code is cx_freeze. I would prefere the vanilla freeze of the python distribution itself and as far as I can see using cx_freeze makes more problems for me as it may solve. But that is not a discussion for this thread or forum.
msg214832 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 15:40
On 25.03.2014 16:21, Christian Bachmaier wrote:
> 
> Sorry I forgot: PyRun seems only support Python 2.x.

Right, because PyRun uses freeze and freeze currently does not work
for Python 3. Which is what this ticket is all about and why
I opened it.

To test what I asked for, please run freeze on this script:

"""
import _psycopg2
print ('Works.')
"""

If it prints 'Works.', then your problem is unrelated to this
ticket. If it fails with an import error or some other error
related to loading the shared lib, then it may be a problem
with freeze and is likely related to the new import lib
machinery.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com
msg214839 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 16:47
> To test what I asked for, please run freeze on this script:
> 
> """
> import _psycopg2
> print ('Works.')
> """

$ xxx/freeze.py hello.py
$ make
$ ./hello
Traceback (most recent call last):
  File "hello.py", line 3, in <module>
    import _psycopg2
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_psycopg2'

Btw the same with import psycopg2, psycopg2._psycopg, or _psycopg. Event with the subdir link as explained above. The first one (import psycopg2) is the one which operates in interpretation mode.

> If it prints 'Works.', then your problem is unrelated to this
> ticket.

Nop, prints the error message instead of 'Works.'

> If it fails with an import error or some other error
> related to loading the shared lib, then it may be a problem
> with freeze and is likely related to the new import lib
> machinery.

=> we have a bug if Marc-Andre is right. Thanks.

Chris
msg214840 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 17:17
On 25.03.2014 17:47, Christian Bachmaier wrote:
> 
> Christian Bachmaier added the comment:
> 
>> To test what I asked for, please run freeze on this script:
>>
>> """
>> import _psycopg2

Sorry. The above should have read "import _psycopg".

>> print ('Works.')
>> """
> 
> $ xxx/freeze.py hello.py
> $ make
> $ ./hello
> Traceback (most recent call last):
>   File "hello.py", line 3, in <module>
>     import _psycopg2
>   File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
>     return _find_and_load_unlocked(name, import_)
>   File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
>     raise ImportError(_ERR_MSG.format(name), name=name)
> ImportError: No module named '_psycopg2'
> 
> Btw the same with import psycopg2, psycopg2._psycopg, or _psycopg. Event with the subdir link as explained above. The first one (import psycopg2) is the one which operates in interpretation mode.

Ok, now we're getting closer.

Could you run this to have Python print the locations where
it looks for the shared lib:

export PYTHONVERBOSE=2
./hello

This should print a long list of messages such as these:

import os # frozen
import errno # builtin
import posix # builtin
import posixpath # frozen
...

to stderr. Of those only the lines which mention _psycopg are relevant.

Please make sure that the dir where the .so file resides is
included in this list. If not, you will need to adjust PYTHONPATH
accordingly.

If the dir is mentioned in the listing, we have to dig deeper
using strace or similar.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com
msg214842 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-25 18:12
$ cat hello.py
import _psycopg
print('Works.')
$ export PYTHONVERBOSE=2
$ ./hello 2> res.txt
$ cat res.txt | grep psycopg           
# trying /export/scratch/chris/pgtest/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /export/scratch/chris/pgtest/_psycopg.cpython-34m.so
# trying /export/scratch/chris/pgtest/_psycopg.abi3.so
# trying /export/scratch/chris/pgtest/_psycopg.so
# trying /export/scratch/chris/pgtest/_psycopg.py
# trying /export/scratch/chris/pgtest/_psycopg.pyc
# trying /usr/lib/python3.4/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3.4/_psycopg.cpython-34m.so
# trying /usr/lib/python3.4/_psycopg.abi3.so
# trying /usr/lib/python3.4/_psycopg.so
# trying /usr/lib/python3.4/_psycopg.py
# trying /usr/lib/python3.4/_psycopg.pyc
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.cpython-34m.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.abi3.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.py
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.pyc
# trying /usr/lib/python3.4/lib-dynload/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.cpython-34m.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.abi3.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.py
# trying /usr/lib/python3.4/lib-dynload/_psycopg.pyc
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.cpython-34m.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.abi3.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.py
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.pyc
# trying /usr/lib/python3/dist-packages/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3/dist-packages/_psycopg.cpython-34m.so
# trying /usr/lib/python3/dist-packages/_psycopg.abi3.so
# trying /usr/lib/python3/dist-packages/_psycopg.so
# trying /usr/lib/python3/dist-packages/_psycopg.py
# trying /usr/lib/python3/dist-packages/_psycopg.pyc
    import _psycopg
ImportError: No module named '_psycopg'

----

$ cat hello.py
import _psycopg
print('Works.')
$ export PYTHONPATH=/usr/lib/python3/dist-packages/psycopg2:$PYTHONPATH
$ ./hello
Works.

----

The correct import (see the doc of psycopg2) is import psycopg2. Otherwise a valid script does even not run in interpreted mode. The library lies in /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython...so Maybe there is the Problem. One has to Import the parent dir of the so-file, which Triggers the bug.

$ cat hello.py
import psycopg2
print('Works.')
$ export PYTHONVERBOSE=2
$ ./hello 2> res.txt
$ cat res.txt | grep psycopg           
    import psycopg2
  File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 67, in <module>
    from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: No module named 'psycopg2._psycopg'
# destroy psycopg2


Marc-Andre: would you mind to install python3-psycopg2 to see that in real?
msg214843 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-03-25 18:27
OK, so trying to import around the package was definitely why the first instance didn't work so that's all expected.

As for the failure when importing psycopg2, my guess is that the freezing of psycopg2.__init__ is not setting __path__ to anything reasonable to work with dynamically loading psycopg2._psycopg. That really shouldn't really ever work anyway since that just doesn't make sense from the perspective of freezing a package unless you made the extension module a built-in module, but I don't think submodules are supported in that case right now anyway.

MAL, do you agree with that assessment?
msg214844 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 18:41
On 25.03.2014 19:27, Brett Cannon wrote:
> 
> Brett Cannon added the comment:
> 
> OK, so trying to import around the package was definitely why the first instance didn't work so that's all expected.
> 
> As for the failure when importing psycopg2, my guess is that the freezing of psycopg2.__init__ is not setting __path__ to anything reasonable to work with dynamically loading psycopg2._psycopg. That really shouldn't really ever work anyway since that just doesn't make sense from the perspective of freezing a package unless you made the extension module a built-in module, but I don't think submodules are supported in that case right now anyway.
> 
> MAL, do you agree with that assessment?

Using C extensions embedded in Python packages is supported in
Python 2's freeze - but not directly:

This works because Python2 search for the module in the top level
directories in case it cannot find the shared mod in the package dir
(which in the case of frozen packages does not exist). So you ship the
frozen app together with the .so shared module in the same directory
or setup sys.path to point to whatever dir you use for this.

I'll have to have a look at how the pyscopg2 package normally
imports its C extension. It's likely that they will have to use
something like this to make things work for frozen apps as well:

try:
    from psycopg2 import _psycopg
except ImportError:
    # try to find the module at the top-level
    import _psyocpg

or setup the package's .__path__ to include the top-level
dir.
msg214849 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 20:07
On 25.03.2014 19:41, M.-A. Lemburg wrote:
> I'll have to have a look at how the pyscopg2 package normally
> imports its C extension. It's likely that they will have to use
> something like this to make things work for frozen apps as well:
> 
> try:
>     from psycopg2 import _psycopg
> except ImportError:
>     # try to find the module at the top-level
>     import _psyocpg
> 
> or setup the package's .__path__ to include the top-level
> dir.

The package uses absolute imports for importing the C extension, e.g.

from psycopg2._psycopg import __version__

This cannot work in Python with frozen packages. Not in Python 2 and
not in Python 3 either.

Christian: Your only option is not to freeze the psycopg2 package
and ship it along side your frozen application or to build the
frozen app with the psycopg2 C extension built statically.

In any case, freeze is not at fault here.
msg214851 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-25 20:25
Wait, Brett :-)

The issue that Christian mentioned was just a side discussion.

We still need to fix the main problem.
msg214939 - (view) Author: Christian Bachmaier (chba) Date: 2014-03-27 07:31
Sorry guys, library loading of a freezed binary is different to interpreter mode. This is a bug in freeze, or at least an undocumented missing feature of freeze. This is no side discussion.

And, in Python 3.2 this was working! As described above, just creating a subdir psycopg2 in the working direktory of the frozen binary containing a link _psycopg.so to the library. This is a fact. I have two working production systems using this mechanism under Ubuntu 12.04 LTS.

Ok, psycopg2 seems to load the so file indirectly over __init__.py, however, the interpreter and older versions of freeze support that. The psycopg2 guys do not see any problem in that, see the closed entry linked above in their bug database.

A workaround/hack I am using successfully now is to replace any 'import psycopg2._psycopg' by 'import _psycopg' in all /usr/lib/python3/dist-packages/psycopg2/*.py files and to set PYTHONPATH=//usr/lib/python3/dist-packages/psycopg2 prior execution of the frozen binary. I found this only by playing around several days, there is NO documentation about that. The advices statically linking etc. given here are too superficial. Any more helful/concrete advices (outside of this thread) seem to cost money.

I wanted and still want help to fix this by giving detailed reports.
msg214940 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-03-27 07:39
Christian: Please understand that it was not helpful to post into this issue. The issue discussed here is separate from the issue you are having. We prefer a strict "one issue at a time" policy in this tracker.

So when this issue gets closed because Marc-Andre's original problem is solved, feel free to report your issue again.
msg214944 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-03-27 08:49
Christian, please open a separate ticket for your problem.

This ticket is about getting freeze, the tool itself, working,
not any other issue you may find with the resulting frozen binary.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com
msg215196 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-30 19:15
New changeset 001a6dc996e7 by Martin v. Löwis in branch '3.4':
Issue #16047: Fix module exception list and __file__ handling in freeze.
http://hg.python.org/cpython/rev/001a6dc996e7

New changeset 87ded2fdac4b by Martin v. Löwis in branch 'default':
Merge 3.4 (#16047)
http://hg.python.org/cpython/rev/87ded2fdac4b
msg215197 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-03-30 19:36
Thanks for the patch. It seems to work now.

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%20Freeze%203.x/builds/9
msg215255 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2014-03-31 17:46
Apologies for not getting around to applying this myself.  I had an extremely busy week with the day job last week.  Thanks for applying Martin.
History
Date User Action Args
2014-03-31 17:46:59meador.ingesetmessages: + msg215255
2014-03-30 19:36:46loewissetstatus: open -> closed
resolution: fixed
messages: + msg215197
2014-03-30 19:15:41python-devsetnosy: + python-dev
messages: + msg215196
2014-03-27 08:49:23lemburgsetmessages: + msg214944
2014-03-27 07:39:14loewissetmessages: + msg214940
2014-03-27 07:31:39chbasetmessages: + msg214939
2014-03-25 20:25:40lemburgsetstatus: closed -> open
resolution: not a bug -> (no value)
messages: + msg214851
2014-03-25 20:19:35brett.cannonsetstatus: open -> closed
resolution: not a bug
2014-03-25 20:07:45lemburgsetmessages: + msg214849
2014-03-25 18:41:41lemburgsetmessages: + msg214844
2014-03-25 18:33:49r.david.murraysetnosy: + r.david.murray
2014-03-25 18:27:19brett.cannonsetmessages: + msg214843
2014-03-25 18:12:21chbasetmessages: + msg214842
2014-03-25 17:17:22lemburgsetmessages: + msg214840
2014-03-25 16:47:36chbasetmessages: + msg214839
2014-03-25 15:40:26lemburgsetmessages: + msg214832
2014-03-25 15:21:23chbasetmessages: + msg214831
2014-03-25 15:14:16chbasetmessages: + msg214829
2014-03-25 14:28:08lemburgsetmessages: + msg214826
2014-03-25 13:33:58meador.ingesetmessages: + msg214824
2014-03-25 12:02:43chbasetmessages: + msg214817
2014-03-25 10:45:19lemburgsetmessages: + msg214808
2014-03-25 10:37:51chbasetmessages: + msg214806
2014-03-25 10:36:36lemburgsetmessages: + msg214805
2014-03-25 10:07:30loewissetmessages: + msg214802
2014-03-25 09:16:46chbasetmessages: + msg214797
2014-03-25 08:58:16chbasetmessages: + msg214794
2014-03-22 01:28:13meador.ingesetmessages: + msg214427
2014-03-21 18:40:03brett.cannonsetmessages: + msg214400
versions: + Python 3.5, - Python 3.3
2014-03-21 12:37:00chbasetfiles: + site.py
nosy: + chba
messages: + msg214352

2013-01-03 01:29:46meador.ingesetassignee: meador.inge
2013-01-03 01:28:42meador.ingesetfiles: + issue16047-1.patch
versions: - Python 3.2
messages: + msg178877

assignee: meador.inge -> (no value)
stage: needs patch -> patch review
2013-01-02 16:53:54brett.cannonsetmessages: + msg178825
2013-01-02 16:34:40barrysetmessages: + msg178824
2013-01-02 16:29:35eric.smithsetmessages: + msg178823
2013-01-02 16:22:15barrysetmessages: + msg178821
2013-01-02 16:06:45meador.ingesetfiles: + issue16047-0.patch

nosy: + brett.cannon, eric.smith
messages: + msg178819

keywords: + patch
2013-01-01 22:42:31meador.ingesetmessages: + msg178771

assignee: meador.inge
keywords: + 3.3regression
type: behavior
stage: needs patch
2012-12-29 06:04:01Arfreversetnosy: + Arfrever
2012-12-29 03:20:27meador.ingesetnosy: + meador.inge
2012-11-13 09:22:05Trundlesetnosy: + Trundle
messages: + msg175492
2012-11-13 07:14:15pitrousetnosy: + loewis, barry
2012-11-13 06:28:51eric.snowsetnosy: + eric.snow
2012-09-26 00:39:27jceasetnosy: + jcea
2012-09-25 17:06:13jklothsetnosy: + jkloth
2012-09-25 16:27:39lemburgcreate