-
-
Notifications
You must be signed in to change notification settings - Fork 29.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tools/freeze no longer works in Python 3 #60251
Comments
The freeze tool used for compiling Python binaries with frozen modules no longer works with Python 3.x. It looks like it was never updated to the various path and symbols changes introduced with PEP-3149 (ABI tags) in Python 3.2. Even with lots of symlinks to restore the non-ABI flagged names, freezing fails with a linker error in Python 3.3: Tools/freeze> python3 freeze.py hello.py |
See also issue bpo-11824 for the ABI tags changes. |
As mentioned, the ABI issues are being handled in bpo-11824. I believe the linking problems in this issue have to do with the changes that were made in 3.3 to bootstrap importlib into Python. I will look into it more. |
After applying the fix for bpo-11824 to tip there ended up being three more
$ ./hello
Traceback (most recent call last):
...
The attached patch fixes (1) by filtering '_imp' instead of 'imp' in I am a little unsure about (3) since I am not sure why the __file__ attribute |
On Jan 02, 2013, at 04:06 PM, Meador Inge wrote:
This is related to PEP-420, which relaxes the requirement that all modules I'm not sure what to do. I still think it doesn't make a lot of sense for |
What Barry said. :) I haven't had time to check yet: but why does site.py need the __file__ attribute? Maybe that's the actual problem. |
On Jan 02, 2013, at 04:29 PM, Eric V. Smith wrote:
It uses it to find the license text when you type license() at the interactive |
I agree that it makes no sense to define __file__ for frozen modules. Originally they did because that was the only way to tell if a module was a builtin module or not, but with the imp module's API on top of sys.builtin_module_names, there is no need to maintain this invariant. As for site.py requiring the module to have a __file__ on os, I think that is somewhat bogus as well and should be optional so issue (3) for Meador should be to patch site.py to not flat-out require os.__file__ exist. |
Thanks for the feedback everyone -- that helped. Here is a new patch The patch from bpo-11824 fixes freeze for 3.2 completely. The patch I also verified that 'license' works from a frozen app. You will see |
Thanks for the patches. After applying bpo-11824-0.patch and then isse16047-1.patch I am successfully able to freeze a hello world python script under ubuntu 14.04 with python 3.4 rc3. I have attached my new site.py file, since the automatical patch application did not run through. However, in my productive project I need the psycopg2 library for database access. Whenever using 'import psycopg2' in my code the binary shows a runtime error that it cannot find the module psycopg2._psycopg. A one line script suffices for reproduction. Executing the script without freezing it works like a charm. In Python 3.2 it helped to create in the working dir a psycopg2 subdir with a symlink _psycopg.so -> /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython-32mu.so . This seems no longer help. Is this a bug/feature of freeze or psycopg2? Thanks for your interest and help. |
Did you want to update your patch for Python 3.4 and 3.5, Meador? |
Sure. I will refresh it tonight or sometime tomorrow. |
Same issue with external libraries under (pachted) Python 3.4.0 final on Ubuntu 14.04 LTS or Debian Wheezy/Sid. Meader, is there any option/possibility zu use an external library (like psycopg2.psycopg.so)from the freezed binary? I alreday tried to manually add various -L -l options to the linker command of the generated make file or adjust LD_LIBRARY_PATH and so on ... always without any success. Unfortunately, I need this for a large productive system Praktomat. Please help. Thanks again. Chris |
I also issued a ticket at the bugtracker of the psycopg2 project: |
Christian: please don't use this bug tracker to get help. Please use e.g. python-list to ask questions on how to use Python. To answer your question: in theory, you have the choice to either continue to use dynamic loading from the frozen interpreter, or to make psycopg.so a builtin module. Freeze won't help here, as it only deals with modules written in Python. |
Will these patches still make it into the Python 3.4 branch ? |
Martin: this is clearly a bug, as it is now (Python 3.3 onwards) impossible to use an external module (in a .so) from a frozen binary. Meanwhile, the psycopg guys are do (also) not see a bug on their side: |
On 25.03.2014 11:37, Christian Bachmaier wrote:
Are you sure about this ? If you freeze an application which imports just the _psycopg*.so file |
# ldd hello and (even strace) does not show an attempt to acces the so-file. Also tried to add -L/usr/lib/python3/dist-packages/psycopg2 -l_psycopg.cpython-34m-x86_64-linux-gnu.so and similar without success. Can anyone try to reproduce it, a freeze (with patches) of hello.py containing only import _psycopg2 or maybe another external library? In the first case apt-get install python3-psycopg2 is necessary. |
Apologies for not replying over the weekend. I am still looking into this one. |
On 25.03.2014 13:02, Christian Bachmaier wrote:
Christian, what you're expecting is not what freeze can offer. The shared library is not linked into the resulting binary See the eGenix PyRun source archive for how it's done:
The question I raised was whether running "hello" will If that's the case (and only then), you have found a bug that Thanks,Marc-Andre Lemburg |
The shared library is not linked into the resulting binary Yes. This is why (at least in Python 3.2) it must be in the right path (subdirectory), see above.
Unfortunately, Debian/Ubuntu does not deliver a static version of psycopg2. So I'd like to use the dynamic version. This is definitively possible with Python 3.2 x86, again, see above.
That's a good question. I think so, but how can I test that? At least the (only) way in Python 3.2 does not work any more. Even with Python 3.2 there must be a link in the subdirectory as shown above. It is not enough to have it only in the usual installation directory /usr/lib/python3/dist-packages/psycopg2/_psycopg.xxx.so
Right. So we should test the library feature which worked somehow magically in Python 3.2. Then we will see if it is a bug. My statement is that it is a bug. Thanks again to all. |
Sorry I forgot: PyRun seems only support Python 2.x. The only other freeze tool I know for Pyhton3 code is cx_freeze. I would prefere the vanilla freeze of the python distribution itself and as far as I can see using cx_freeze makes more problems for me as it may solve. But that is not a discussion for this thread or forum. |
On 25.03.2014 16:21, Christian Bachmaier wrote:
Right, because PyRun uses freeze and freeze currently does not work To test what I asked for, please run freeze on this script: """ If it prints 'Works.', then your problem is unrelated to this Thanks,Marc-Andre Lemburg |
$ xxx/freeze.py hello.py
$ make
$ ./hello
Traceback (most recent call last):
File "hello.py", line 3, in <module>
import _psycopg2
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_psycopg2' Btw the same with import psycopg2, psycopg2._psycopg, or _psycopg. Event with the subdir link as explained above. The first one (import psycopg2) is the one which operates in interpretation mode.
Nop, prints the error message instead of 'Works.'
=> we have a bug if Marc-Andre is right. Thanks. Chris |
On 25.03.2014 17:47, Christian Bachmaier wrote:
Sorry. The above should have read "import _psycopg". >> print ('Works.')
>> """
>
> $ xxx/freeze.py hello.py
> $ make
> $ ./hello
> Traceback (most recent call last):
> File "hello.py", line 3, in <module>
> import _psycopg2
> File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
> return _find_and_load_unlocked(name, import_)
> File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
> raise ImportError(_ERR_MSG.format(name), name=name)
> ImportError: No module named '_psycopg2'
>
> Btw the same with import psycopg2, psycopg2._psycopg, or _psycopg. Event with the subdir link as explained above. The first one (import psycopg2) is the one which operates in interpretation mode. Ok, now we're getting closer. Could you run this to have Python print the locations where export PYTHONVERBOSE=2 This should print a long list of messages such as these: import os # frozen
import errno # builtin
import posix # builtin
import posixpath # frozen
... to stderr. Of those only the lines which mention _psycopg are relevant. Please make sure that the dir where the .so file resides is If the dir is mentioned in the listing, we have to dig deeper Thanks,Marc-Andre Lemburg |
$ cat hello.py
import _psycopg
print('Works.')
$ export PYTHONVERBOSE=2
$ ./hello 2> res.txt
$ cat res.txt | grep psycopg
# trying /export/scratch/chris/pgtest/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /export/scratch/chris/pgtest/_psycopg.cpython-34m.so
# trying /export/scratch/chris/pgtest/_psycopg.abi3.so
# trying /export/scratch/chris/pgtest/_psycopg.so
# trying /export/scratch/chris/pgtest/_psycopg.py
# trying /export/scratch/chris/pgtest/_psycopg.pyc
# trying /usr/lib/python3.4/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3.4/_psycopg.cpython-34m.so
# trying /usr/lib/python3.4/_psycopg.abi3.so
# trying /usr/lib/python3.4/_psycopg.so
# trying /usr/lib/python3.4/_psycopg.py
# trying /usr/lib/python3.4/_psycopg.pyc
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.cpython-34m.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.abi3.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.so
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.py
# trying /usr/lib/python3.4/plat-x86_64-linux-gnu/_psycopg.pyc
# trying /usr/lib/python3.4/lib-dynload/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.cpython-34m.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.abi3.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.so
# trying /usr/lib/python3.4/lib-dynload/_psycopg.py
# trying /usr/lib/python3.4/lib-dynload/_psycopg.pyc
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.cpython-34m.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.abi3.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.so
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.py
# trying /usr/local/lib/python3.4/dist-packages/_psycopg.pyc
# trying /usr/lib/python3/dist-packages/_psycopg.cpython-34m-x86_64-linux-gnu.so
# trying /usr/lib/python3/dist-packages/_psycopg.cpython-34m.so
# trying /usr/lib/python3/dist-packages/_psycopg.abi3.so
# trying /usr/lib/python3/dist-packages/_psycopg.so
# trying /usr/lib/python3/dist-packages/_psycopg.py
# trying /usr/lib/python3/dist-packages/_psycopg.pyc
import _psycopg
ImportError: No module named '_psycopg'
----
$ cat hello.py
import _psycopg
print('Works.')
$ export PYTHONPATH=/usr/lib/python3/dist-packages/psycopg2:$PYTHONPATH
$ ./hello
Works. The correct import (see the doc of psycopg2) is import psycopg2. Otherwise a valid script does even not run in interpreted mode. The library lies in /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython...so Maybe there is the Problem. One has to Import the parent dir of the so-file, which Triggers the bug. $ cat hello.py
import psycopg2
print('Works.')
$ export PYTHONVERBOSE=2
$ ./hello 2> res.txt
$ cat res.txt | grep psycopg
import psycopg2
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 67, in <module>
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: No module named 'psycopg2._psycopg'
# destroy psycopg2 Marc-Andre: would you mind to install python3-psycopg2 to see that in real? |
OK, so trying to import around the package was definitely why the first instance didn't work so that's all expected. As for the failure when importing psycopg2, my guess is that the freezing of psycopg2.__init__ is not setting __path__ to anything reasonable to work with dynamically loading psycopg2._psycopg. That really shouldn't really ever work anyway since that just doesn't make sense from the perspective of freezing a package unless you made the extension module a built-in module, but I don't think submodules are supported in that case right now anyway. MAL, do you agree with that assessment? |
On 25.03.2014 19:27, Brett Cannon wrote:
Using C extensions embedded in Python packages is supported in This works because Python2 search for the module in the top level I'll have to have a look at how the pyscopg2 package normally try: or setup the package's .__path__ to include the top-level |
On 25.03.2014 19:41, M.-A. Lemburg wrote:
The package uses absolute imports for importing the C extension, e.g. from psycopg2._psycopg import __version__ This cannot work in Python with frozen packages. Not in Python 2 and Christian: Your only option is not to freeze the psycopg2 package In any case, freeze is not at fault here. |
Wait, Brett :-) The issue that Christian mentioned was just a side discussion. We still need to fix the main problem. |
Sorry guys, library loading of a freezed binary is different to interpreter mode. This is a bug in freeze, or at least an undocumented missing feature of freeze. This is no side discussion. And, in Python 3.2 this was working! As described above, just creating a subdir psycopg2 in the working direktory of the frozen binary containing a link _psycopg.so to the library. This is a fact. I have two working production systems using this mechanism under Ubuntu 12.04 LTS. Ok, psycopg2 seems to load the so file indirectly over __init__.py, however, the interpreter and older versions of freeze support that. The psycopg2 guys do not see any problem in that, see the closed entry linked above in their bug database. A workaround/hack I am using successfully now is to replace any 'import psycopg2._psycopg' by 'import _psycopg' in all /usr/lib/python3/dist-packages/psycopg2/*.py files and to set PYTHONPATH=//usr/lib/python3/dist-packages/psycopg2 prior execution of the frozen binary. I found this only by playing around several days, there is NO documentation about that. The advices statically linking etc. given here are too superficial. Any more helful/concrete advices (outside of this thread) seem to cost money. I wanted and still want help to fix this by giving detailed reports. |
Christian: Please understand that it was not helpful to post into this issue. The issue discussed here is separate from the issue you are having. We prefer a strict "one issue at a time" policy in this tracker. So when this issue gets closed because Marc-Andre's original problem is solved, feel free to report your issue again. |
Christian, please open a separate ticket for your problem. This ticket is about getting freeze, the tool itself, working, Thanks,Marc-Andre Lemburg |
New changeset 001a6dc996e7 by Martin v. Löwis in branch '3.4': New changeset 87ded2fdac4b by Martin v. Löwis in branch 'default': |
Thanks for the patch. It seems to work now. http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%20Freeze%203.x/builds/9 |
Apologies for not getting around to applying this myself. I had an extremely busy week with the day job last week. Thanks for applying Martin. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: