Created on 2011-09-14 20:32 by lebigot, last changed 2012-06-19 07:04 by lebigot.
|issue12982.diff||Ronan.Lamy, 2012-06-14 16:53|
|msg144049 - (view)||Author: Eric O. LEBIGOT (lebigot)||Date: 2011-09-14 20:32|
When creating a .pyo file (either with -O or -OO) and removing any .pyc or .py original file, import <module_name> complains with "No module called <module_name>". The import does work with .pyc files. I'm not sure that this is the correct behavior, as the documentation does not seem to make any difference between .pyc and .pyo files on this point: "It is possible to have a file called spam.pyc (or spam.pyo when -O is used) without a file spam.py for the same module. This can be used to distribute a library of Python code in a form that is moderately hard to reverse engineer."
|msg144050 - (view)||Author: STINNER Victor (haypo) *||Date: 2011-09-14 20:35|
You need to run Python with -O command line option to ask Python to search for *.pyo files.
|msg144067 - (view)||Author: Eric O. LEBIGOT (lebigot)||Date: 2011-09-15 08:39|
Indeed. Thanks. I wish it had been in the documentation. :) This is yet another reason for me to check how I can submit patches to the doc. :) I also found out that renaming the .pyo file as .pyc makes Python happy upon import. That's the solution I chose, because I don't want users to have to type "python -O" when calling programs compiled with -O or -OO.
|msg144068 - (view)||Author: STINNER Victor (haypo) *||Date: 2011-09-15 09:11|
> Indeed. Thanks. I wish it had been in the documentation. :) This is yet another reason for me to check how I can submit patches to the doc. :) Read http://docs.python.org/devguide/ to learn how to get the source of the documentation and how to write a patch. > I also found out that renaming the .pyo file as .pyc makes Python happy upon import. That's the solution I chose, because I don't want users to have to type "python -O" when calling programs compiled with -O or -OO. Oh, I didn't know this trick, but yeah PYO and PYC are basically the same thing. The main difference is that PYO doesn't contain docstrings and doesn't contain "set the current line to ..." bytecode instructions.
|msg144151 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2011-09-16 19:21|
Eric, you can initially give a doc patch as text in a message, though an actual diff in nicer and may get action sooner. I agree that something could be added. Perhaps the quote "It is possible...." should be followed with something like "However, one must use the python -O option to search for and import .pyo files, as well as for creating them. An alternative is to rename them to .pyc." Or make whatever suggestion you think is better. Also, please test with 3.2 if you possible can.
|msg144198 - (view)||Author: Éric Araujo (eric.araujo) *||Date: 2011-09-17 15:45|
I would not suggest to rename a .pyo to .pyc, they have different promises (if I can even say that, as they’re implementation details).
|msg162667 - (view)||Author: Michael Herrmann (mherrmann.at)||Date: 2012-06-12 11:57|
Hi, I need to use a third-party library that ships as a mixture of .pyc and .pyo files. I found it a little surprising and inconvenient that I have to set the -O flag just to read .pyo files. I don't mind whether .pyc or .pyo files are being created during compilation, I just want to be able to read from both .pyc and .pyo files without having to use the -O flag. Can somebody fix this? I unfortunately have absolutely no clue how. Thanks!
|msg162670 - (view)||Author: Éric Araujo (eric.araujo) *||Date: 2012-06-12 13:15|
Michael, I don’t think your proposed change would be considered favorably: importing .pyc or .pyo is well defined for CPython and the -O switch is really required for .pyo. However you may be able to import them anyway without any change to Python if you write a custom importer (more info in PEP 302 and the import docs) which reuses the low-level imp module to find and load .pyo files.
|msg162672 - (view)||Author: Michael Herrmann (mherrmann.at)||Date: 2012-06-12 13:25|
Hi Eric, thank you for your quick reply. I'm not the first one who encounters this problem and in my opinion it is simply counter-intuitive that you cannot read a mixture of .pyo and .pyc files. That is why I think that my proposed change is valuable. In the meantime, I will follow your suggestion and try to use the imp-module to load the .pyo-files myself. Thank you! Best regards, Michael
|msg162673 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2012-06-12 13:26|
Actually it's a lot easier than that, although it is very much a hack: just rename the .pyo files to .pyc, and python without -O will happily import them. Since the optimization happens when the bytecode is written, this does what you want.
|msg162674 - (view)||Author: Eric O. LEBIGOT (lebigot)||Date: 2012-06-12 13:28|
Hi Michael, Thank you for your message. You are mentioning the suggestion of "the other Eric" (Araujo). My suggestion was to rename your .pyo files as .pyc files; it is hackish (according to a previous post from Eric Araujo), but might save you some trouble, on the short term, as .pyc do not need -O. Best wishes, EOL On Jun 12, 2012, at 21:25, Michael Herrmann wrote: > > Michael Herrmann <firstname.lastname@example.org> added the comment: > > Hi Eric, > > thank you for your quick reply. I'm not the first one who encounters this problem and in my opinion it is simply counter-intuitive that you cannot read a mixture of .pyo and .pyc files. That is why I think that my proposed change is valuable. In the meantime, I will follow your suggestion and try to use the imp-module to load the .pyo-files myself. Thank you! > > Best regards, > Michael > > ---------- > > _______________________________________ > Python tracker <email@example.com> > <http://bugs.python.org/issue12982> > _______________________________________
|msg162676 - (view)||Author: Michael Herrmann (mherrmann.at)||Date: 2012-06-12 13:39|
Dear Eric OL, I see - I had read your e-mail but because of the similar names I thought the message here was yours too, and thus only replied once. I apologize! I can of course find a workaround such as renaming .pyo to .pyc. However, I would like to avoid having to modify the distribution of the third party library I am using in any way. The reason is that if a new version of this third party library is released and I have to upgrade to this new version, I will again have to manually do all the changes I had to do for the old version for the new version, and of course some changes won't work any more and there will be problems... I'm finding it tedious to use the imp-module to read in the .pyo-files by hand and might just fall back to your suggestion. Thank you for it. Nevertheless, I am still hoping that this may be resolved in a future release. Best, Michael
|msg162689 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2012-06-12 18:21|
Michael, you should ask the closed source library distributor to distribute all files as .pyc so you have access to docstrings while programming and to avoid the problem with reading them. He could also distribute an all-.pyo version. A mixture seems accidental. Is the import restriction currently intended? If a change *is* made, should it be regarded as an enhancement rather than a bug fix? I asked both question on pydev for more input.
|msg162749 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2012-06-14 03:19|
The response from pydev is that running .pyc with -O or running .pyo without is not officially supported. Even if mixing usually works now, it does not always work properly for code with __debug__, assert, or __doc__. The scope of possible malfunctions may increase if ideas for more aggressive optimization are implemented. So mixing the two types of cache files is 'do at your own risk' and 'don't complain if it does not work now or ceases to work in the future'. A principle reason for the above is that __debug__ == <not -O> and that it affects complilation, not just execution. The doc should say the above more clearly. Eric L., where is the doc quote from?
|msg162782 - (view)||Author: Eric O. LEBIGOT (lebigot)||Date: 2012-06-14 13:08|
Terry, it seems that the doc I was quoting is for version 1.5.1 (http://docs.python.org/release/1.5.1p1/tut/node43.html). I can't find it in more recent versions of the doc. I should not have quoted an obsolete version of the documentation—I'm not sure how this happened. :) I am not fully sure why -O is essentially required for running .pyo files: why not have the Python interpreter handle everything automatically based on the extension?
|msg162783 - (view)||Author: Michael Herrmann (mherrmann.at)||Date: 2012-06-14 13:09|
That is *exactly* my point :)
|msg162784 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2012-06-14 13:20|
Because: 1) The __debug__ flag is defined to be process-global. If you test it in one module, your code should be able to assume that it has the same value in all other modules 2) python-dev does not support running .pyo code without -O turned on. In the future it might be the case that -O would actually change the behavior of the running python interpreter such that .pyo code would fail of -O was not on. So, you can do the rename or importer trick if you want, and it will work right now as long as your program does not depend on __debug__ being globally consistent (which would be the case for almost all programs), but we do not guarantee it will continue to work in future versions of Python.
|msg162788 - (view)||Author: Eric Snow (eric.snow) *||Date: 2012-06-14 13:40|
> I am not fully sure why -O is essentially required for running .pyo > files: why not have the Python interpreter handle everything > automatically based on the extension? In part because it would take work to make it happen and apparently no one feels strongly enough about adding that functionality, or convinced enough that it's appropriate, to do the work. If you're interested a good first step would be to write a PEP 302 metapath hook (finder that gets inserted to sys.metapath) that makes import of .pyo files work even when -O is not used. Maybe you'd also want to have .pyc files work even when -O *is* used. Keep in mind that there may be other reasons why such functionality would not go into the interpreter. However, the beauty of import hooks is that it wouldn't matter once you have yours in hand.
|msg162807 - (view)||Author: Ronan Lamy (Ronan.Lamy) *||Date: 2012-06-14 16:53|
Doing it at the interpreter level is trivial (cf. patch), except for an annoying bug I noticed (see below). Doing it from user code might require some care to avoid disrupting existing import hooks, but AFAICT something like sys.path_hooks.append(FileFinder.path_hook(['.pyo', SourcelessFileLoader, True])) is supposed to work. The bug is that -O has currently no effect on sourceless imports: it seems that frozen_importlib actually uses freeze-time __debug__ instead of the current interpreter's.
|msg162814 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2012-06-14 18:59|
Eric, can you find a place in the current doc where -O and .pyo are mentioned, and where you think a sentence should go. What sentence(s) would you like to see. Other comments: __debug__ is intended to be a process-global compilation value (implemented as a keyword) set on startup >>> __debug__ True >>> __debug__ = False SyntaxError: assignment to keyword The devs are not willing to support having contradictory values in the same process. Indeed, since I posted last night, the pydev discussion has moved to the question of whether -O, __debug__, and .pyo as now defined are worth the nuisance they cause or whether some or all should be deprecated. (Docstring stripping for saving space could then be a separate tool.) --- Python interpreters exist to run Python code. The existence, persistence, and other details of compilation caches are version-dependent implementation details. Being able to execute from such caches without source present is also an implementation detail, and for CPython, it gets secondary support at best. (This is a compromise between full support and no support.)
|msg163139 - (view)||Author: Eric O. LEBIGOT (lebigot)||Date: 2012-06-19 07:04|
Thank you for this lucid account of the situation, Terry. As for where in the documentation something additional could be said about .pyo files and the -O option, I must say that it is already mentioned in some relevant places (http://docs.python.org/tutorial/modules.html#compiled-python-files), However, I can see one other place where some additional information would be useful: in the documentation for the -O option itself (http://docs.python.org/using/cmdline.html#miscellaneous-options). The current documentation only mentions the .pyo-producing effect of -O. Mentioning there that -O is *required* for interpreting .pyo files would be useful. Thanks!
|2012-06-19 07:04:13||lebigot||set||messages: + msg163139|
|2012-06-14 18:59:08||terry.reedy||set||messages: + msg162814|
keywords: + patch
messages: + msg162807
messages: + msg162788
|2012-06-14 13:20:04||r.david.murray||set||messages: + msg162784|
|2012-06-14 13:09:49||mherrmann.at||set||messages: + msg162783|
|2012-06-14 13:08:02||lebigot||set||messages: + msg162782|
components: + Documentation, - Interpreter Core
|2012-06-12 18:21:14||terry.reedy||set||messages: + msg162689|
|2012-06-12 13:39:31||mherrmann.at||set||messages: + msg162676|
|2012-06-12 13:28:36||lebigot||set||messages: + msg162674|
messages: + msg162673
|2012-06-12 13:25:49||mherrmann.at||set||messages: + msg162672|
|2012-06-12 13:16:46||eric.araujo||set||title: .pyo file can't be imported unless -O is given -> Document that importing .pyo files needs python -O|
|2012-06-12 13:15:57||eric.araujo||set||messages: + msg162670|
messages: + msg162667
components: + Interpreter Core, - Documentation
title: .pyo file cannot be imported -> .pyo file can't be imported unless -O is given
messages: + msg144198
versions: + Python 3.2, Python 3.3
stage: needs patch
+ docs@python, terry.reedy|
messages: + msg144151
components: + Documentation, - Interpreter Core
type: compile error -> behavior
|2011-09-15 09:11:15||haypo||set||messages: + msg144068|
|2011-09-15 08:39:45||lebigot||set||messages: + msg144067|
messages: + msg144050