classification
Title: Document that importing .pyo files needs python -O
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Ronan.Lamy, berker.peksag, cjrh, docs@python, eric.araujo, eric.snow, lebigot, mherrmann.at, python-dev, r.david.murray, terry.reedy, vstinner
Priority: normal Keywords: patch

Created on 2011-09-14 20:32 by lebigot, last changed 2016-08-20 08:13 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
issue12982.diff Ronan.Lamy, 2012-06-14 16:53
Messages (24)
msg144049 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2011-09-14 20:32
When creating a .pyo file (either with -O or -OO) and removing any .pyc or .py original file, import <module_name> complains with "No module called <module_name>".  The import does work with .pyc files.

I'm not sure that this is the correct behavior, as the documentation does not seem to make any difference between .pyc and .pyo files on this point:

"It is possible to have a file called spam.pyc (or spam.pyo when -O is used) without a file spam.py for the same module. This can be used to distribute a library of Python code in a form that is moderately hard to reverse engineer."
msg144050 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-14 20:35
You need to run Python with -O command line option to ask Python to search for *.pyo files.
msg144067 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2011-09-15 08:39
Indeed.  Thanks.  I wish it had been in the documentation. :)  This is yet another reason for me to check how I can submit patches to the doc. :)

I also found out that renaming the .pyo file as .pyc makes Python happy upon import.  That's the solution I chose, because I don't want users to have to type "python -O" when calling programs compiled with -O or -OO.
msg144068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-15 09:11
> Indeed.  Thanks.  I wish it had been in the documentation. :)  This is yet another reason for me to check how I can submit patches to the doc. :)

Read http://docs.python.org/devguide/ to learn how to get the source of 
the documentation and how to write a patch.

> I also found out that renaming the .pyo file as .pyc makes Python happy upon import.  That's the solution I chose, because I don't want users to have to type "python -O" when calling programs compiled with -O or -OO.

Oh, I didn't know this trick, but yeah PYO and PYC are basically the 
same thing. The main difference is that PYO doesn't contain docstrings 
and doesn't contain "set the current line to ..." bytecode instructions.
msg144151 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-09-16 19:21
Eric, you can initially give a doc patch as text in a message, though an actual diff in nicer and may get action sooner. I agree that something could be added. Perhaps the quote "It is possible...." should be followed with something like "However, one must use the python -O option to search for and import .pyo files, as well as for creating them. An alternative is to rename them to .pyc." Or make whatever suggestion you think is better.

Also, please test with 3.2 if you possible can.
msg144198 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-17 15:45
I would not suggest to rename a .pyo to .pyc, they have different promises (if I can even say that, as they’re implementation details).
msg162667 - (view) Author: Michael Herrmann (mherrmann.at) Date: 2012-06-12 11:57
Hi,

I need to use a third-party library that ships as a mixture of .pyc and .pyo files. I found it a little surprising and inconvenient that I have to set the -O flag just to read .pyo files. I don't mind whether .pyc or .pyo files are being created during compilation, I just want to be able to read from both .pyc and .pyo files without having to use the -O flag. Can somebody fix this? I unfortunately have absolutely no clue how.

Thanks!
msg162670 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-06-12 13:15
Michael, I don’t think your proposed change would be considered favorably: importing .pyc or .pyo is well defined for CPython and the -O switch is really required for .pyo.  However you may be able to import them anyway without any change to Python if you write a custom importer (more info in PEP 302 and the import docs) which reuses the low-level imp module to find and load .pyo files.
msg162672 - (view) Author: Michael Herrmann (mherrmann.at) Date: 2012-06-12 13:25
Hi Eric,

thank you for your quick reply. I'm not the first one who encounters this problem and in my opinion it is simply counter-intuitive that you cannot read a mixture of .pyo and .pyc files. That is why I think that my proposed change is valuable. In the meantime, I will follow your suggestion and try to use the imp-module to load the .pyo-files myself. Thank you!

Best regards,
Michael
msg162673 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-12 13:26
Actually it's a lot easier than that, although it is very much a hack: just rename the .pyo files to .pyc, and python without -O will happily import them.  Since the optimization happens when the bytecode is written, this does what you want.
msg162674 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2012-06-12 13:28
Hi Michael,

Thank you for your message.

You are mentioning the suggestion of "the other Eric" (Araujo). My suggestion was to rename your .pyo files as .pyc files; it is hackish (according to a previous post from Eric Araujo), but might save you some trouble, on the short term, as .pyc do not need -O.

Best wishes,

EOL

On Jun 12, 2012, at 21:25, Michael Herrmann wrote:

> 
> Michael Herrmann <mherrmann.at@gmail.com> added the comment:
> 
> Hi Eric,
> 
> thank you for your quick reply. I'm not the first one who encounters this problem and in my opinion it is simply counter-intuitive that you cannot read a mixture of .pyo and .pyc files. That is why I think that my proposed change is valuable. In the meantime, I will follow your suggestion and try to use the imp-module to load the .pyo-files myself. Thank you!
> 
> Best regards,
> Michael
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12982>
> _______________________________________
msg162676 - (view) Author: Michael Herrmann (mherrmann.at) Date: 2012-06-12 13:39
Dear Eric OL,

I see - I had read your e-mail but because of the similar names I thought the message here was yours too, and thus only replied once. I apologize!

I can of course find a workaround such as renaming .pyo to .pyc. However, I would like to avoid having to modify the distribution of the third party library I am using in any way. The reason is that if a new version of this third party library is released and I have to upgrade to this new version, I will again have to manually do all the changes I had to do for the old version for the new version, and of course some changes won't work any more and there will be problems...

I'm finding it tedious to use the imp-module to read in the .pyo-files by hand and might just fall back to your suggestion. Thank you for it. Nevertheless, I am still hoping that this may be resolved in a future release.

Best,
Michael
msg162689 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-06-12 18:21
Michael, you should ask the closed source library distributor to distribute all files as .pyc so you have access to docstrings while programming and to avoid the problem with reading them. He could also distribute an all-.pyo version. A mixture seems accidental.

Is the import restriction currently intended? If a change *is* made, should it be regarded as an enhancement rather than a bug fix? I asked both question on pydev for more input.
msg162749 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-06-14 03:19
The response from pydev is that running .pyc with -O or running .pyo without is not officially supported. Even if mixing usually works now, it does not always work properly for code with __debug__, assert, or __doc__. The scope of possible malfunctions may increase if ideas for more aggressive optimization are implemented. So mixing the two types of cache files is 'do at your own risk' and 'don't complain if it does not work now or ceases to work in the future'.

A principle reason for the above is that __debug__ == <not  -O> and that it affects complilation, not just execution.

The doc should say the above more clearly. Eric L., where is the doc quote from?
msg162782 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2012-06-14 13:08
Terry, it seems that the doc I was quoting is for version 1.5.1 (http://docs.python.org/release/1.5.1p1/tut/node43.html). I can't find it in more recent versions of the doc. I should not have quoted an obsolete version of the documentation—I'm not sure how this happened. :)

I am not fully sure why -O is essentially required for running .pyo files: why not have the Python interpreter handle everything automatically based on the extension?
msg162783 - (view) Author: Michael Herrmann (mherrmann.at) Date: 2012-06-14 13:09
That is *exactly* my point :)
msg162784 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-14 13:20
Because:

  1) The __debug__ flag is defined to be process-global.  If you test it in one module, your code should be able to assume that it has the same value in all other modules

  2) python-dev does not support running .pyo code without -O turned on.  In the future it might be the case that -O would actually change the behavior of the running python interpreter such that .pyo code would fail of -O was not on.

So, you can do the rename or importer trick if you want, and it will work right now as long as your program does not depend on __debug__ being globally consistent (which would be the case for almost all programs), but we do not guarantee it will continue to work in future versions of Python.
msg162788 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-06-14 13:40
> I am not fully sure why -O is essentially required for running .pyo
> files: why not have the Python interpreter handle everything
> automatically based on the extension?

In part because it would take work to make it happen and apparently no one feels strongly enough about adding that functionality, or convinced enough that it's appropriate, to do the work.  If you're interested a good first step would be to write a PEP 302 metapath hook (finder that gets inserted to sys.metapath) that makes import of .pyo files work even when -O is not used.  Maybe you'd also want to have .pyc files work even when -O *is* used.

Keep in mind that there may be other reasons why such functionality would not go into the interpreter.  However, the beauty of import hooks is that it wouldn't matter once you have yours in hand.
msg162807 - (view) Author: Ronan Lamy (Ronan.Lamy) * Date: 2012-06-14 16:53
Doing it at the interpreter level is trivial (cf. patch), except for an annoying bug I noticed (see below). Doing it from user code might require some care to avoid disrupting existing import hooks, but AFAICT something like sys.path_hooks.append(FileFinder.path_hook(['.pyo', SourcelessFileLoader, True])) is supposed to work. 

The bug is that -O has currently no effect on sourceless imports: it seems that frozen_importlib actually uses freeze-time __debug__ instead of the current interpreter's.
msg162814 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-06-14 18:59
Eric, can you find a place in the current doc where -O and .pyo are mentioned, and where you think a sentence should go. What sentence(s) would you like to see.

Other comments:

__debug__ is intended to be a process-global compilation value (implemented as a keyword) set on startup

>>> __debug__
True
>>> __debug__ = False
SyntaxError: assignment to keyword

The devs are not willing to support having contradictory values in the same process. Indeed, since I posted last night, the pydev discussion has moved to the question of whether -O, __debug__, and .pyo as now defined are worth the nuisance they cause or whether some or all should be deprecated. (Docstring stripping for saving space could then be a separate tool.)

---
Python interpreters exist to run Python code. The existence, persistence, and other details of compilation caches are version-dependent implementation details. Being able to execute from such caches without source present is also an implementation detail, and for CPython, it gets secondary support at best. (This is a compromise between full support and no support.)
msg163139 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2012-06-19 07:04
Thank you for this lucid account of the situation, Terry.

As for where in the documentation something additional could be said about .pyo files and the -O option, I must say that it is already mentioned in some relevant places (http://docs.python.org/tutorial/modules.html#compiled-python-files),

However, I can see one other place where some additional information would be useful: in the documentation for the -O option itself (http://docs.python.org/using/cmdline.html#miscellaneous-options). The current documentation only mentions the .pyo-producing effect of -O. Mentioning there that -O is *required* for interpreting .pyo files would be useful.

Thanks!
msg273187 - (view) Author: Caleb Hattingh (cjrh) * Date: 2016-08-20 05:49
Presumably PEP488 (and the 4 years of inactivity) means that this issue could be closed?
msg273195 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-20 08:11
New changeset 8e9dc3e4ea91 by Berker Peksag in branch '3.5':
Issue #12982: Thanks to PEP 488, Python no longer creates .pyo files
https://hg.python.org/cpython/rev/8e9dc3e4ea91

New changeset 1455851e7332 by Berker Peksag in branch 'default':
Issue #12982: Merge from 3.5
https://hg.python.org/cpython/rev/1455851e7332
msg273196 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-08-20 08:13
Yes, this can be closed as 'out of date'. I've just updated -O documentation to remove the mention of .pyo extension. Thanks!
History
Date User Action Args
2016-08-20 08:13:39berker.peksagsetstatus: open -> closed

nosy: + berker.peksag
messages: + msg273196

resolution: out of date
stage: needs patch -> resolved
2016-08-20 08:11:42python-devsetnosy: + python-dev
messages: + msg273195
2016-08-20 05:49:05cjrhsetnosy: + cjrh
messages: + msg273187
2012-06-19 07:04:13lebigotsetmessages: + msg163139
2012-06-14 18:59:08terry.reedysetmessages: + msg162814
2012-06-14 16:53:27Ronan.Lamysetfiles: + issue12982.diff
keywords: + patch
messages: + msg162807
2012-06-14 13:40:36eric.snowsetnosy: + eric.snow
messages: + msg162788
2012-06-14 13:20:04r.david.murraysetmessages: + msg162784
2012-06-14 13:09:49mherrmann.atsetmessages: + msg162783
2012-06-14 13:08:02lebigotsetmessages: + msg162782
2012-06-14 03:19:31terry.reedysetmessages: + msg162749
components: + Documentation, - Interpreter Core
2012-06-12 19:05:10Ronan.Lamysetnosy: + Ronan.Lamy
2012-06-12 18:21:14terry.reedysetmessages: + msg162689
2012-06-12 13:39:31mherrmann.atsetmessages: + msg162676
2012-06-12 13:28:36lebigotsetmessages: + msg162674
2012-06-12 13:26:09r.david.murraysetnosy: + r.david.murray
messages: + msg162673
2012-06-12 13:25:49mherrmann.atsetmessages: + msg162672
2012-06-12 13:16:46eric.araujosettitle: .pyo file can't be imported unless -O is given -> Document that importing .pyo files needs python -O
2012-06-12 13:15:57eric.araujosetmessages: + msg162670
2012-06-12 11:57:36mherrmann.atsetnosy: + mherrmann.at
messages: + msg162667
components: + Interpreter Core, - Documentation
2011-09-17 15:45:10eric.araujosetnosy: + eric.araujo
title: .pyo file cannot be imported -> .pyo file can't be imported unless -O is given
messages: + msg144198

versions: + Python 3.2, Python 3.3
stage: needs patch
2011-09-16 19:21:19terry.reedysetnosy: + docs@python, terry.reedy
messages: + msg144151

assignee: docs@python
components: + Documentation, - Interpreter Core
type: compile error -> behavior
2011-09-15 09:11:15vstinnersetmessages: + msg144068
2011-09-15 08:39:45lebigotsetmessages: + msg144067
2011-09-14 20:35:14vstinnersetnosy: + vstinner
messages: + msg144050
2011-09-14 20:32:42lebigotcreate