classification
Title: Bad .pyc files prevent import of otherwise valid .py files.
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.snow Nosy List: barry, brett.cannon, eric.snow, hulettbh, ncoghlan, r.david.murray, syeberman, terry.reedy
Priority: normal Keywords: patch

Created on 2016-09-07 19:49 by eric.snow, last changed 2021-02-19 19:07 by brett.cannon.

Files
File name Uploaded Description Edit
issue28007.diff eric.snow, 2016-09-12 20:50 review
Messages (12)
msg274872 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2016-09-07 19:49
As pointed out in #16384, a bad .pyc file will cause a module to fail importing.  Sometimes the .py file is okay, however.  The question posed in the other issue is if it would be desirable to fall back to the .py file.

FWIW, I don't think it's worth it.
msg274877 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-09-07 20:06
I agree, I don't think it is worth it.  If we did do it, there should be a warning, and things should still fail if the new .pyc file can't be written to replace the corrupt one.
msg274885 - (view) Author: Sye van der Veen (syeberman) * Date: 2016-09-07 20:50
Consider a process that is terminated while writing a large .pyc file.  Currently a user can only fix this by deleting the .pyc file, which requires knowing where to look.  A developer can be expected to know this, but the end user of their application certainly would not.

.pyc files are a cache.  When your cache is corrupted, you recreate it.
msg274942 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-08 01:24
I run into this kind of problem when switching back and forth between running things directly on my laptop and running them as a mounted volume in a container, since the permissions and SELinux labels on the cache can get messed up.

However, in those cases, writing back to the cache will also fail (since it's a "those are not your files, hands off" permissions problem).

So I'd be a fan of downgrading problems with the .pyc cache to warnings in general - if we read it and that fails, OK, we'll revert to using the source. If we try to write it, and that fails, well that's potentially OK too - writing to the cache is to speed up the *next* invocation of the module, and whether or not we actually care about that is going to be situational.
msg274943 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-09-08 01:28
That makes sense to me as a use case for warnings only.
msg275426 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-09-09 20:30
I think we should definitely fall back to using the .py file.  A cache is an implementation convenience, not a language feature.

A student using an all-users install typically does not have admin access to delete bad .pyc files.  In Windows, a user install is by default hidden in the hidden %APPDATA% directory (location somewhat Windows version dependent).  Even I did not know until recently how to access files there.
msg275476 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-09-09 22:32
On Sep 09, 2016, at 08:30 PM, Terry J. Reedy wrote:

>I think we should definitely fall back to using the .py file.  A cache is an
>implementation convenience, not a language feature.

+1.  If the cache is corrupt, it should be ignored and recreated.  There are
of course situations where that may not be possible though, e.g. if the .pyc
is on a read-only file system or the user doesn't have permission to write the
.pyc.  Still in those cases, if the .py is okay, but the .pyc is not, we
should do a source import.  I believe we already don't fall over if the .pyc
isn't writable, so let's keep that.  Although I don't think it's worth it, we
could add some extra knobs to warn about this case, or to be strict about not
allowing source imports.
msg276094 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2016-09-12 19:22
Here's a patch to soften the impact of pyc-related problems.

Note:
* Each pyc-related case that previously caused import to fail now results in an ImportWarning.  This applies to both reading and writing .pyc files.
* Each warning has __cause__ set to the caught exception.
* Some cases where the exception was already being swallowed now result in a warning while others are still swallowed.
* The tests verify that the warnings, with appropriate causes, are issued.
* The tests verify that existing bad .pyc file get overwritten.
* importlib._bootstrap_external.StaleBytecodeError is new.  It should surface only from SourcelessFileLoader.
* SourcelessFileLoader is not changed.
msg276119 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-12 21:03
I don't think we should try and squeeze this in for 3.6 since we have now hit beta and this is a feature since this is changing semantics that date back to Python 2. I'm going to go ahead and change the target to 3.7.
msg276124 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2016-09-12 21:34
I was going to ask Ned, but I'm fine with holding off until 3.7. :)
msg387342 - (view) Author: Brian Hulette (hulettbh) Date: 2021-02-19 18:58
Hey there, I just came across this bug when looking into a problem with corrupted pyc files. Was the patch ever applied? I'm still seeing the original behavior in Python 3.7.

Thanks!
msg387344 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-02-19 19:07
Since this is still open I would assume it never made it in.
History
Date User Action Args
2021-02-19 19:07:12brett.cannonsetmessages: + msg387344
2021-02-19 18:58:49hulettbhsetnosy: + hulettbh
messages: + msg387342
2016-09-12 22:20:00eric.snowlinkissue16384 superseder
2016-09-12 21:34:51eric.snowsetmessages: + msg276124
2016-09-12 21:03:19brett.cannonsetmessages: + msg276119
versions: + Python 3.7, - Python 3.6
2016-09-12 20:50:28eric.snowsetfiles: - issue28007.diff
2016-09-12 20:50:17eric.snowsetfiles: + issue28007.diff
2016-09-12 19:22:37eric.snowsetfiles: + issue28007.diff
keywords: + patch
messages: + msg276094
2016-09-10 00:20:57eric.snowsetassignee: eric.snow
2016-09-09 22:32:36barrysetmessages: + msg275476
2016-09-09 20:30:55terry.reedysetnosy: + terry.reedy
messages: + msg275426
2016-09-08 01:28:43r.david.murraysetmessages: + msg274943
2016-09-08 01:24:47ncoghlansetmessages: + msg274942
2016-09-07 20:50:40syebermansetmessages: + msg274885
2016-09-07 20:06:55r.david.murraysetnosy: + r.david.murray
messages: + msg274877
2016-09-07 19:49:38eric.snowcreate