Some source code control tools, like Perforce, by default sync files readonly, and it's useful to 
leave them readonly so that you can mark files changed by making them writeable even if they're not 
checked out (e.g. working offline).

When python implicitly compiles a .py file that's readonly, it will create a .pyc file that's 
readonly. This would be fine, except that when the .py file changes, python *silently* refuses to 
overwrite a .pyc file that's readonly, even if the directory is completely writable by the user 
executing Python.

The attached script shows this behavior. Running "python -v" will produce the following line 
showing this:

  import b # from c:\Documents and Settings\psimanyi\tmp\b.py
  # can't create c:\Documents and Settings\psimanyi\tmp\b.pyc

And because I hoped this was a Windows-only problem, I tested on OSX (Leopard) and Ubuntu (9.10 
alpha): it occurs on all three platforms.

Thanks!

--------------------------------

I fixed showpycreadonly.sh after attaching it, so to be sure you have the current version, it 
follows:

rm -f [ab].py{,c}

echo 'import b' > a.py
echo 'print "b"' > b.py
ls -l b.py
python a.py
ls -l b.pyc
rm b.pyc
chmod -w b.py
python a.py
ls -l b.pyc
touch b.py
python -v a.py 2>&1 | grep b.py

It seems Python is using a default mask to compile modules.
If you open a file b.py as "r", and you import it from a.py, you can see
that b.pyc is:
-rwxr-xr-x 1 marco marco    9 24 mag 16:17 a.py
-r--r--r-- 1 marco marco   10 24 mag 16:17 b.py
-rw-r--r-- 1 marco marco  122 24 mag 16:46 b.pyc

If you try to run a.py:
-rwxr-xr-x 1 marco marco    9 24 mag 16:17 a.py
-r-xr-xr-x 1 marco marco   10 24 mag 16:17 b.py

you'll get:
-rw-r--r-- 1 marco marco  122 24 mag 16:49 b.pyc

as you can see, there is a default "mode" to create compiled modules...

Which version are you using?

I've seen the source code of import.c (2.5 and 2.6) and it seems that in
the 2.6 some bugfixes/features have been added.

However, I've just used the version 2.7 and it works fine (as 2.6, since
they have the same code (at least in import.c), until now).

bye

Oops, I was not accurate in reporting behavior: I now believe that this fails on Windows XP, but may work 
fine on Linux. Below are my results, showing reproducible failure on XP. The most important thing about 
this is: on XP, once the .pyc file is "broken", Python will not overwrite it, and it must be *manually* 
removed. This is bad!

I will append Ubuntu results in a few minutes.

To reproduce the problem, I had to modify the script to include a "sleep", see attached 
showpyreadonlysleep.sh (cygwin bash script).

Here's the results of running showpyreadonlysleep.sh, then manually reproducing the problem with 
"touch/python":

$ ./showpycreadonlysleep.sh
umask = 0022
CYGWIN_NT-5.1 psimanyi_xp 1.5.25(0.156/4/2) 2008-06-12 19:34 i686 Cygwin
Python 2.6.2
Running python.exe from: /cygdrive/c/Python26/python
-rw-r--r-- 1 psimanyi mkgroup-l-d 10 May 26 12:43 b.py
b
-rwx------+ 1 psimanyi mkgroup-l-d 137 May 26 12:43 b.pyc
b
-r-x------+ 1 psimanyi mkgroup-l-d 137 May 26 12:43 b.pyc
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

And now, running manually:


$ touch b.py

psimanyi@psimanyi_xp ~
$ python -v a.py 2>&1 | grep 'b\.py'
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

psimanyi@psimanyi_xp ~
$ touch b.py

psimanyi@psimanyi_xp ~
$ python -v a.py 2>&1 | grep 'b\.py'
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

Again, this shows that without removing the .pyc manually, the pyc will not be overwritten, and therefore 
program startup will be slow!

Re: Ubuntu: I can't reproduce the problem. At this point, I consider this problem Windows XP-only.

Re: Windows & bash: I decided to verify that this problem occurs even if running outside of Bash in 
the native (and wonderful) shell CMD.EXE. It *does* occur. Here's the state:

[1] Run my script in Bash, which fails as in my previous append from today.
[2] Start CMD.EXE. Then run a "touch b.py" in Bash, then verify using Bash "ls -l" that b.py is 
protection 444 (readonly).
[3] In CMD.EXE, run "python -v a.py" (I don't know how to use something like grep in CMD.EXE, so I 
just manually copy/pasted the "python -v" lines.) The following lines in CMD.EXE's output show that 
the problem still occurs in CMD.EXE:

# C:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from C:\Documents and Settings\psimanyi\b.py
# can't create C:\Documents and Settings\psimanyi\b.pyc
b
# clear __builtin__._


If you are unable to reproduce this on XP, I will be happy to help.

I would not be surprised if this had something to do with the granularity of mtime or some other 
oddity of Windows, because: if I delete the sleep in the Bash script, then rerun the Bash script very 
quickly using "control-P control-J" typed ahead, sometimes the Bash script fails and sometimes it 
succeeds! (Nice work, Windows.)

Thanks much.

I may have a reason to analyze that would explain why this fails on 
Windows: the non-Windows code path in import.c's open_exclusive() contians 
an unlink(filename), but the Windows path does not.

I'm going to try to build Python on Windows and add an unlink for both 
code paths.

More later.

I have a fix for this. The code is Windows-only. It has been verified to work for attached test script 
showpycreadonlysleep.sh. It simply adds a "chmod" call. The issue is that the unlink() call silently fails 
if the file is readonly, but unlink() succeeds if chmod() makes the file writable.

Out company would really appreciate having this fix integrated into 2.6 since we use the ActiveState 2.6 
builds on Windows. I haven't test this on non-Windows platforms but it should not change the behavior on 
non-Windows platforms if the #ifdef MS_WINDOWS is correct. The diff is below:

$ svn diff
Index: import.c
===================================================================
--- import.c    (revision 72946)
+++ import.c    (working copy)
@@ -840,6 +840,7 @@
 static FILE *
 open_exclusive(char *filename, mode_t mode)
 {
+
 #if defined(O_EXCL)&&defined(O_CREAT)&&defined(O_WRONLY)&&defined(O_TRUNC)
        /* Use O_EXCL to avoid a race condition when another process tries to
           write the same file.  When that happens, our open() call fails,
@@ -848,6 +849,9 @@
           writable, the file will never be written.  Oh well.
        */
        int fd;
+#ifdef MS_WINDOWS
+        (void) chmod(filename, 0600);
+#endif
        (void) unlink(filename);
        fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC
 #ifdef O_BINARY


It may be appropriate to document that the chmod() is only required on Windows, and therefore it is only 
called on Windows to avoid slowing down non-Windows platforms.

I realized that my patch, with a chmod(..., 0600), may not work under 
certain conditions where the original file is owned by a different user 
but the directory is group- or world-writable. A more likely correct fix 
would use chmod(..., 0666).

At any rate: someone should test the various combinations of whether this 
user or another user owns the original .pyc file.

TO pdsimanyi:

chmod it's not appropriate to change the permissions as you made: 0666
et voilà.

However, I don't understand if it can be a problem since you're using
Cygwin (and Bash under Windows) or not. I think this because Windows is
not POSIX compliant and under Windows the permissions are different than
the others under *NIX-like platforms (POSIX compliant).

Maybe in these days I can check the permissions on Windows.

Re: chmod 0666 inappropriate: I agree that leaving a file 0666 is inappropriate, however theoretically 
the code should never leave the file 0666 since the unlink will delete the file.

The code was intended to show what the problem is by demonstrating a solution. Note that the line 
immediately following the chmod 0666 is an unlink call, which will delete the file if the user had 
sufficient permission to chmod the file to 0666.

I suppose the meta-level description of the problem is:
[1] Without the chmod 0666, the unlink will fail if the file is 0400 or similar.
[2] With the chmod 0666, the unlink succeeds, and the file will be recreated with new permissions (not 
0666: I believe my tests show 0444) after the post-unlink file creation.
[3] Someone familiar with windows should analyze, propose a solution, test it under all relevant 
scenarios of file/group ownership (and again the chmod is only required on Windows), and on 
XP/Vista/Win7.

The only problem I can see with the chmod 0666 is if for some reason the file still exists with bad 
permissions after the unlink. Since this is a low-overhead operation compared to the time to regenerate 
the .pyc file, I think that it would be reasonable to have an #ifdef MS_WINDOWS, if the file exists 
after the unlink(), print a big warning message that will always be visible even if -v is not used when 
invoking Python.

This is getting complicated enough that you may want to see if you can get someone else to review it 
for ideas about how to address the chmod/unlink code. I am certainly not a Windows expert hence I'm 
making a best Unix-like attempt to analyze and test fixes.

Good luck and thanks.

I verified that launching the script from a CMD.EXE window instead of
from Cygwin also reproduces the "can't overwrite ZZZ.pyc". Here's a
sample of "python -v" showing this:

 # C:\depot\central\ntf\tools\bin\build1.pyc has bad mtime
 import build1 # from C:\depot\central\ntf\tools\bin\build1.py
 # can't create C:\depot\central\ntf\tools\bin\build1.pyc

If you would like some help, I can discuss this with Windows expert
programmers here and see whether they can suggest a solution you might
find acceptable. Thank you.

Applies also to Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32.

Apparently this may have become Windows-specific at some point. I can't reproduce under Linux with Python 2.6.5, 2.7 or 3.2.

I get a strange warning with -v under 2.7, though:

$ touch b.py && ~/cpython/27/python -v a.py 2>&1 | grep b.py
# /home/antoine/py3k/__svn__/b.pyc has bad mtime
import b # from /home/antoine/py3k/__svn__/b.py
# wrote /home/antoine/py3k/__svn__/b.pyc

("bad mtime"??)

So what's the current status of this on Windows Platforms?

This is still the case: on Windows, if foo.py is readonly,
    python -c "import foo"
generates foo.pyc with the readonly attribute.

Tested with 3.1 and current py3k (where the file is named __pycache__\foo.cpython-32.pyc)

Any idea if this will be fixed (at all) and/or back ported to 2.6.x or
2.7.x?

On Fri, Sep 3, 2010 at 3:36 PM, Amaury Forgeot d'Arc <report@bugs.python.org
> wrote:

>
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> This is still the case: on Windows, if foo.py is readonly,
>    python -c "import foo"
> generates foo.pyc with the readonly attribute.
>
> Tested with 3.1 and current py3k (where the file is named
> __pycache__\foo.cpython-32.pyc)
>
> ----------
> nosy: +amaury.forgeotdarc
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue6074>
> _______________________________________
>

Bug fixes go into 2.7 but not 2.6. See msg115664.