Some source code control tools, like Perforce, by default sync files readonly, and it's useful to 
leave them readonly so that you can mark files changed by making them writeable even if they're not 
checked out (e.g. working offline).

When python implicitly compiles a .py file that's readonly, it will create a .pyc file that's 
readonly. This would be fine, except that when the .py file changes, python *silently* refuses to 
overwrite a .pyc file that's readonly, even if the directory is completely writable by the user 
executing Python.

The attached script shows this behavior. Running "python -v" will produce the following line 
showing this:

  import b # from c:\Documents and Settings\psimanyi\tmp\b.py
  # can't create c:\Documents and Settings\psimanyi\tmp\b.pyc

And because I hoped this was a Windows-only problem, I tested on OSX (Leopard) and Ubuntu (9.10 
alpha): it occurs on all three platforms.

Thanks!

--------------------------------

I fixed showpycreadonly.sh after attaching it, so to be sure you have the current version, it 
follows:

rm -f [ab].py{,c}

echo 'import b' > a.py
echo 'print "b"' > b.py
ls -l b.py
python a.py
ls -l b.pyc
rm b.pyc
chmod -w b.py
python a.py
ls -l b.pyc
touch b.py
python -v a.py 2>&1 | grep b.py

It seems Python is using a default mask to compile modules.
If you open a file b.py as "r", and you import it from a.py, you can see
that b.pyc is:
-rwxr-xr-x 1 marco marco    9 24 mag 16:17 a.py
-r--r--r-- 1 marco marco   10 24 mag 16:17 b.py
-rw-r--r-- 1 marco marco  122 24 mag 16:46 b.pyc

If you try to run a.py:
-rwxr-xr-x 1 marco marco    9 24 mag 16:17 a.py
-r-xr-xr-x 1 marco marco   10 24 mag 16:17 b.py

you'll get:
-rw-r--r-- 1 marco marco  122 24 mag 16:49 b.pyc

as you can see, there is a default "mode" to create compiled modules...

Which version are you using?

I've seen the source code of import.c (2.5 and 2.6) and it seems that in
the 2.6 some bugfixes/features have been added.

However, I've just used the version 2.7 and it works fine (as 2.6, since
they have the same code (at least in import.c), until now).

bye

Oops, I was not accurate in reporting behavior: I now believe that this fails on Windows XP, but may work 
fine on Linux. Below are my results, showing reproducible failure on XP. The most important thing about 
this is: on XP, once the .pyc file is "broken", Python will not overwrite it, and it must be *manually* 
removed. This is bad!

I will append Ubuntu results in a few minutes.

To reproduce the problem, I had to modify the script to include a "sleep", see attached 
showpyreadonlysleep.sh (cygwin bash script).

Here's the results of running showpyreadonlysleep.sh, then manually reproducing the problem with 
"touch/python":

$ ./showpycreadonlysleep.sh
umask = 0022
CYGWIN_NT-5.1 psimanyi_xp 1.5.25(0.156/4/2) 2008-06-12 19:34 i686 Cygwin
Python 2.6.2
Running python.exe from: /cygdrive/c/Python26/python
-rw-r--r-- 1 psimanyi mkgroup-l-d 10 May 26 12:43 b.py
b
-rwx------+ 1 psimanyi mkgroup-l-d 137 May 26 12:43 b.pyc
b
-r-x------+ 1 psimanyi mkgroup-l-d 137 May 26 12:43 b.pyc
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

And now, running manually:


$ touch b.py

psimanyi@psimanyi_xp ~
$ python -v a.py 2>&1 | grep 'b\.py'
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

psimanyi@psimanyi_xp ~
$ touch b.py

psimanyi@psimanyi_xp ~
$ python -v a.py 2>&1 | grep 'b\.py'
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

Again, this shows that without removing the .pyc manually, the pyc will not be overwritten, and therefore 
program startup will be slow!

Re: Ubuntu: I can't reproduce the problem. At this point, I consider this problem Windows XP-only.

Re: Windows & bash: I decided to verify that this problem occurs even if running outside of Bash in 
the native (and wonderful) shell CMD.EXE. It *does* occur. Here's the state:

[1] Run my script in Bash, which fails as in my previous append from today.
[2] Start CMD.EXE. Then run a "touch b.py" in Bash, then verify using Bash "ls -l" that b.py is 
protection 444 (readonly).
[3] In CMD.EXE, run "python -v a.py" (I don't know how to use something like grep in CMD.EXE, so I 
just manually copy/pasted the "python -v" lines.) The following lines in CMD.EXE's output show that 
the problem still occurs in CMD.EXE:

# C:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from C:\Documents and Settings\psimanyi\b.py
# can't create C:\Documents and Settings\psimanyi\b.pyc
b
# clear __builtin__._


If you are unable to reproduce this on XP, I will be happy to help.

I would not be surprised if this had something to do with the granularity of mtime or some other 
oddity of Windows, because: if I delete the sleep in the Bash script, then rerun the Bash script very 
quickly using "control-P control-J" typed ahead, sometimes the Bash script fails and sometimes it 
succeeds! (Nice work, Windows.)

Thanks much.

I may have a reason to analyze that would explain why this fails on 
Windows: the non-Windows code path in import.c's open_exclusive() contians 
an unlink(filename), but the Windows path does not.

I'm going to try to build Python on Windows and add an unlink for both 
code paths.

More later.

I have a fix for this. The code is Windows-only. It has been verified to work for attached test script 
showpycreadonlysleep.sh. It simply adds a "chmod" call. The issue is that the unlink() call silently fails 
if the file is readonly, but unlink() succeeds if chmod() makes the file writable.

Out company would really appreciate having this fix integrated into 2.6 since we use the ActiveState 2.6 
builds on Windows. I haven't test this on non-Windows platforms but it should not change the behavior on 
non-Windows platforms if the #ifdef MS_WINDOWS is correct. The diff is below:

$ svn diff
Index: import.c
===================================================================
--- import.c    (revision 72946)
+++ import.c    (working copy)
@@ -840,6 +840,7 @@
 static FILE *
 open_exclusive(char *filename, mode_t mode)
 {
+
 #if defined(O_EXCL)&&defined(O_CREAT)&&defined(O_WRONLY)&&defined(O_TRUNC)
        /* Use O_EXCL to avoid a race condition when another process tries to
           write the same file.  When that happens, our open() call fails,
@@ -848,6 +849,9 @@
           writable, the file will never be written.  Oh well.
        */
        int fd;
+#ifdef MS_WINDOWS
+        (void) chmod(filename, 0600);
+#endif
        (void) unlink(filename);
        fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC
 #ifdef O_BINARY


It may be appropriate to document that the chmod() is only required on Windows, and therefore it is only 
called on Windows to avoid slowing down non-Windows platforms.

I realized that my patch, with a chmod(..., 0600), may not work under 
certain conditions where the original file is owned by a different user 
but the directory is group- or world-writable. A more likely correct fix 
would use chmod(..., 0666).

At any rate: someone should test the various combinations of whether this 
user or another user owns the original .pyc file.

TO pdsimanyi:

chmod it's not appropriate to change the permissions as you made: 0666
et voilà.

However, I don't understand if it can be a problem since you're using
Cygwin (and Bash under Windows) or not. I think this because Windows is
not POSIX compliant and under Windows the permissions are different than
the others under *NIX-like platforms (POSIX compliant).

Maybe in these days I can check the permissions on Windows.

Re: chmod 0666 inappropriate: I agree that leaving a file 0666 is inappropriate, however theoretically 
the code should never leave the file 0666 since the unlink will delete the file.

The code was intended to show what the problem is by demonstrating a solution. Note that the line 
immediately following the chmod 0666 is an unlink call, which will delete the file if the user had 
sufficient permission to chmod the file to 0666.

I suppose the meta-level description of the problem is:
[1] Without the chmod 0666, the unlink will fail if the file is 0400 or similar.
[2] With the chmod 0666, the unlink succeeds, and the file will be recreated with new permissions (not 
0666: I believe my tests show 0444) after the post-unlink file creation.
[3] Someone familiar with windows should analyze, propose a solution, test it under all relevant 
scenarios of file/group ownership (and again the chmod is only required on Windows), and on 
XP/Vista/Win7.

The only problem I can see with the chmod 0666 is if for some reason the file still exists with bad 
permissions after the unlink. Since this is a low-overhead operation compared to the time to regenerate 
the .pyc file, I think that it would be reasonable to have an #ifdef MS_WINDOWS, if the file exists 
after the unlink(), print a big warning message that will always be visible even if -v is not used when 
invoking Python.

This is getting complicated enough that you may want to see if you can get someone else to review it 
for ideas about how to address the chmod/unlink code. I am certainly not a Windows expert hence I'm 
making a best Unix-like attempt to analyze and test fixes.

Good luck and thanks.

I verified that launching the script from a CMD.EXE window instead of
from Cygwin also reproduces the "can't overwrite ZZZ.pyc". Here's a
sample of "python -v" showing this:

 # C:\depot\central\ntf\tools\bin\build1.pyc has bad mtime
 import build1 # from C:\depot\central\ntf\tools\bin\build1.py
 # can't create C:\depot\central\ntf\tools\bin\build1.pyc

If you would like some help, I can discuss this with Windows expert
programmers here and see whether they can suggest a solution you might
find acceptable. Thank you.

Applies also to Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32.

Apparently this may have become Windows-specific at some point. I can't reproduce under Linux with Python 2.6.5, 2.7 or 3.2.

I get a strange warning with -v under 2.7, though:

$ touch b.py && ~/cpython/27/python -v a.py 2>&1 | grep b.py
# /home/antoine/py3k/__svn__/b.pyc has bad mtime
import b # from /home/antoine/py3k/__svn__/b.py
# wrote /home/antoine/py3k/__svn__/b.pyc

("bad mtime"??)

So what's the current status of this on Windows Platforms?

This is still the case: on Windows, if foo.py is readonly,
    python -c "import foo"
generates foo.pyc with the readonly attribute.

Tested with 3.1 and current py3k (where the file is named __pycache__\foo.cpython-32.pyc)

Any idea if this will be fixed (at all) and/or back ported to 2.6.x or
2.7.x?

On Fri, Sep 3, 2010 at 3:36 PM, Amaury Forgeot d'Arc <report@bugs.python.org
> wrote:

>
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> This is still the case: on Windows, if foo.py is readonly,
>    python -c "import foo"
> generates foo.pyc with the readonly attribute.
>
> Tested with 3.1 and current py3k (where the file is named
> __pycache__\foo.cpython-32.pyc)
>
> ----------
> nosy: +amaury.forgeotdarc
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue6074>
> _______________________________________
>

Bug fixes go into 2.7 but not 2.6. See msg115664.

I propose a fix:

static FILE *
open_exclusive(char *filename, mode_t mode)
{
#if defined(O_EXCL)&&defined(O_CREAT)&&defined(O_WRONLY)&&defined(O_TRUNC)
    /* Use O_EXCL to avoid a race condition when another process tries to
       write the same file.  When that happens, our open() call fails,
       which is just fine (since it's only a cache).
       XXX If the file exists and is writable but the directory is not
       writable, the file will never be written.  Oh well.
    */
    int fd;
    (void) unlink(filename);
    fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC
#ifdef O_BINARY
                            |O_BINARY   /* necessary for Windows */
#endif
#ifdef __VMS
            , mode, "ctxt=bin", "shr=nil"
#elif defined(MS_WINDOWS)
			, mode | _S_IWRITE
#else
            , mode
#endif
          );

    if (fd < 0 )
        return NULL;
    return fdopen(fd, "wb");
#else
    /* Best we can do -- on Windows this can't happen anyway */
    return fopen(filename, "wb");
#endif
}

----------------------

so doesn't matter what the .py file permission is under windows, the .pyc file will always have both read and write permissions.

This is essentially the way things were done in version 2.5.4 of Python. The unlink() always succeeded, because the created .pyc file permission is always set to 0666 in the fd=open() function. This means the .pyc will never be created as read-only, and as long as they are never set to read-only manually by the user, everything will be okay.

You might say this will be a problem if someone accidentally set the it to read-only. Well we have been using Python 2.5 for many years, and we have managed quite well, so it is not really going to be a big issue, and can be fixed in the future. 

Whereas at the moment we have a PROBLEM!! With the current state of the Python 2.6, 2.7 and 3.2 interpreters, many Windows users cannot even think about upgrade from Python 2.5 because source control tools like Perforce will set all .py files to read-only, and so ALL created .pyc files will become read-only every time you run Python. I cannot stress how much pain this causes. PLEASE MAKE IT 2.5 EQUIVALENT ASAP!  THANK YOU!!

even just this will do the trick:
-------------------------------------

    fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC
#ifdef O_BINARY
                            |O_BINARY   /* necessary for Windows */
#endif
#ifdef __VMS
            , mode, "ctxt=bin", "shr=nil"
#elif defined(MS_WINDOWS)
	    , 0666
#else
            , mode
#endif
          );

-----------------------------------


And a side note, for the VC8 build for PC, the file random.c was left out from the visual studio project file for pythoncore project. It gives these errors when I try recompile:

Error	2	error LNK2019: unresolved external symbol __PyRandom_Init referenced in function _Py_Main	main.obj
Error	3	error LNK2001: unresolved external symbol __PyRandom_Init	pythonrun.obj
Error	4	error LNK2019: unresolved external symbol __PyOS_URandom referenced in function _posix_urandom	posixmodule.obj

This is my first time using bug tracker, so please point me to the appropriate place to put this. Thank you.

OK, I tracked down the original reason for the behavioural change: #2051

The important aspect in that patch is preserving the *read* permissions from the .py file so we don't accidentally expose data to other users.

Kevin's patch seems like a good approach for restoring compatibility in 2.7 and 3.2

In 3.3, it appears that #2051 has regressed, since no test was added to confirm the correct behaviour and importlib appears to behave more like 2.5.

(Note: Kevin is looking into this at the PyConAU sprints)

New test case needed (probably in Lib/test/test_import.py):

- create .py file as 0400
- import it
- check .pyc (or .pyo) file is 0600

That ensures the #2501 security bug remains fixed, while also addressing this problem.

Hi I made the following patch for this issue.

It addresses the #2051 security bug. So the .pyc and .pyo files are created using the same permission as .py

The MS_WINDOWS version will chmod the permission of .pyc and .pyo files to write accessible before deleting them.

I have also created a test for it. This test needs cleaning up, because it is the first time I have written a test for cpython. I would love to have a look at the final committed test so I can learn how to make it better next time. I will also be happy to keep extending that test to make sure it covers more cases.

A great big thanks to Nick for your help!

Adding 3.3 to affected versions, since I'm about to check in a fix for #2051 that will reintroduce this problem.

Here's a patch for 3.3 with a test.  Though, I'm still setting up my windows box for building Python, I wanted to get this patch up.  On its own the test should fail on Windows.  In that case, it should be backport-able.  If no one has a chance to verify the test on windows (or the fix) before I get everything set up, I'll do it.

Well, the workflow on Windows is certainly different.  I've run out of time to get this sorted out in the short term.  However, the patch should be pretty close to what's needed.  Obviously the posix-only bit doesn't fit for a Windows-specific test.

Hi Eric,

I am happy to install Python 3.3 on Windows and help you test this. Can you please give me some ideas on what I need to do?

Do I sync to the latest Python 3.3 branch and compile?  Then run some unit-test code? Which test files should I run? What should I be expecting?

Also, do you know if anyone is working on adding the fix and test to Python 2.7?

It's still on my list to follow up (along with the other contributions from the PyCon AU sprints). However, if Eric gets a Windows environment up and running before I get to it, I'm happy to hand the issue over (I rely on the buildbots to sanity check Windows fixes).

Interesting - the test case I wrote for this was failing on Linux as well. Same fix worked both places, though (i.e. ensuring S_IWUSR is always set on .pyc files so they can be updated properly)

2.7 update incoming, we'll see what the Windows buildbots have to say before doing anything to 3.x.

(Kevin: I didn't end up using your patch, as I decided it was simpler to just create the files with the user's write permission set. This doesn't weaken the security, as any attacker running with the user's privileges could just change the permissions anyway)

Ah, my test was buggy - the mtime wasn't reliably changing by enough, so the pyc wasn't always being rewritten. I changed it to force a big time delta by backdating the original source file.

New changeset 321414874b26 by Nick Coghlan in branch '2.7':
Issue #6074: Restore the long-broken support for running with read-only source files on Windows
http://hg.python.org/cpython/rev/321414874b26

Buildbots look happy, removing 2.7 from affected versions.

New changeset 1856d57abfc8 by Nick Coghlan in branch '3.2':
Issue #6074: Forward port Windows read-only source file fix from 2.7
http://hg.python.org/cpython/rev/1856d57abfc8

New changeset 3f6db10b7a69 by Nick Coghlan in branch '3.3':
Dummy merge from 3.2, as that approach to resolving #6074 is not applicable to importlib
http://hg.python.org/cpython/rev/3f6db10b7a69

OK, it should just be the two importlib based releases affected now.

New changeset df83d3bbd934 by Nick Coghlan in branch '2.7':
Issue #6074: Actually delete the source file in the test as intended
http://hg.python.org/cpython/rev/df83d3bbd934

New changeset bbb3459fbcb8 by Nick Coghlan in branch '3.3':
Issue #6074: Apply an appropriate fix for importlib based imports
http://hg.python.org/cpython/rev/bbb3459fbcb8

New changeset bedcc790c6e7 by Nick Coghlan in branch 'default':
Merge fix from #6074 from 3.3
http://hg.python.org/cpython/rev/bedcc790c6e7

New changeset 8cf7f6fe4282 by Nick Coghlan in branch '3.2':
Issue #6074: Actually delete the source file in the test as intended
http://hg.python.org/cpython/rev/8cf7f6fe4282

For 3.3+ I ended up using a tweaked version of Eric's patch - I kept the general test structure, but used the behavioural test I had created (i.e. does updating the initially read-only source file affect a subsequent bytecode only import?), as well as the simple rule of just always making the cached files writable by the owning user.

On 3.3+, this permissions tweak occurs regardless of OS, because I didn't think it was worth the runtime hit to make it OS specific (whereas the use of the C preprocessor previously made that not an issue).

Sounds good.  Thanks for getting this done, Nick.

Hi Nick,

Thank you so much for rolling out this change. I think your implementation of the fix is more elegant. I have learned a few things from you.

I would love to keep helping to test/fix bugs for Python on Windows. Please contact me if there is anything you would like me to take a look at. 

There will be many Windows users who will feel very grateful for this change you have implemented. This will actually give many the chance to upgrade to the latest 2.7 when it is released.

No worries. If you could do a fresh build of the 2.7 branch and confirm this fix works for you, that would be great.

Okay just tried it. This is under Windows 7 x64.

I created a .py file with read-only permission. When I import the .py module, the created .pyc doesn't have read-only permission. 

So it works as intended under Windows OS. As long as under posix system the created .pyc files have the same permissions as the .py file, then everything is good.