classification
Title: Standard open() does not allow to specify file permissions.
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, christian.heimes, giampaolo.rodola, serhiy.storchaka, socketpair, vstinner
Priority: normal Keywords:

Created on 2017-01-09 14:17 by socketpair, last changed 2017-07-27 01:32 by giampaolo.rodola. This issue is now closed.

Messages (18)
msg285044 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-09 14:17
1. Syscall open() allows that.
2. This is important to prevent race-conditions between creating a file with default permissions and calling fchmod().
msg285046 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-09 14:39
You can actually specify permission:

fd = os.open("document.txt", os.O_WRONLY | os.O_CREAT, 0o777)
fp = open(fd, "wb")
with fp:
   ...

You can also use the dir_fd parameter for even better security ;-)

Is your request for the standard open() function, rather than os.open()?

Note: io.FileIO(), the underlying object when calling the Python open(), calls the C open() function with 0o666 for the permission mode.
msg285047 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-01-09 14:46
I would be nice to have a public API to set permissions of files for io.open(), too.

Note to your note: the actual permission is influenced by the process' global umask. If you are concerned about security, you should set the umask to 0o077 early.
msg285054 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-09 17:01
1. Yes, I'm reporting problem about standard open(), not os.open().
2. Yes, I know about umask. But unfortunatelly, umask affects whole process, including all threads, so it is not possible to force specific permissions without of race conditions with other threds.
3. Calling C's `open()` with `0666` MUST be documented (with mentioning of umask, which also affects these bits). This is `MUST` because `man 2 open` says that there is no default for `mode` argument, and it must be specified when O_CREAT / O_TMPFILE is specified.
msg285061 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-09 17:55
io.open() is high-level function. It handles buffering, encoding, newline translating. It is even higher-level than C's fopen().

Syscall open() is low-level. Python os.open() is an interface to this low-level feature.

There is a connection between low and high level -- io.open() accepts a file descriptor returned by os.open(). You also can provide the opener argument.

I don't think io.open() needs the support of mode and dir_fd arguments and all possible O_* flags. They are low-level features.
msg285063 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-09 19:29
Permissions -- are very important thing. As I think, such high-level functions should not hide that important functionality from end-users.

Also, it is not difficult to make a patch (as I think).
msg285068 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-01-09 20:11
I agree with Serhiy that I don't think this is necessary for io.open() when os.open() supports this. But this really can't be discussed further until someone provides a clear design proposal on how to make this work with io.open().
msg285069 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-09 20:22
I expect this:

open(...., perms=0o644)

Why not?
msg285070 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-09 20:23
so, io.open() should just pass that value down to implementation. implementation should use this value instead of hard-coded 0o666.
msg285073 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-09 20:53
open(..., opener=partial(os.open, mode=0o644))

Not every combination of functions deserves a new parameter of a builtin.
msg285077 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-09 22:27
Such construction is not so easy. Especially for beginners. Not everyone even uses context managers to work with files. They will try to use os.chmod(). More clever will use os.fchmod(fileobj.fileno()). And in rare case someone asks about race condition between threads. So, not implementing that is error prone, and also forces users to make hidden bugs.

Why not to follow the principle of Least Surprise ?

In you way, we did not have to implement a lot of things, that we have now. Like tempfile, shutil and so on. They may be implemented using complex combination of simplier items. Answer: these things make Python easy, secure and (insert good words here by yourself).

Why are you afraid to add one very simple thing ?

Next, what if I want to use, say, 'r+b', in that case I should remember combination of O_XXXXX constants like O_TRUNC|O_CREAT|O_APPEND and so on.. It's great that Python allows such things! really great! but how many people want to write that in their code?

(Sorry for long text, but I really can not understand)
msg285144 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-01-10 19:01
The point Serhiy is trying to make is that not everyone needs or cares about setting specific file permissions. Python's built-in open() function has not supported this for 26 years and so there's obviously not that much of a need if this has not consistently come up as a shortcoming.

Two, you say to "just pass" something like "0o644" as the mode. OK, but what does that even mean? As a Linux user you may know thanks to chmod usage, but a Windows user or someone who doesn't use a Linux shell won't have any idea what that octal constant represents. So how do you propose to help people set the proper mode? If you say "use os.O_CREAT" then you just made open() have functionality depend on the os module which no other built-in directly does for their API.

In other words you might call this simple, but I call it an advanced feature that's supported by os.fdopen(os.open()) already. So when I say a "clear design" I mean addressing the fact that it's non-obvious for beginners on how to use and something not everyone needs.
msg285155 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-10 21:11
I don't understand the problem with umask(). It's standard and affect all
code even C extension calling directly or indirectly open(). It is more
secure to use umask() than setting mode on a few Python open() calls no?

For example, Python creates .pyc files. You cannot (easily) modify the
open() call to set mode, whereas it is affected by umask().

If you call umask() at startup, it affects all threads, there is no race
condition like open()+fchmod().
msg285158 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-01-10 21:22
Victor, you are correct. That was exactly my point.

The most secure way is to tighten security and set umask to 0o077. It's basically a white list or locked down approach. With umask 0o077 all subsequent files, directories and other resources will be created without any permission for group and others. This is even true for operations that create a Unix socket.

You have to change the permission of files to a more permissive mode explicitly. Any mistake is easy to spot (access denied) and not catastrophic.

By the way fchmod() isn't necessarily the optimal way to change permission by file descriptor. The behavior of fchmod() isn't well defined for socket files. On Linux fchmod() of a Unix socket file does not alter the permission bits of the socket device file.
msg285197 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-01-11 07:37
@haypo, suppose, one thread wants file with permissions A, and other thread wants permissions B. If they will "set" them through umask(), race condition may occur.
msg285198 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-11 07:46
If you needs non-default permission for particular file, you can use one of available ways mentioned above.

I think this issue can be closed. The original issues are resolved:

1. os.open() (corresponding to syscall open()) allows this.
2. There are other ways to create Python file with specific permissions (pass fd or use opener).
msg285199 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-11 07:55
> @haypo, suppose, one thread wants file with permissions A, and other thread wants permissions B. If they will "set" them through umask(), race condition may occur.

Ok, let's say thta you want file "x" with permission A=0o777 and file "y" with permission B=0b700.

You start with os.umask(0o077), as suggested by Christian.

>>> import os
>>> os.umask(0o077)
2
>>> x=open("x", "w")
>>> y=open("y", "w")
>>> os.system("ls -l x y")
-rw-------. 1 haypo haypo 0 11 janv. 08:48 x
-rw-------. 1 haypo haypo 0 11 janv. 08:48 y
0
>>> os.fchmod(x.fileno(), 0o777)
>>> os.system("ls -l x y")
-rwxrwxrwx. 1 haypo haypo 0 11 janv. 08:48 x
-rw-------. 1 haypo haypo 0 11 janv. 08:48 y
0

This is a time window where the file "x" has the permission -rw------- instead of the expected -rwxrwxrwx. Ok, it's expected. But it's not an easy since initial permissions are are more strict that expected permisions.

The race condition only occurs if you work the other way, start with os.umask(0777) (default umask) and then change permissions of the file "x" to stricter. In this specific case, you *already* have two ways to do that in Python 3 (no change needed):

  fp = open("x", opener=partial(os.open, mode=0o700))
or:
  fd = os.open("x", os.O_WRONLY | os.O_CREAT, 0o700)
  fp = open(fd, "wb")

--

About documenting the default mode: the problem is that I'm not sure that *all* functions in Python, especially when using third party libraries, use the same default mode. Morever, the mode on Windows has a very different meaning. Permission per group is handled by ACLs, not with the integer "mode":
https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx

On Windows, pmode of _open() only contains read and write permissions of the file, for everyone.

For all these reasons, I close the issue.
msg285209 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-01-11 10:29
I like the partial opener trick. Can we have the examples in the public documentation of open(), too?
History
Date User Action Args
2017-07-27 01:32:42giampaolo.rodolasetnosy: + giampaolo.rodola
2017-01-11 10:29:04christian.heimessetmessages: + msg285209
2017-01-11 07:55:12vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg285199
2017-01-11 07:46:21serhiy.storchakasetmessages: + msg285198
2017-01-11 07:37:30socketpairsetmessages: + msg285197
2017-01-10 21:22:29christian.heimessetmessages: + msg285158
2017-01-10 21:11:59vstinnersetmessages: + msg285155
2017-01-10 19:01:12brett.cannonsetmessages: + msg285144
2017-01-09 22:27:18socketpairsetmessages: + msg285077
2017-01-09 20:53:27serhiy.storchakasetmessages: + msg285073
2017-01-09 20:23:45socketpairsetmessages: + msg285070
2017-01-09 20:22:38socketpairsetmessages: + msg285069
2017-01-09 20:11:36brett.cannonsetversions: - Python 3.5, Python 3.6
nosy: + brett.cannon

messages: + msg285068

type: behavior -> enhancement
2017-01-09 19:29:20socketpairsetmessages: + msg285063
2017-01-09 17:55:37serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg285061
2017-01-09 17:01:34socketpairsetmessages: + msg285054
2017-01-09 14:46:11christian.heimessetmessages: + msg285047
2017-01-09 14:39:20vstinnersetnosy: + vstinner
messages: + msg285046
2017-01-09 14:30:57christian.heimessetnosy: + christian.heimes
2017-01-09 14:17:44socketpairsetversions: + Python 3.7
2017-01-09 14:17:34socketpairsettype: behavior
2017-01-09 14:17:01socketpaircreate