classification
Title: Add create mode to open()
Type: enhancement Stage: resolved
Components: IO, Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, David.Townshend, Devin Jeanpierre, Julian, amaury.forgeotdarc, benjamin.peterson, cvrebert, docs@python, eric.araujo, haypo, ncoghlan, neologix, petri.lehtinen, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2011-08-16 13:49 by David.Townshend, last changed 2012-05-20 09:43 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
open_create.patch David.Townshend, 2011-08-16 13:49 Patch implementing 'c' mode for open()
open_create_x-2.patch David.Townshend, 2011-08-18 16:06 Implement 'x' mode for open() (doc typo corrected) review
open_create_x-3.patch neologix, 2012-01-08 17:25 review
x_flag.diff neologix, 2012-01-10 19:37 review
x_diff2.patch David.Townshend, 2012-01-12 06:04 review
Messages (44)
msg142193 - (view) Author: David Townshend (David.Townshend) Date: 2011-08-16 13:49
Currently, opening a file with open(file, 'w') overwrites existing files.  It would be useful for open() to raise an error when the file exists.  This proposal is to add a 'c' mode to open, which has the effect to creating a file and opening it for writing, but raising an IOError if it already exists (i.e. the same as os.open(file, os.O_EXCL|os.O_CREAT).

The attached patch implements this, including tests and documentation.
msg142194 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-08-16 13:57
I'm not sure that O_EXCL is portable (exist on all platforms) because Python source code uses "#ifdef O_EXCL".
msg142198 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2011-08-16 14:53
I think this should be brought up on python-ideas or python-dev.
msg142200 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-16 15:37
See also issue 12105.
A couple downsides:
- O_EXCL is not necessarily portable (doesn't work well with NFS, maybe not on Windows?)
- O_EXCL is useful, as is O_CLOEXEC: to be consistent, we should also end up adding all other commonly-used flags, which are really OS-specific

Furthermore, you can achieve the same thing with:
os.fdopen(os.open('/etc/fstab', os.O_WRONLY|os.O_CLOEXEC|os.O_CREAT))
it's more verbose, though.
msg142201 - (view) Author: David Townshend (David.Townshend) Date: 2011-08-16 15:43
It was discussed on python-ideas, but the subject of the thread was actually on shutils.move so it was not really discussed much. I will repost this idea separately.
msg142308 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-18 11:33
The "#ifdef O_EXCL" in the source code is probably very old. Copying a message I posted on python-ideas:

O_EXCL is a POSIX standard. It is also supported
under Windows by the _open/_wopen compatibility functions (which we use
for file I/O).

Probably there are very old systems which don't support it, and perhaps
new systems that don't implement it *correctly* (meaning not
atomically); for the former I'd say we just don't care (who's gonna run
Python 3 on a 1995 system?) and for the latter, well, if the OS
designers think it's fine, let's just expose it as it is.

As for NFS, there's an interesting comment from 2007 here:
http://lwn.net/Articles/251971/

“My NFS tester shows that it at least appears to work with Linux,
Solaris and FreeBSD:
http://www.dovecot.org/list/dovecot/2007-July/024102.html. Looking at
Linux 2.6 sources it doesn't look like it tries to implement a racy
O_EXCL check in client side (fs/nfs/nfs3proc.c nfs3_proc_create()), so
the test's results should be correct. I don't know if other OSes do
that. I guess it would be nice to have a better O_EXCL tester which
tries to catch race conditions.”
msg142309 - (view) Author: David Townshend (David.Townshend) Date: 2011-08-18 11:56
My aim isn't to add all the commonly used flags, that would be pointless since its already possible using os.open.  The aim is to add a missing feature to the builtin open(), i.e. file creation.  At the moment open() implements read, write, and append, and creation is only implied by write. But in many cases, an explicit creation is desired (i.e, specifically create a new file, with an exception on failure).  It is true that this is possible with os.open, but it is somewhat obscure, especially for beginners, and is not as easy to read as "open(file, 'c')"
msg142314 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-18 13:24
> The "#ifdef O_EXCL" in the source code is probably very old. Copying a message I posted on python-ideas:
>
> O_EXCL is a POSIX standard. It is also supported
> under Windows by the _open/_wopen compatibility functions (which we use
> for file I/O).
>

If it's supported by Windows then I'm OK (not that I personaly care
about Windows :-).

> and is not as easy to read as "open(file, 'c')"

Well, I'd rather have this flag called 'x', to be consistent with
glibc's fopen():

"""
       c (since glibc 2.3.3)
              Do not make the open operation, or subsequent read and write
              operations, thread cancellation points.

       x      Open the file exclusively (like the O_EXCL flag of
open(2)).  If the
              file already exists, fopen() fails, and sets errno to
EEXIST.  This
              flag is ignored for fdopen().
"""

By the way, could you submit your patch as a mercurial diff ("hg diff")?
It makes it easier to review under Rietveld.
msg142315 - (view) Author: David Townshend (David.Townshend) Date: 2011-08-18 13:28
Changing form 'c' to 'x' is easy enough, and if there is already a convention it makes sense to stick to it.

I thought I had done a mercurial diff! I'll try again and resubmit.
msg142320 - (view) Author: Julian Berman (Julian) * Date: 2011-08-18 14:15
A minor documentation error in io.rst line 475 which was changed to:

+   The *mode* can be ``'c'``, ``'r'``, ``'w'`` or ``'a'`` for reading
+   (default), writing, or appending.

and should have "creating" at the front I assume, like you have it later.
msg142321 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-18 14:19
> Well, I'd rather have this flag called 'x', to be consistent with
> glibc's fopen():
> 
> """
>        c (since glibc 2.3.3)
>               Do not make the open operation, or subsequent read and write
>               operations, thread cancellation points.
> 
>        x      Open the file exclusively (like the O_EXCL flag of
> open(2)).  If the
>               file already exists, fopen() fails, and sets errno to
> EEXIST.  This
>               flag is ignored for fdopen().

Yeah, but I think "exclusively" is quite misleading since it does not
perform any locking of any kind. Also, I don't think we'll ever
integrate the glibc's "c" option in io.open().
msg142333 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-18 15:14
> Yeah, but I think "exclusively" is quite misleading since it does not
> perform any locking of any kind.

It might be misleading, but I find it clear enough, and this name has been endorsed by POSIX.

Furthermore, there's an added bonus: actually, with the old I/O layer, one can already pass an 'x' flag to open, since it just calls fopen:
"""
cf@neobox:~$ strace -e open python -c "open('/tmp/foo', 'wx')"
[...]
open("/tmp/foo", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0666) = 3
cf@neobox:~$ strace -e open python -c "open('/tmp/foo', 'wx')"
[...]
open("/usr/lib/pymodules/python2.6/<string>", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
IOError: [Errno 17] File exists: '/tmp/foo'
"""

I don't know if it's documented behavior, but the OP in issue 12105 was using it with python 2.
Changing it to 'x' would make such code backward-compatible.

Finally, when I read open('/tmp/foo', 'wx'), it's immediately clear to me what's going on, while I'd have to look at open()'s documentation to find out what the 'c' flag does.
msg142335 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-08-18 15:16
> I don't know if it's documented behavior

See #12103: it is not documented.
msg142352 - (view) Author: David Townshend (David.Townshend) Date: 2011-08-18 16:06
I hope this patch suits you better :-)

I've updated the documentation typo (thanks for pointing that out). I've also changed 'c' to 'x', since I think that if there is a convention we should stick to it.  I don't think that it matters if the glibc docs say 'exclusive', as long it its clear in the python docs what it does, which I think it is. Having said that, I don't really have a strong opinion either way, so I'll happily change it back to 'c' if its preferred.
msg142902 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-24 19:18
So, what was the conclusion of the discussion brought up on python-dev?
I had a feeling some core devs were opposed to this (I'm personally -0).
msg142910 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-24 20:12
I haven't seen any discussion on python-dev. Have I missed something?
msg142912 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-24 20:17
> I haven't seen any discussion on python-dev. Have I missed something?

It was on Python-ideas actually:
http://mail.python.org/pipermail/python-ideas/2011-August/011183.html
msg142916 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-24 20:35
Ah, right. Well I think all arguments against it were quite weak (or even wrong). Let's see:

- it's not cross-platform: actually it is (OS_EXCL has been POSIX since at least 1997 (*), and Windows also has it)

- os.open followed by os.fdopen is easy: it isn't that easy to get the incantation right (the pure Python open() in _pyio is 70 lines of code), especially if you want the file object to have the right "name" attribute

- it doesn't fill a use case: actually, avoiding race conditions is an important use case, even though many people may never encounter it (I must admit I myself never really cared about this)

So this looks like a reasonable feature request IMHO.

(*) http://pubs.opengroup.org/onlinepubs/007908799/xsh/open.html
msg142923 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-08-24 21:19
> - os.open followed by os.fdopen is easy: it isn't that easy to get
> the incantation right (the pure Python open() in _pyio is 70 lines
> of code), especially if you want the file object to have the right
> "name" attribute

What if we can override the inner call to os.open()?
For example, io.open() could grow an additional argument "fd_opener" which defaults to the equivalent of os.open.

Then creation mode can be expressed like this:
    open(filename, 'w',
         fd_opener=lambda path, mode: os.open(path, mode|os.O_CREAT)
Another use case with openat (see #12797):
    open(filename,
         fd_opener=lambda path, mode: os.openat(fd, path, mode)
msg142928 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-24 21:27
> What if we can override the inner call to os.open()?
> For example, io.open() could grow an additional argument "fd_opener"
> which defaults to the equivalent of os.open.

I agree it would be a much better situation than what we have now.
msg142930 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-08-24 21:34
>    open(filename, 'w',
>         fd_opener=lambda path, mode: os.open(path, mode|os.O_CREAT)

I prefer open(name, "c").

> it doesn't fill a use case: actually, avoiding race conditions
> is an important use case, ...

It may help the issue #8604 (and maybe #8828).
msg146684 - (view) Author: David Townshend (David.Townshend) Date: 2011-10-31 10:22
I see this has been marked as a duplicate of http://bugs.python.org/issue12797.  Please explain how this is, since that proposal does not appear to provide the functionality discussed here.
msg146686 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-31 10:32
issue12797 would allow things like:

def create_exclusive_file(filename):
  return open(filename, "w",
              opener=lambda path, mode: os.open(path, mode|os.O_CREAT|os.O_EXCL))
msg147205 - (view) Author: David Townshend (David.Townshend) Date: 2011-11-07 06:53
It is already possible to write a wrapper function that does it:

def create(file):
    fd = os.open(file, os.O_EXCL | os.O_CREAT | os.O_WRONLY)
    return os.fdopen(fd)

The point it not that it can't be done, but that it is not straight forward.  The docs say this about os.open(): "This function is intended for low-level I/O. For normal usage, use the built-in function open()" 

I wouldn't call creating a new file low-level I/O, but it is normal usage.
msg147841 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-18 09:55
See #13424 for a doc request about this.
msg150262 - (view) Author: Devin Jeanpierre (Devin Jeanpierre) Date: 2011-12-26 15:42
C11 uses 'x' for this, for what it's worth.

This is not a "duplicate issue". The openat solution is no easier than the os.open solution.
msg150273 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-27 17:11
> C11 uses 'x' for this, for what it's worth.
> 
> This is not a "duplicate issue". The openat solution is no easier than 
> the os.open solution.

Ok, let's re-open then. I'm not sold on the feature, but the fact C11 adds a dedicated letter mode for it could be a good excuse for us to mimick it :)
msg150391 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-12-30 20:59
> This is not a "duplicate issue". The openat solution is no easier than the os.open
> solution.

Amaury did not suggest to use openat, but the new opener argument to open, which was especially added for use cases such as the one discussed here:

    ...
    open_exclusive = lambda path, mode: os.open(path, mode|os.O_CREAT|os.O_EXCL))
    ...
    fp = open(filename, 'w', opener=open_exclusive)
    ...

That’s why this bug was initially closed as duplicate.
msg150402 - (view) Author: Devin Jeanpierre (Devin Jeanpierre) Date: 2011-12-30 22:07
> Amaury did not suggest to use openat, but the new opener argument to open, which was especially added for use cases such as the one discussed here:

Sorry, yes. Wrong words, same thought. We can implement this using opener, but we could implement this with os.open before. What's changed, except that there's more ways to do it? (There is slightly more versatility with the opener method, but no more obviousness and no less typing).

My understanding from reading the other thread is that this is not the primary use-case of the new parameter for open(). In fact, this ticket was not really mentioned at all there.
msg150406 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-12-31 00:37
> [...] There is slightly more versatility with the opener method, but no more obviousness
> and no less typing.

I agree with your opinion.  I re-read this report:
- Antoine thinks this fills an important use case, namely avoiding race conditions
- Amaury then suggested the opener argument idea, which was implemented in the openat bug
- Antoine judged it would be better than what we had before

I don’t have a strong opinion on “opener is generic and good enough” vs. “not as ice as it could be”.  Antoine seems to agree with you, so your patch will get reviewed and eventually accepted.  Cheers
msg150674 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-05 17:47
I've done a small review.
msg150860 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-08 11:59
I intend to commit this patch within a couple days (unless anyone objects of course).
msg150867 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-01-08 14:36
I don't think the "created()" method has to be exposed. People can inspect the "mode" attribute if they want to have that information.
(besides, the semantics are misleading since a new file opened with "w" has also be created, but created() would return False)

There's some bogus indentation in the patch (it uses tab characters in some places).

Also:

+    if not (creating + reading or writing or appending):

Not sure why the "+". Shouldn't it be "or" instead?
msg150881 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-08 17:25
Here's a new version of the patch that should address all the comments.
msg150884 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-01-08 17:39
> Here's a new version of the patch that should address all the comments.

Just a small note: FileExistsError is raised, not exactly OSError, when
the file exists.
msg150901 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-08 19:43
> Just a small note: FileExistsError is raised, not exactly OSError,
> when the file exists.

I've updated the doc accordingly.
msg150982 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-09 21:40
New changeset bf609baff4d3 by Charles-François Natali in branch 'default':
Issue #12760: Add a create mode to open(). Patch by David Townshend.
http://hg.python.org/cpython/rev/bf609baff4d3
msg151008 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-10 07:57
Committed.
David, thanks for the patch!
msg151028 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-10 19:37
Nick suggested to call the new flag "exclusive create" in the doc (and explain in whatsnew that it's based C11 new 'x' flag).
Could someone please check the attached patch?
My wording sounds really clumsy, so I'd prefer if a native speaker could review it.
msg151113 - (view) Author: David Townshend (David.Townshend) Date: 2012-01-12 06:04
I've done a bit or rewording in the io docs, but I honestly don't know if this is any better that what you already had!
msg151244 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-14 10:53
New changeset 8bcbe2dc3835 by Charles-François Natali in branch 'default':
Issue #12760: Refer to the new 'x' open mode as "exclusive creation" mode.
http://hg.python.org/cpython/rev/8bcbe2dc3835
msg151249 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-01-14 12:30
Thanks, I've committed your version.
msg161150 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2012-05-19 18:49
Shouldn't the documentation of builtin open() (in Doc/library/functions.rst) be updated too?
msg161193 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-20 09:43
New changeset ef406c4c6463 by Charles-François Natali in branch 'default':
Issue #12760: Add some mising documentation about the new `x` exclusive
http://hg.python.org/cpython/rev/ef406c4c6463
History
Date User Action Args
2012-05-20 09:43:36python-devsetmessages: + msg161193
2012-05-19 18:49:55petri.lehtinensetnosy: + petri.lehtinen
messages: + msg161150
2012-01-14 12:30:00neologixsetmessages: + msg151249
2012-01-14 10:53:45python-devsetmessages: + msg151244
2012-01-12 06:04:53David.Townshendsetfiles: + x_diff2.patch

messages: + msg151113
2012-01-10 19:37:32neologixsetfiles: + x_flag.diff
nosy: + ncoghlan
messages: + msg151028

2012-01-10 07:57:38neologixsetstatus: open -> closed
resolution: fixed
messages: + msg151008

stage: commit review -> resolved
2012-01-09 21:40:39python-devsetnosy: + python-dev
messages: + msg150982
2012-01-08 19:43:17neologixsetmessages: + msg150901
stage: patch review -> commit review
2012-01-08 17:39:12pitrousetmessages: + msg150884
2012-01-08 17:25:08neologixsetfiles: + open_create_x-3.patch

messages: + msg150881
2012-01-08 14:36:57pitrousetmessages: + msg150867
2012-01-08 11:59:58neologixsetmessages: + msg150860
stage: patch review
2012-01-05 17:47:46neologixsetmessages: + msg150674
2011-12-31 00:37:42eric.araujosetmessages: + msg150406
2011-12-30 22:07:18Devin Jeanpierresetmessages: + msg150402
2011-12-30 20:59:04eric.araujosetmessages: + msg150391
2011-12-27 19:04:30cvrebertsetnosy: + cvrebert
2011-12-27 17:11:34pitrousetstatus: closed -> open
resolution: duplicate -> (no value)
superseder: io.FileIO and io.open should support openat ->
messages: + msg150273
2011-12-26 15:42:38Devin Jeanpierresetnosy: + Devin Jeanpierre
messages: + msg150262
2011-11-18 09:55:28eric.araujosetnosy: + eric.araujo
messages: + msg147841
2011-11-07 06:53:29David.Townshendsetmessages: + msg147205
2011-10-31 10:32:11amaury.forgeotdarcsetmessages: + msg146686
2011-10-31 10:22:03David.Townshendsetmessages: + msg146684
2011-10-29 15:58:21neologixsetstatus: open -> closed
resolution: duplicate
superseder: io.FileIO and io.open should support openat
2011-09-08 23:18:43Arfreversetnosy: + Arfrever
2011-08-24 21:34:40hayposetmessages: + msg142930
2011-08-24 21:27:00pitrousetmessages: + msg142928
2011-08-24 21:19:02amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg142923
2011-08-24 20:35:54pitrousetmessages: + msg142916
2011-08-24 20:17:21neologixsetmessages: + msg142912
2011-08-24 20:12:39pitrousetmessages: + msg142910
2011-08-24 19:18:33neologixsetmessages: + msg142902
2011-08-18 16:06:34David.Townshendsetfiles: + open_create_x-2.patch

messages: + msg142352
2011-08-18 15:16:33hayposetmessages: + msg142335
2011-08-18 15:14:16neologixsetmessages: + msg142333
2011-08-18 14:19:17pitrousetmessages: + msg142321
2011-08-18 14:15:58Juliansetnosy: + Julian
messages: + msg142320
2011-08-18 13:28:44David.Townshendsetmessages: + msg142315
2011-08-18 13:24:52neologixsetmessages: + msg142314
2011-08-18 11:56:53David.Townshendsetmessages: + msg142309
2011-08-18 11:33:55pitrousetnosy: + pitrou
messages: + msg142308

assignee: docs@python ->
components: - Documentation, Tests
2011-08-16 15:43:20David.Townshendsetmessages: + msg142201
2011-08-16 15:37:05neologixsetnosy: + neologix
messages: + msg142200
2011-08-16 14:53:24benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg142198
2011-08-16 13:57:33hayposetnosy: + haypo
messages: + msg142194
2011-08-16 13:49:18David.Townshendcreate