msg142193 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-08-16 13:49 |
Currently, opening a file with open(file, 'w') overwrites existing files. It would be useful for open() to raise an error when the file exists. This proposal is to add a 'c' mode to open, which has the effect to creating a file and opening it for writing, but raising an IOError if it already exists (i.e. the same as os.open(file, os.O_EXCL|os.O_CREAT).
The attached patch implements this, including tests and documentation.
|
msg142194 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2011-08-16 13:57 |
I'm not sure that O_EXCL is portable (exist on all platforms) because Python source code uses "#ifdef O_EXCL".
|
msg142198 - (view) |
Author: Benjamin Peterson (benjamin.peterson) *  |
Date: 2011-08-16 14:53 |
I think this should be brought up on python-ideas or python-dev.
|
msg142200 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2011-08-16 15:37 |
See also issue 12105.
A couple downsides:
- O_EXCL is not necessarily portable (doesn't work well with NFS, maybe not on Windows?)
- O_EXCL is useful, as is O_CLOEXEC: to be consistent, we should also end up adding all other commonly-used flags, which are really OS-specific
Furthermore, you can achieve the same thing with:
os.fdopen(os.open('/etc/fstab', os.O_WRONLY|os.O_CLOEXEC|os.O_CREAT))
it's more verbose, though.
|
msg142201 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-08-16 15:43 |
It was discussed on python-ideas, but the subject of the thread was actually on shutils.move so it was not really discussed much. I will repost this idea separately.
|
msg142308 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-08-18 11:33 |
The "#ifdef O_EXCL" in the source code is probably very old. Copying a message I posted on python-ideas:
O_EXCL is a POSIX standard. It is also supported
under Windows by the _open/_wopen compatibility functions (which we use
for file I/O).
Probably there are very old systems which don't support it, and perhaps
new systems that don't implement it *correctly* (meaning not
atomically); for the former I'd say we just don't care (who's gonna run
Python 3 on a 1995 system?) and for the latter, well, if the OS
designers think it's fine, let's just expose it as it is.
As for NFS, there's an interesting comment from 2007 here:
http://lwn.net/Articles/251971/
“My NFS tester shows that it at least appears to work with Linux,
Solaris and FreeBSD:
http://www.dovecot.org/list/dovecot/2007-July/024102.html. Looking at
Linux 2.6 sources it doesn't look like it tries to implement a racy
O_EXCL check in client side (fs/nfs/nfs3proc.c nfs3_proc_create()), so
the test's results should be correct. I don't know if other OSes do
that. I guess it would be nice to have a better O_EXCL tester which
tries to catch race conditions.”
|
msg142309 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-08-18 11:56 |
My aim isn't to add all the commonly used flags, that would be pointless since its already possible using os.open. The aim is to add a missing feature to the builtin open(), i.e. file creation. At the moment open() implements read, write, and append, and creation is only implied by write. But in many cases, an explicit creation is desired (i.e, specifically create a new file, with an exception on failure). It is true that this is possible with os.open, but it is somewhat obscure, especially for beginners, and is not as easy to read as "open(file, 'c')"
|
msg142314 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2011-08-18 13:24 |
> The "#ifdef O_EXCL" in the source code is probably very old. Copying a message I posted on python-ideas:
>
> O_EXCL is a POSIX standard. It is also supported
> under Windows by the _open/_wopen compatibility functions (which we use
> for file I/O).
>
If it's supported by Windows then I'm OK (not that I personaly care
about Windows :-).
> and is not as easy to read as "open(file, 'c')"
Well, I'd rather have this flag called 'x', to be consistent with
glibc's fopen():
"""
c (since glibc 2.3.3)
Do not make the open operation, or subsequent read and write
operations, thread cancellation points.
x Open the file exclusively (like the O_EXCL flag of
open(2)). If the
file already exists, fopen() fails, and sets errno to
EEXIST. This
flag is ignored for fdopen().
"""
By the way, could you submit your patch as a mercurial diff ("hg diff")?
It makes it easier to review under Rietveld.
|
msg142315 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-08-18 13:28 |
Changing form 'c' to 'x' is easy enough, and if there is already a convention it makes sense to stick to it.
I thought I had done a mercurial diff! I'll try again and resubmit.
|
msg142320 - (view) |
Author: Julian Berman (Julian) * |
Date: 2011-08-18 14:15 |
A minor documentation error in io.rst line 475 which was changed to:
+ The *mode* can be ``'c'``, ``'r'``, ``'w'`` or ``'a'`` for reading
+ (default), writing, or appending.
and should have "creating" at the front I assume, like you have it later.
|
msg142321 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-08-18 14:19 |
> Well, I'd rather have this flag called 'x', to be consistent with
> glibc's fopen():
>
> """
> c (since glibc 2.3.3)
> Do not make the open operation, or subsequent read and write
> operations, thread cancellation points.
>
> x Open the file exclusively (like the O_EXCL flag of
> open(2)). If the
> file already exists, fopen() fails, and sets errno to
> EEXIST. This
> flag is ignored for fdopen().
Yeah, but I think "exclusively" is quite misleading since it does not
perform any locking of any kind. Also, I don't think we'll ever
integrate the glibc's "c" option in io.open().
|
msg142333 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2011-08-18 15:14 |
> Yeah, but I think "exclusively" is quite misleading since it does not
> perform any locking of any kind.
It might be misleading, but I find it clear enough, and this name has been endorsed by POSIX.
Furthermore, there's an added bonus: actually, with the old I/O layer, one can already pass an 'x' flag to open, since it just calls fopen:
"""
cf@neobox:~$ strace -e open python -c "open('/tmp/foo', 'wx')"
[...]
open("/tmp/foo", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0666) = 3
cf@neobox:~$ strace -e open python -c "open('/tmp/foo', 'wx')"
[...]
open("/usr/lib/pymodules/python2.6/<string>", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
IOError: [Errno 17] File exists: '/tmp/foo'
"""
I don't know if it's documented behavior, but the OP in issue 12105 was using it with python 2.
Changing it to 'x' would make such code backward-compatible.
Finally, when I read open('/tmp/foo', 'wx'), it's immediately clear to me what's going on, while I'd have to look at open()'s documentation to find out what the 'c' flag does.
|
msg142335 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2011-08-18 15:16 |
> I don't know if it's documented behavior
See #12103: it is not documented.
|
msg142352 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-08-18 16:06 |
I hope this patch suits you better :-)
I've updated the documentation typo (thanks for pointing that out). I've also changed 'c' to 'x', since I think that if there is a convention we should stick to it. I don't think that it matters if the glibc docs say 'exclusive', as long it its clear in the python docs what it does, which I think it is. Having said that, I don't really have a strong opinion either way, so I'll happily change it back to 'c' if its preferred.
|
msg142902 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2011-08-24 19:18 |
So, what was the conclusion of the discussion brought up on python-dev?
I had a feeling some core devs were opposed to this (I'm personally -0).
|
msg142910 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-08-24 20:12 |
I haven't seen any discussion on python-dev. Have I missed something?
|
msg142912 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2011-08-24 20:17 |
> I haven't seen any discussion on python-dev. Have I missed something?
It was on Python-ideas actually:
http://mail.python.org/pipermail/python-ideas/2011-August/011183.html
|
msg142916 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-08-24 20:35 |
Ah, right. Well I think all arguments against it were quite weak (or even wrong). Let's see:
- it's not cross-platform: actually it is (OS_EXCL has been POSIX since at least 1997 (*), and Windows also has it)
- os.open followed by os.fdopen is easy: it isn't that easy to get the incantation right (the pure Python open() in _pyio is 70 lines of code), especially if you want the file object to have the right "name" attribute
- it doesn't fill a use case: actually, avoiding race conditions is an important use case, even though many people may never encounter it (I must admit I myself never really cared about this)
So this looks like a reasonable feature request IMHO.
(*) http://pubs.opengroup.org/onlinepubs/007908799/xsh/open.html
|
msg142923 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *  |
Date: 2011-08-24 21:19 |
> - os.open followed by os.fdopen is easy: it isn't that easy to get
> the incantation right (the pure Python open() in _pyio is 70 lines
> of code), especially if you want the file object to have the right
> "name" attribute
What if we can override the inner call to os.open()?
For example, io.open() could grow an additional argument "fd_opener" which defaults to the equivalent of os.open.
Then creation mode can be expressed like this:
open(filename, 'w',
fd_opener=lambda path, mode: os.open(path, mode|os.O_CREAT)
Another use case with openat (see #12797):
open(filename,
fd_opener=lambda path, mode: os.openat(fd, path, mode)
|
msg142928 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-08-24 21:27 |
> What if we can override the inner call to os.open()?
> For example, io.open() could grow an additional argument "fd_opener"
> which defaults to the equivalent of os.open.
I agree it would be a much better situation than what we have now.
|
msg142930 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2011-08-24 21:34 |
> open(filename, 'w',
> fd_opener=lambda path, mode: os.open(path, mode|os.O_CREAT)
I prefer open(name, "c").
> it doesn't fill a use case: actually, avoiding race conditions
> is an important use case, ...
It may help the issue #8604 (and maybe #8828).
|
msg146684 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-10-31 10:22 |
I see this has been marked as a duplicate of http://bugs.python.org/issue12797. Please explain how this is, since that proposal does not appear to provide the functionality discussed here.
|
msg146686 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *  |
Date: 2011-10-31 10:32 |
issue12797 would allow things like:
def create_exclusive_file(filename):
return open(filename, "w",
opener=lambda path, mode: os.open(path, mode|os.O_CREAT|os.O_EXCL))
|
msg147205 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2011-11-07 06:53 |
It is already possible to write a wrapper function that does it:
def create(file):
fd = os.open(file, os.O_EXCL | os.O_CREAT | os.O_WRONLY)
return os.fdopen(fd)
The point it not that it can't be done, but that it is not straight forward. The docs say this about os.open(): "This function is intended for low-level I/O. For normal usage, use the built-in function open()"
I wouldn't call creating a new file low-level I/O, but it is normal usage.
|
msg147841 - (view) |
Author: Éric Araujo (eric.araujo) *  |
Date: 2011-11-18 09:55 |
See #13424 for a doc request about this.
|
msg150262 - (view) |
Author: Devin Jeanpierre (Devin Jeanpierre) * |
Date: 2011-12-26 15:42 |
C11 uses 'x' for this, for what it's worth.
This is not a "duplicate issue". The openat solution is no easier than the os.open solution.
|
msg150273 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2011-12-27 17:11 |
> C11 uses 'x' for this, for what it's worth.
>
> This is not a "duplicate issue". The openat solution is no easier than
> the os.open solution.
Ok, let's re-open then. I'm not sold on the feature, but the fact C11 adds a dedicated letter mode for it could be a good excuse for us to mimick it :)
|
msg150391 - (view) |
Author: Éric Araujo (eric.araujo) *  |
Date: 2011-12-30 20:59 |
> This is not a "duplicate issue". The openat solution is no easier than the os.open
> solution.
Amaury did not suggest to use openat, but the new opener argument to open, which was especially added for use cases such as the one discussed here:
...
open_exclusive = lambda path, mode: os.open(path, mode|os.O_CREAT|os.O_EXCL))
...
fp = open(filename, 'w', opener=open_exclusive)
...
That’s why this bug was initially closed as duplicate.
|
msg150402 - (view) |
Author: Devin Jeanpierre (Devin Jeanpierre) * |
Date: 2011-12-30 22:07 |
> Amaury did not suggest to use openat, but the new opener argument to open, which was especially added for use cases such as the one discussed here:
Sorry, yes. Wrong words, same thought. We can implement this using opener, but we could implement this with os.open before. What's changed, except that there's more ways to do it? (There is slightly more versatility with the opener method, but no more obviousness and no less typing).
My understanding from reading the other thread is that this is not the primary use-case of the new parameter for open(). In fact, this ticket was not really mentioned at all there.
|
msg150406 - (view) |
Author: Éric Araujo (eric.araujo) *  |
Date: 2011-12-31 00:37 |
> [...] There is slightly more versatility with the opener method, but no more obviousness
> and no less typing.
I agree with your opinion. I re-read this report:
- Antoine thinks this fills an important use case, namely avoiding race conditions
- Amaury then suggested the opener argument idea, which was implemented in the openat bug
- Antoine judged it would be better than what we had before
I don’t have a strong opinion on “opener is generic and good enough” vs. “not as ice as it could be”. Antoine seems to agree with you, so your patch will get reviewed and eventually accepted. Cheers
|
msg150674 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-05 17:47 |
I've done a small review.
|
msg150860 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-08 11:59 |
I intend to commit this patch within a couple days (unless anyone objects of course).
|
msg150867 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2012-01-08 14:36 |
I don't think the "created()" method has to be exposed. People can inspect the "mode" attribute if they want to have that information.
(besides, the semantics are misleading since a new file opened with "w" has also be created, but created() would return False)
There's some bogus indentation in the patch (it uses tab characters in some places).
Also:
+ if not (creating + reading or writing or appending):
Not sure why the "+". Shouldn't it be "or" instead?
|
msg150881 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-08 17:25 |
Here's a new version of the patch that should address all the comments.
|
msg150884 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2012-01-08 17:39 |
> Here's a new version of the patch that should address all the comments.
Just a small note: FileExistsError is raised, not exactly OSError, when
the file exists.
|
msg150901 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-08 19:43 |
> Just a small note: FileExistsError is raised, not exactly OSError,
> when the file exists.
I've updated the doc accordingly.
|
msg150982 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-01-09 21:40 |
New changeset bf609baff4d3 by Charles-François Natali in branch 'default':
Issue #12760: Add a create mode to open(). Patch by David Townshend.
http://hg.python.org/cpython/rev/bf609baff4d3
|
msg151008 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-10 07:57 |
Committed.
David, thanks for the patch!
|
msg151028 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-10 19:37 |
Nick suggested to call the new flag "exclusive create" in the doc (and explain in whatsnew that it's based C11 new 'x' flag).
Could someone please check the attached patch?
My wording sounds really clumsy, so I'd prefer if a native speaker could review it.
|
msg151113 - (view) |
Author: David Townshend (David.Townshend) |
Date: 2012-01-12 06:04 |
I've done a bit or rewording in the io docs, but I honestly don't know if this is any better that what you already had!
|
msg151244 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-01-14 10:53 |
New changeset 8bcbe2dc3835 by Charles-François Natali in branch 'default':
Issue #12760: Refer to the new 'x' open mode as "exclusive creation" mode.
http://hg.python.org/cpython/rev/8bcbe2dc3835
|
msg151249 - (view) |
Author: Charles-François Natali (neologix) *  |
Date: 2012-01-14 12:30 |
Thanks, I've committed your version.
|
msg161150 - (view) |
Author: Petri Lehtinen (petri.lehtinen) *  |
Date: 2012-05-19 18:49 |
Shouldn't the documentation of builtin open() (in Doc/library/functions.rst) be updated too?
|
msg161193 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-05-20 09:43 |
New changeset ef406c4c6463 by Charles-François Natali in branch 'default':
Issue #12760: Add some mising documentation about the new `x` exclusive
http://hg.python.org/cpython/rev/ef406c4c6463
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:20 | admin | set | github: 56969 |
2012-05-20 09:43:36 | python-dev | set | messages:
+ msg161193 |
2012-05-19 18:49:55 | petri.lehtinen | set | nosy:
+ petri.lehtinen messages:
+ msg161150
|
2012-01-14 12:30:00 | neologix | set | messages:
+ msg151249 |
2012-01-14 10:53:45 | python-dev | set | messages:
+ msg151244 |
2012-01-12 06:04:53 | David.Townshend | set | files:
+ x_diff2.patch
messages:
+ msg151113 |
2012-01-10 19:37:32 | neologix | set | files:
+ x_flag.diff nosy:
+ ncoghlan messages:
+ msg151028
|
2012-01-10 07:57:38 | neologix | set | status: open -> closed resolution: fixed messages:
+ msg151008
stage: commit review -> resolved |
2012-01-09 21:40:39 | python-dev | set | nosy:
+ python-dev messages:
+ msg150982
|
2012-01-08 19:43:17 | neologix | set | messages:
+ msg150901 stage: patch review -> commit review |
2012-01-08 17:39:12 | pitrou | set | messages:
+ msg150884 |
2012-01-08 17:25:08 | neologix | set | files:
+ open_create_x-3.patch
messages:
+ msg150881 |
2012-01-08 14:36:57 | pitrou | set | messages:
+ msg150867 |
2012-01-08 11:59:58 | neologix | set | messages:
+ msg150860 stage: patch review |
2012-01-05 17:47:46 | neologix | set | messages:
+ msg150674 |
2011-12-31 00:37:42 | eric.araujo | set | messages:
+ msg150406 |
2011-12-30 22:07:18 | Devin Jeanpierre | set | messages:
+ msg150402 |
2011-12-30 20:59:04 | eric.araujo | set | messages:
+ msg150391 |
2011-12-27 19:04:30 | cvrebert | set | nosy:
+ cvrebert
|
2011-12-27 17:11:34 | pitrou | set | status: closed -> open resolution: duplicate -> (no value) superseder: io.FileIO and io.open should support openat -> messages:
+ msg150273
|
2011-12-26 15:42:38 | Devin Jeanpierre | set | nosy:
+ Devin Jeanpierre messages:
+ msg150262
|
2011-11-18 09:55:28 | eric.araujo | set | nosy:
+ eric.araujo messages:
+ msg147841
|
2011-11-07 06:53:29 | David.Townshend | set | messages:
+ msg147205 |
2011-10-31 10:32:11 | amaury.forgeotdarc | set | messages:
+ msg146686 |
2011-10-31 10:22:03 | David.Townshend | set | messages:
+ msg146684 |
2011-10-29 15:58:21 | neologix | set | status: open -> closed resolution: duplicate superseder: io.FileIO and io.open should support openat |
2011-09-08 23:18:43 | Arfrever | set | nosy:
+ Arfrever
|
2011-08-24 21:34:40 | vstinner | set | messages:
+ msg142930 |
2011-08-24 21:27:00 | pitrou | set | messages:
+ msg142928 |
2011-08-24 21:19:02 | amaury.forgeotdarc | set | nosy:
+ amaury.forgeotdarc messages:
+ msg142923
|
2011-08-24 20:35:54 | pitrou | set | messages:
+ msg142916 |
2011-08-24 20:17:21 | neologix | set | messages:
+ msg142912 |
2011-08-24 20:12:39 | pitrou | set | messages:
+ msg142910 |
2011-08-24 19:18:33 | neologix | set | messages:
+ msg142902 |
2011-08-18 16:06:34 | David.Townshend | set | files:
+ open_create_x-2.patch
messages:
+ msg142352 |
2011-08-18 15:16:33 | vstinner | set | messages:
+ msg142335 |
2011-08-18 15:14:16 | neologix | set | messages:
+ msg142333 |
2011-08-18 14:19:17 | pitrou | set | messages:
+ msg142321 |
2011-08-18 14:15:58 | Julian | set | nosy:
+ Julian messages:
+ msg142320
|
2011-08-18 13:28:44 | David.Townshend | set | messages:
+ msg142315 |
2011-08-18 13:24:52 | neologix | set | messages:
+ msg142314 |
2011-08-18 11:56:53 | David.Townshend | set | messages:
+ msg142309 |
2011-08-18 11:33:55 | pitrou | set | nosy:
+ pitrou messages:
+ msg142308
assignee: docs@python -> components:
- Documentation, Tests |
2011-08-16 15:43:20 | David.Townshend | set | messages:
+ msg142201 |
2011-08-16 15:37:05 | neologix | set | nosy:
+ neologix messages:
+ msg142200
|
2011-08-16 14:53:24 | benjamin.peterson | set | nosy:
+ benjamin.peterson messages:
+ msg142198
|
2011-08-16 13:57:33 | vstinner | set | nosy:
+ vstinner messages:
+ msg142194
|
2011-08-16 13:49:18 | David.Townshend | create | |