classification
Title: os.posix_fallocate() generate exception with errno 0
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mariatta, asvetlov, izbyshev, koobs, larry, ned.deily, serhiy.storchaka, socketpair, vstinner
Priority: Keywords: patch

Created on 2017-08-02 15:03 by socketpair, last changed 2018-05-26 22:03 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
koobs-freebsd-current.non-debug.3.x.build.16.stdio.txt koobs, 2018-02-21 03:05
Pull Requests
URL Status Linked Edit
PR 3000 merged socketpair, 2017-08-04 12:37
PR 3088 closed larry, 2017-08-14 13:59
PR 4101 merged python-dev, 2017-10-24 09:01
Messages (19)
msg299660 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-02 15:03
===
os.posix_fallocate(os.open('qwe.qwe', os.O_RDONLY|os.O_CREAT), 0, 1024*1024)
===

generates OSError with errno 0. Suppose this happen due to O_RDONLY flag.

strace :

====
open("qwe.qwe", O_RDONLY|O_CREAT|O_CLOEXEC, 0777) = 3
fallocate(3, 0, 0, 1048576)             = -1 EBADF (Bad file descriptor)
====


Python 3.5.3, Ubuntu 64-bit.
msg299740 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-04 11:58
man posix_fallocate:

RETURN VALUE

posix_fallocate() returns zero on success, or an error number on failure.  Note that errno is not set.
msg299741 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-04 12:12
Also, EINTR will not be caught too (!)
msg299742 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-04 12:38
https://github.com/python/cpython/pull/3000
msg299746 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-04 13:09
All checks passed.

Please merge.
msg299747 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-04 13:20
posix_fadvise() is also affected :(

Fixed in this patch.
msg299830 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-08-07 08:23
Adding Larry who is listed as posix expert.
msg300244 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-08-14 13:55
New changeset d4b93e21c2664d6a78e0656e7a7be0807be1c352 by larryhastings (Коренберг Марк) in branch 'master':
bpo-31106: Fix handling of erros in posix_fallocate() and posix_fadvise() (#3000) (#3000)
https://github.com/python/cpython/commit/d4b93e21c2664d6a78e0656e7a7be0807be1c352
msg300259 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-08-14 18:09
Needed a NEWS entry.
msg300264 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-08-14 20:39
If I understand right, Python 3.5 will not be fixed with this pathc. Right ?

If yes, I will tell Debian maintainers to backport this patch to Python 3.5, which is shipped with latest stable Debian 9.
msg300266 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-08-14 22:14
You understand correctly.  It won't be backported to Python 3.5.
msg304876 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-24 09:02
Марк, do you mind to add a NEWS entry?
msg308955 - (view) Author: Марк Коренберг (socketpair) * Date: 2017-12-23 14:46
Yes, I can. Should I create new PR ? to which branch ?
msg311429 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-02-01 14:07
New changeset 3a04c52a9eb03e31c60037248b872f3662002a4d by Andrew Svetlov (Miss Islington (bot)) in branch '3.6':
bpo-31106: Fix handling of erros in posix_fallocate() and posix_fadvise() (GH-3000) (GH-3000) (#4101)
https://github.com/python/cpython/commit/3a04c52a9eb03e31c60037248b872f3662002a4d
msg312453 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2018-02-21 03:05
I believe changeset d4b93e21c2664d6a78e0656e7a7be0807be1c352 may be the cause of buildbot failures on FreeBSD (at least koobs-freebsd-current, log attached), due to only the EBADF error being handled (not EINVAL, et al).

Unfortunately the worker was been offline for a longer than anticipated period of time and only recently was restored which delayed picking it up.

The issue (in this workers case) is related to the buildbot home/data directory being on a ZFS filesystem, who's host was recently updated (from current late last year to yesterday).

Initial investigation/isolation/references:

Disable posix_fallocate(2) for ZFS
https://lists.freebsd.org/pipermail/svn-src-head/2017-November/105709.html

Note: Above change will be relevant (merged) for at least FreeBSD 12 and 11, but perhaps even 10.

0000684: posix_fallocate() should be allowed to return ENOTSUP (Interp Status: Approved)
http://austingroupbugs.net/view.php?id=684

[HEADS UP] posix_fallocate support removed from ZFS, lld affected
https://lists.freebsd.org/pipermail/freebsd-current/2017-November/067463.html
Snip: "illumos and ZoL seem to do the same."

Gold fails when output file is lying on ZFS 
https://sourceware.org/bugzilla/show_bug.cgi?id=22540

Given the wide scope of use of zfs and both syscalls across multiple OS's, and possible changes to POSIX documentation/standards, changes to the underlying fallocate/fadvise functions may also be indicated here.
msg312475 - (view) Author: Alexey Izbyshev (izbyshev) * (Python triager) Date: 2018-02-21 07:05
The failed test is test_posix_fallocate, not test_posix_fallocate_errno. The former special-cases Solaris for EINVAL already, and the latter merely passes an invalid descriptor.

It seems that this particular issue is still open only because there is no NEWS entry in the merged commits.
msg313013 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-02-27 14:58
I agree with Alexey's analysis.

Koobs, could you please open a new issue about the failing test?  Unless someone knows of a foolproof way to determine on all platforms that a file is resident on ZFS, the fix is to skip the test on FreeBSD as well if ZFS is likely to be used there.

Also, can someone add a NEWS entry and close this issue?
msg317538 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 06:39
Considering we have already shipped this fix in 3.6.5, I guess we can live without a NEWS entry at this point -> closing this issue.
msg317772 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-26 22:03
P.S.:
> Koobs, could you please open a new issue about the failing test?  Unless someone knows of a foolproof way to determine on all platforms that a file is resident on ZFS, the fix is to skip the test on FreeBSD as well if ZFS is likely to be used there.

The unrelated problem identified in msg312453 has now been fixed in Issue33655.
History
Date User Action Args
2018-07-11 04:41:31benjamin.petersonlinkissue29342 superseder
2018-05-26 22:03:05ned.deilysetmessages: + msg317772
2018-05-24 06:39:10ned.deilysetpriority: deferred blocker ->
status: open -> closed
messages: + msg317538

stage: patch review -> resolved
2018-02-27 14:58:26ned.deilysetpriority: normal -> deferred blocker
nosy: + ned.deily
messages: + msg313013

2018-02-21 07:05:42izbyshevsetnosy: + izbyshev
messages: + msg312475
2018-02-21 03:05:20koobssetfiles: + koobs-freebsd-current.non-debug.3.x.build.16.stdio.txt
versions: + Python 3.8
nosy: + koobs

messages: + msg312453
2018-02-01 14:07:11asvetlovsetnosy: + asvetlov
messages: + msg311429
2017-12-23 14:46:19socketpairsetmessages: + msg308955
2017-10-24 09:02:49serhiy.storchakasetmessages: + msg304876
2017-10-24 09:01:10python-devsetkeywords: + patch
pull_requests: + pull_request4071
2017-08-14 22:14:15larrysetmessages: + msg300266
2017-08-14 20:39:24socketpairsetmessages: + msg300264
2017-08-14 18:09:36serhiy.storchakasetnosy: + serhiy.storchaka

messages: + msg300259
versions: - Python 3.5
2017-08-14 13:59:16larrysetpull_requests: + pull_request3129
2017-08-14 13:55:19larrysetmessages: + msg300244
2017-08-07 09:09:47vstinnersetnosy: + vstinner
2017-08-07 08:23:53Mariattasetnosy: + Mariatta, larry

messages: + msg299830
stage: patch review
2017-08-04 13:20:05socketpairsetmessages: + msg299747
2017-08-04 13:09:56socketpairsetmessages: + msg299746
2017-08-04 12:38:50socketpairsetmessages: + msg299742
2017-08-04 12:37:31socketpairsetpull_requests: + pull_request3036
2017-08-04 12:12:23socketpairsetmessages: + msg299741
2017-08-04 11:58:39socketpairsetmessages: + msg299740
2017-08-02 15:03:06socketpaircreate