Issue35633
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-01-01 18:09 by Michael.Felt, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Pull Requests | |||
---|---|---|---|
URL | Status | Linked | Edit |
PR 11424 | merged | Michael.Felt, 2019-01-03 17:29 | |
PR 11424 | merged | Michael.Felt, 2019-01-03 17:29 | |
PR 11858 | merged | miss-islington, 2019-02-14 18:22 | |
PR 11500 | Michael.Felt, 2019-02-18 11:05 |
Messages (18) | |||
---|---|---|---|
msg332846 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-01 18:09 | |
test_eintr fails on AIX since fcntl functions were modified In issue35189 the fnctl() module was modified so that the EINTR interruption should be retried automatically. On AIX the test for flock() passes, but the test for lockf() fails: ====================================================================== > ERROR: test_lockf (__main__.FNTLEINTRTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 522, in test_lockf > self._lock(fcntl.lockf, "lockf") > File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 507, in _lock > lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB) > PermissionError: [Errno 13] Permission denied > Researching... |
|||
msg332936 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-03 17:04 | |
After reading the PEP I realized it is much simpler. The test is for interrupts that occur at a low-level - and not for permission issues. The test is failing because there is a permission issue, not a missed interrupt issue. Modifying the code to: (see line 510) +506 try: +507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB) +508 lock_func(f, fcntl.LOCK_UN) +509 time.sleep(0.01) +510 except (BlockingIOError, PermissionError): +511 break +512 # the child locked the file just a moment ago for 'sleep_time' seconds +513 # that means that the lock below will block for 'sleep_time' minus some +514 # potential context switch delay +515 lock_func(f, fcntl.LOCK_EX) +516 dt = time.monotonic() - start_time +517 self.assertGreaterEqual(dt, self.sleep_time) +518 self.stop_alarm() fixes this. |
|||
msg333015 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-04 22:42 | |
Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode? I'm talking about this line: open(support.TESTFN, 'wb') Note: if you want to test, you have the modify the mode twice: "with open('%s', 'wb') as f:" % support.TESTFN, and with open(support.TESTFN, 'wb') as f: |
|||
msg333020 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-04 23:03 | |
Previous discussion at: https://bugs.python.org/issue35189#msg332580 |
|||
msg333048 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-05 08:25 | |
I had tried “wb+”, not “w+b”. Is there a difference? I forget if I tried just “w+”. But I’ll do them anyway/again to be sure. Sent from my iPhone > On 4 Jan 2019, at 23:43, STINNER Victor <report@bugs.python.org> wrote: > > > Change by STINNER Victor <vstinner@redhat.com>: > > > ---------- > title: test_eintr fails on AIX since fcntl functions were modified -> test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue35633> > _______________________________________ > |
|||
msg333064 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-05 16:26 | |
On 04/01/2019 23:42, STINNER Victor wrote: > STINNER Victor <vstinner@redhat.com> added the comment: > > Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode? > > I'm talking about this line: > > open(support.TESTFN, 'wb') > > Note: if you want to test, you have the modify the mode twice: > "with open('%s', 'wb') as f:" % support.TESTFN, > and > with open(support.TESTFN, 'wb') as f: Without except containing PermissionError: +500 with kill_on_error(proc): +501 with open(support.TESTFN, 'w+b') as f: +502 while True: # synchronize the subprocess +503 dt = time.monotonic() - start_time +504 if dt > 60.0: +505 raise Exception("failed to sync child in %.1f sec" % dt) +506 try: +507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB) +508 lock_func(f, fcntl.LOCK_UN) +509 time.sleep(0.01) +510 except BlockingIOError: +511 break FAILS on test_lockf, passes on test_flock -> with open(support.TESTFN, 'w+b') as f: FAILS on both test_lockf and test_flock (as expected I am guessing) -> with open(support.TESTFN, 'r+b') as f: I am not python savvy enough to get it tested using the syntax: ' "with open('%s', 'wb') as f:" % support.TESTFN, ' +500 with kill_on_error(proc): +501 "with open('%s', 'wb') as f:" % support.TESTFN, +502 while True: # synchronize the subprocess +503 dt = time.monotonic() - start_time +504 if dt > 60.0: +505 raise Exception("failed to sync child in %.1f sec" % dt) +506 try: +507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB) +508 lock_func(f, fcntl.LOCK_UN) +509 time.sleep(0.01) +510 except BlockingIOError: +511 break --- run eintr_tester.py --- File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 502 while True: # synchronize the subprocess ^ IndentationError: unexpected indent --- eintr_tester.py completed: exit code 1 --- > > ---------- > nosy: +vstinner > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue35633> > _______________________________________ > |
|||
msg333157 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-07 14:46 | |
Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError? diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py index 25c169bde5..4db5dc9045 100644 --- a/Lib/test/eintrdata/eintr_tester.py +++ b/Lib/test/eintrdata/eintr_tester.py @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest): self.addCleanup(support.unlink, support.TESTFN) code = '\n'.join(( "import fcntl, time", - "with open('%s', 'wb') as f:" % support.TESTFN, + "with open('%s', 'w+b') as f:" % support.TESTFN, " fcntl.%s(f, fcntl.LOCK_EX)" % lock_name, " time.sleep(%s)" % self.sleep_time)) start_time = time.monotonic() proc = self.subprocess(code) with kill_on_error(proc): - with open(support.TESTFN, 'wb') as f: + with open(support.TESTFN, 'w+b') as f: while True: # synchronize the subprocess dt = time.monotonic() - start_time if dt > 60.0: |
|||
msg333179 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-07 17:44 | |
On 07/01/2019 15:46, STINNER Victor wrote: > STINNER Victor <vstinner@redhat.com> added the comment: > > Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError? I was clearly a bit blind.. Now with: +490 class FNTLEINTRTest(EINTRBaseTest): +491 def _lock(self, lock_func, lock_name): +492 self.addCleanup(support.unlink, support.TESTFN) +493 code = '\n'.join(( +494 "import fcntl, time", +495 "with open('%s', 'w+b') as f:" % support.TESTFN, +496 " fcntl.%s(f, fcntl.LOCK_EX)" % lock_name, +497 " time.sleep(%s)" % self.sleep_time)) +498 start_time = time.monotonic() +499 proc = self.subprocess(code) +500 with kill_on_error(proc): +501 with open(support.TESTFN, 'w+b') as f: +502 while True: # synchronize the subprocess +503 dt = time.monotonic() - start_time +504 if dt > 60.0: +505 raise Exception("failed to sync child in %.1f sec" % dt) +506 try: +507 lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB) +508 lock_func(f, fcntl.LOCK_UN) +509 time.sleep(0.01) +510 except BlockingIOError: +511 break I get - PermissionError - as before. root@x066:[/data/prj/python/python3-3.8]./python -m test -v test_eintr == CPython 3.8.0a0 (heads/master-dirty:b4ea8bb080, Jan 1 2019, 18:24:36) [C] == AIX-1-00C291F54C00-powerpc-32bit big-endian == cwd: /data/prj/python/python3-3.8/build/test_python_3342514 == CPU count: 8 == encodings: locale=ISO8859-1, FS=iso8859-1 Run tests sequentially 0:00:00 [1/1] test_eintr test_all (test.test_eintr.EINTRTests) ... --- run eintr_tester.py --- test_flock (__main__.FNTLEINTRTest) ... ok test_lockf (__main__.FNTLEINTRTest) ... ERROR test_read (__main__.OSEINTRTest) ... ok test_wait (__main__.OSEINTRTest) ... ok test_wait3 (__main__.OSEINTRTest) ... ok test_wait4 (__main__.OSEINTRTest) ... ok test_waitpid (__main__.OSEINTRTest) ... ok test_write (__main__.OSEINTRTest) ... ok test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll' test_epoll (__main__.SelectEINTRTest) ... skipped 'need select.epoll' test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue' test_poll (__main__.SelectEINTRTest) ... ok test_select (__main__.SelectEINTRTest) ... ok test_sigtimedwait (__main__.SignalEINTRTest) ... ok test_sigwaitinfo (__main__.SignalEINTRTest) ... ok test_accept (__main__.SocketEINTRTest) ... ok test_open (__main__.SocketEINTRTest) ... ok test_os_open (__main__.SocketEINTRTest) ... ok test_recv (__main__.SocketEINTRTest) ... ok test_recvmsg (__main__.SocketEINTRTest) ... ok test_send (__main__.SocketEINTRTest) ... ok test_sendall (__main__.SocketEINTRTest) ... ok test_sendmsg (__main__.SocketEINTRTest) ... ok test_sleep (__main__.TimeEINTRTest) ... ok ====================================================================== ERROR: test_lockf (__main__.FNTLEINTRTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 522, in test_lockf self._lock(fcntl.lockf, "lockf") File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 507, in _lock lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB) PermissionError: [Errno 13] Permission denied ---------------------------------------------------------------------- Ran 24 tests in 9.098s FAILED (errors=1, skipped=3) --- eintr_tester.py completed: exit code 1 --- FAIL ====================================================================== FAIL: test_all (test.test_eintr.EINTRTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/prj/python/git/python3-3.8/Lib/test/test_eintr.py", line 31, in test_all self.fail("eintr_tester.py failed") AssertionError: eintr_tester.py failed ---------------------------------------------------------------------- Ran 1 test in 9.674s FAILED (failures=1) test test_eintr failed test_eintr failed == Tests result: FAILURE == > > diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py > index 25c169bde5..4db5dc9045 100644 > --- a/Lib/test/eintrdata/eintr_tester.py > +++ b/Lib/test/eintrdata/eintr_tester.py > @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest): > self.addCleanup(support.unlink, support.TESTFN) > code = '\n'.join(( > "import fcntl, time", > - "with open('%s', 'wb') as f:" % support.TESTFN, > + "with open('%s', 'w+b') as f:" % support.TESTFN, > " fcntl.%s(f, fcntl.LOCK_EX)" % lock_name, > " time.sleep(%s)" % self.sleep_time)) > start_time = time.monotonic() > proc = self.subprocess(code) > with kill_on_error(proc): > - with open(support.TESTFN, 'wb') as f: > + with open(support.TESTFN, 'w+b') as f: > while True: # synchronize the subprocess > dt = time.monotonic() - start_time > if dt > 60.0: > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue35633> > _______________________________________ > |
|||
msg333658 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-15 10:07 | |
On 07/01/2019 15:46, STINNER Victor wrote: > STINNER Victor <vstinner@redhat.com> added the comment: > > Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError? Answer - Yes. Still get PermissionError > > diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py > index 25c169bde5..4db5dc9045 100644 > --- a/Lib/test/eintrdata/eintr_tester.py > +++ b/Lib/test/eintrdata/eintr_tester.py > @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest): > self.addCleanup(support.unlink, support.TESTFN) > code = '\n'.join(( > "import fcntl, time", > - "with open('%s', 'wb') as f:" % support.TESTFN, > + "with open('%s', 'w+b') as f:" % support.TESTFN, > " fcntl.%s(f, fcntl.LOCK_EX)" % lock_name, > " time.sleep(%s)" % self.sleep_time)) > start_time = time.monotonic() > proc = self.subprocess(code) > with kill_on_error(proc): > - with open(support.TESTFN, 'wb') as f: > + with open(support.TESTFN, 'w+b') as f: > while True: # synchronize the subprocess > dt = time.monotonic() - start_time > if dt > 60.0: > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue35633> > _______________________________________ > |
|||
msg333662 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-15 10:23 | |
> On AIX the test for flock() passes, but the test for lockf() fails: (...) I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that? There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX. |
|||
msg334021 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-18 22:54 | |
I’ll make a new PR and delete the news entry. Sent from my iPhone > On 15 Jan 2019, at 11:23, STINNER Victor <report@bugs.python.org> wrote: > > > STINNER Victor <vstinner@redhat.com> added the comment: > >> On AIX the test for flock() passes, but the test for lockf() fails: (...) > > I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that? > > There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX. > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue35633> > _______________________________________ > |
|||
msg334147 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-01-21 15:31 | |
Done. > On 1/15/2019 11:23 AM, STINNER Victor wrote: > I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that? > > There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX. |
|||
msg335550 - (view) | Author: miss-islington (miss-islington) | Date: 2019-02-14 18:22 | |
New changeset b94d4bed67c60834b1a5a0936b8c13934cf3b872 by Miss Islington (bot) (Michael Felt) in branch 'master': bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424) https://github.com/python/cpython/commit/b94d4bed67c60834b1a5a0936b8c13934cf3b872 |
|||
msg335552 - (view) | Author: Mariatta (Mariatta) * | Date: 2019-02-14 18:25 | |
Thanks all! |
|||
msg335553 - (view) | Author: miss-islington (miss-islington) | Date: 2019-02-14 18:41 | |
New changeset 7e618f3154404613cd9774fb8ffb5b48e7233a9e by Miss Islington (bot) in branch '3.7': bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424) https://github.com/python/cpython/commit/7e618f3154404613cd9774fb8ffb5b48e7233a9e |
|||
msg337630 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-03-10 18:34 | |
Could this also be backported to version 3.6? |
|||
msg337701 - (view) | Author: Michael Felt (Michael.Felt) * | Date: 2019-03-11 21:44 | |
On 10/03/2019 19:34, Michael Felt wrote: > Michael Felt <aixtools@felt.demon.nl> added the comment: > > Could this also be backported to version 3.6? > > ---------- > versions: +Python 3.6 Not to worry: As 3.6 is in security mode. > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue35633> > _______________________________________ > |
|||
msg393161 - (view) | Author: Kevin (kadler) * | Date: 2021-05-07 01:57 | |
FYI, the problem here is that AIX fcntl returns EACCES in the case that the lock is held and non-blocking behavior was requested: > The lockfx and lockf subroutines fail if one of the following is true: Item > > EACCES The Command parameter is F_SETLK, the l_type field is F_RDLCK, and the segment of the file to be locked is already write-locked by another process. > EACCES The Command parameter is F_SETLK, the l_type field is F_WRLCK, and the segment of a file to be locked is already read-locked or write-locked by another process. https://www.ibm.com/docs/en/aix/7.1?topic=l-lockfx-lockf-flock-lockf64-subroutine (Note the docs are a bit wonky referring to lockf/lockfx but talking about parameters and fields which apply to fcntl instead) The lockf/flock APIs provided by AIX handle this appropriately, mapping EACCES to EWOULDBLOCK, but while Python calls the libbsd flock API, it uses its own lockf implementation which calls fcntl directly: https://github.com/python/cpython/blob/main/Modules/fcntlmodule.c#L426 |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:09 | admin | set | github: 79814 |
2021-05-10 21:30:57 | vstinner | set | keywords:
patch, patch nosy: - vstinner |
2021-05-07 01:57:38 | kadler | set | nosy:
+ kadler messages: + msg393161 |
2019-03-11 21:44:55 | Michael.Felt | set | messages: + msg337701 |
2019-03-10 18:34:12 | Michael.Felt | set | messages:
+ msg337630 versions: + Python 3.6 |
2019-02-18 11:05:07 | Michael.Felt | set | pull_requests: + pull_request11940 |
2019-02-14 18:41:41 | miss-islington | set | messages: + msg335553 |
2019-02-14 18:25:02 | Mariatta | set | status: open -> closed keywords: patch, patch nosy: + Mariatta messages: + msg335552 resolution: fixed stage: patch review -> resolved |
2019-02-14 18:22:48 | miss-islington | set | pull_requests: + pull_request11890 |
2019-02-14 18:22:39 | miss-islington | set | nosy:
+ miss-islington messages: + msg335550 |
2019-01-21 15:31:56 | Michael.Felt | set | messages: + msg334147 |
2019-01-18 22:54:54 | Michael.Felt | set | messages: + msg334021 |
2019-01-15 10:23:14 | vstinner | set | keywords:
patch, patch messages: + msg333662 |
2019-01-15 10:07:38 | Michael.Felt | set | messages: + msg333658 |
2019-01-07 17:44:27 | Michael.Felt | set | messages: + msg333179 |
2019-01-07 14:46:53 | vstinner | set | keywords:
patch, patch messages: + msg333157 |
2019-01-05 16:26:53 | Michael.Felt | set | messages:
+ msg333064 title: test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX -> test_eintr fails on AIX since fcntl functions were modified |
2019-01-05 08:25:56 | Michael.Felt | set | messages: + msg333048 |
2019-01-04 23:03:13 | vstinner | set | keywords:
patch, patch messages: + msg333020 |
2019-01-04 22:43:26 | vstinner | set | keywords:
patch, patch title: test_eintr fails on AIX since fcntl functions were modified -> test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX |
2019-01-04 22:42:50 | vstinner | set | keywords:
patch, patch nosy: + vstinner messages: + msg333015 |
2019-01-03 17:34:55 | Michael.Felt | set | components: - IO |
2019-01-03 17:29:56 | Michael.Felt | set | keywords:
+ patch stage: patch review pull_requests: + pull_request10841 |
2019-01-03 17:29:54 | Michael.Felt | set | keywords:
+ patch stage: (no value) pull_requests: + pull_request10840 |
2019-01-03 17:04:54 | Michael.Felt | set | messages: + msg332936 |
2019-01-01 18:09:08 | Michael.Felt | create |