classification
Title: test_eintr fails on AIX since fcntl functions were modified
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mariatta, Michael.Felt, kadler, miss-islington
Priority: normal Keywords: patch, patch

Created on 2019-01-01 18:09 by Michael.Felt, last changed 2021-05-10 21:30 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11424 merged Michael.Felt, 2019-01-03 17:29
PR 11424 merged Michael.Felt, 2019-01-03 17:29
PR 11858 merged miss-islington, 2019-02-14 18:22
PR 11500 Michael.Felt, 2019-02-18 11:05
Messages (18)
msg332846 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-01 18:09
test_eintr fails on AIX since fcntl functions were modified
In issue35189 the fnctl() module was modified so that the EINTR interruption should be retried automatically.

On AIX the test for flock() passes, but the test for lockf() fails:

 ======================================================================
> ERROR: test_lockf (__main__.FNTLEINTRTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 522, in test_lockf
>     self._lock(fcntl.lockf, "lockf")
>   File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 507, in _lock
>     lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
> PermissionError: [Errno 13] Permission denied
>

Researching...
msg332936 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-03 17:04
After reading the PEP I realized it is much simpler. The test is for interrupts that occur at a low-level - and not for permission issues. The test is failing because there is a permission issue, not a missed interrupt issue. Modifying the code to: (see line 510)

  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except (BlockingIOError, PermissionError):
  +511                          break
  +512                  # the child locked the file just a moment ago for 'sleep_time' seconds
  +513                  # that means that the lock below will block for 'sleep_time' minus some
  +514                  # potential context switch delay
  +515                  lock_func(f, fcntl.LOCK_EX)
  +516                  dt = time.monotonic() - start_time
  +517                  self.assertGreaterEqual(dt, self.sleep_time)
  +518                  self.stop_alarm()

fixes this.
msg333015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-04 22:42
Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode?

I'm talking about this line:

open(support.TESTFN, 'wb')

Note: if you want to test, you have the modify the mode twice:
  "with open('%s', 'wb') as f:" % support.TESTFN,
and
  with open(support.TESTFN, 'wb') as f:
msg333020 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-04 23:03
Previous discussion at:
https://bugs.python.org/issue35189#msg332580
msg333048 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-05 08:25
I had tried “wb+”, not “w+b”. Is there a difference? I forget if I tried just “w+”. 

But I’ll do them anyway/again to be sure. 

Sent from my iPhone

> On 4 Jan 2019, at 23:43, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> Change by STINNER Victor <vstinner@redhat.com>:
> 
> 
> ----------
> title: test_eintr fails on AIX since fcntl functions were modified -> test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>
msg333064 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-05 16:26
On 04/01/2019 23:42, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode?
>
> I'm talking about this line:
>
> open(support.TESTFN, 'wb')
>
> Note: if you want to test, you have the modify the mode twice:
>   "with open('%s', 'wb') as f:" % support.TESTFN,
> and
>   with open(support.TESTFN, 'wb') as f:

Without except containing PermissionError:

  +500          with kill_on_error(proc):
  +501              with open(support.TESTFN, 'w+b') as f:
  +502                  while True:  # synchronize the subprocess
  +503                      dt = time.monotonic() - start_time
  +504                      if dt > 60.0:
  +505                          raise Exception("failed to sync child in
%.1f sec" % dt)
  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except BlockingIOError:
  +511                          break
FAILS on test_lockf, passes on test_flock -> with open(support.TESTFN,
'w+b') as f:

FAILS on both test_lockf and test_flock (as expected I am guessing) ->
with open(support.TESTFN, 'r+b') as f:

I am not python savvy enough to get it tested using the syntax: ' "with
open('%s', 'wb') as f:" % support.TESTFN, '

  +500          with kill_on_error(proc):
  +501              "with open('%s', 'wb') as f:" % support.TESTFN,
  +502                  while True:  # synchronize the subprocess
  +503                      dt = time.monotonic() - start_time
  +504                      if dt > 60.0:
  +505                          raise Exception("failed to sync child in
%.1f sec" % dt)
  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except BlockingIOError:
  +511                          break

--- run eintr_tester.py ---
  File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 502
    while True:  # synchronize the subprocess
    ^
IndentationError: unexpected indent
--- eintr_tester.py completed: exit code 1 ---

>
> ----------
> nosy: +vstinner
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>
msg333157 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-07 14:46
Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?

diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py
index 25c169bde5..4db5dc9045 100644
--- a/Lib/test/eintrdata/eintr_tester.py
+++ b/Lib/test/eintrdata/eintr_tester.py
@@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest):
         self.addCleanup(support.unlink, support.TESTFN)
         code = '\n'.join((
             "import fcntl, time",
-            "with open('%s', 'wb') as f:" % support.TESTFN,
+            "with open('%s', 'w+b') as f:" % support.TESTFN,
             "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
             "   time.sleep(%s)" % self.sleep_time))
         start_time = time.monotonic()
         proc = self.subprocess(code)
         with kill_on_error(proc):
-            with open(support.TESTFN, 'wb') as f:
+            with open(support.TESTFN, 'w+b') as f:
                 while True:  # synchronize the subprocess
                     dt = time.monotonic() - start_time
                     if dt > 60.0:
msg333179 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-07 17:44
On 07/01/2019 15:46, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?

I was clearly a bit blind.. Now with:

  +490  class FNTLEINTRTest(EINTRBaseTest):
  +491      def _lock(self, lock_func, lock_name):
  +492          self.addCleanup(support.unlink, support.TESTFN)
  +493          code = '\n'.join((
  +494              "import fcntl, time",
  +495              "with open('%s', 'w+b') as f:" % support.TESTFN,
  +496              "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
  +497              "   time.sleep(%s)" % self.sleep_time))
  +498          start_time = time.monotonic()
  +499          proc = self.subprocess(code)
  +500          with kill_on_error(proc):
  +501              with open(support.TESTFN, 'w+b') as f:
  +502                  while True:  # synchronize the subprocess
  +503                      dt = time.monotonic() - start_time
  +504                      if dt > 60.0:
  +505                          raise Exception("failed to sync child in
%.1f sec" % dt)
  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except BlockingIOError:
  +511                          break

I get - PermissionError - as before.

root@x066:[/data/prj/python/python3-3.8]./python -m test -v test_eintr
== CPython 3.8.0a0 (heads/master-dirty:b4ea8bb080, Jan 1 2019, 18:24:36) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/python3-3.8/build/test_python_3342514
== CPU count: 8
== encodings: locale=ISO8859-1, FS=iso8859-1
Run tests sequentially
0:00:00 [1/1] test_eintr
test_all (test.test_eintr.EINTRTests) ...
--- run eintr_tester.py ---
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ERROR
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... skipped 'need select.epoll'
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

======================================================================
ERROR: test_lockf (__main__.FNTLEINTRTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 522, in test_lockf
    self._lock(fcntl.lockf, "lockf")
  File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 507, in _lock
    lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
PermissionError: [Errno 13] Permission denied

----------------------------------------------------------------------
Ran 24 tests in 9.098s

FAILED (errors=1, skipped=3)
--- eintr_tester.py completed: exit code 1 ---
FAIL

======================================================================
FAIL: test_all (test.test_eintr.EINTRTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/python/git/python3-3.8/Lib/test/test_eintr.py", line
31, in test_all
    self.fail("eintr_tester.py failed")
AssertionError: eintr_tester.py failed

----------------------------------------------------------------------

Ran 1 test in 9.674s

FAILED (failures=1)
test test_eintr failed
test_eintr failed

== Tests result: FAILURE ==

>
> diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py
> index 25c169bde5..4db5dc9045 100644
> --- a/Lib/test/eintrdata/eintr_tester.py
> +++ b/Lib/test/eintrdata/eintr_tester.py
> @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest):
>          self.addCleanup(support.unlink, support.TESTFN)
>          code = '\n'.join((
>              "import fcntl, time",
> -            "with open('%s', 'wb') as f:" % support.TESTFN,
> +            "with open('%s', 'w+b') as f:" % support.TESTFN,
>              "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
>              "   time.sleep(%s)" % self.sleep_time))
>          start_time = time.monotonic()
>          proc = self.subprocess(code)
>          with kill_on_error(proc):
> -            with open(support.TESTFN, 'wb') as f:
> +            with open(support.TESTFN, 'w+b') as f:
>                  while True:  # synchronize the subprocess
>                      dt = time.monotonic() - start_time
>                      if dt > 60.0:
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>
msg333658 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-15 10:07
On 07/01/2019 15:46, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?
Answer - Yes. Still get PermissionError
>
> diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py
> index 25c169bde5..4db5dc9045 100644
> --- a/Lib/test/eintrdata/eintr_tester.py
> +++ b/Lib/test/eintrdata/eintr_tester.py
> @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest):
>          self.addCleanup(support.unlink, support.TESTFN)
>          code = '\n'.join((
>              "import fcntl, time",
> -            "with open('%s', 'wb') as f:" % support.TESTFN,
> +            "with open('%s', 'w+b') as f:" % support.TESTFN,
>              "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
>              "   time.sleep(%s)" % self.sleep_time))
>          start_time = time.monotonic()
>          proc = self.subprocess(code)
>          with kill_on_error(proc):
> -            with open(support.TESTFN, 'wb') as f:
> +            with open(support.TESTFN, 'w+b') as f:
>                  while True:  # synchronize the subprocess
>                      dt = time.monotonic() - start_time
>                      if dt > 60.0:
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>
msg333662 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-15 10:23
> On AIX the test for flock() passes, but the test for lockf() fails: (...)

I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?

There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.
msg334021 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-18 22:54
I’ll make a new PR and delete the news entry. 

Sent from my iPhone

> On 15 Jan 2019, at 11:23, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@redhat.com> added the comment:
> 
>> On AIX the test for flock() passes, but the test for lockf() fails: (...)
> 
> I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?
> 
> There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>
msg334147 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-01-21 15:31
Done.

> On 1/15/2019 11:23 AM, STINNER Victor wrote:
> I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?
> 
> There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.
msg335550 - (view) Author: miss-islington (miss-islington) Date: 2019-02-14 18:22
New changeset b94d4bed67c60834b1a5a0936b8c13934cf3b872 by Miss Islington (bot) (Michael Felt) in branch 'master':
bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424)
https://github.com/python/cpython/commit/b94d4bed67c60834b1a5a0936b8c13934cf3b872
msg335552 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2019-02-14 18:25
Thanks all!
msg335553 - (view) Author: miss-islington (miss-islington) Date: 2019-02-14 18:41
New changeset 7e618f3154404613cd9774fb8ffb5b48e7233a9e by Miss Islington (bot) in branch '3.7':
bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424)
https://github.com/python/cpython/commit/7e618f3154404613cd9774fb8ffb5b48e7233a9e
msg337630 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-03-10 18:34
Could this also be backported to version 3.6?
msg337701 - (view) Author: Michael Felt (Michael.Felt) * Date: 2019-03-11 21:44
On 10/03/2019 19:34, Michael Felt wrote:
> Michael Felt <aixtools@felt.demon.nl> added the comment:
>
> Could this also be backported to version 3.6?
>
> ----------
> versions: +Python 3.6
Not to worry: As 3.6 is in security mode.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>
msg393161 - (view) Author: Kevin (kadler) * Date: 2021-05-07 01:57
FYI, the problem here is that AIX fcntl returns EACCES in the case that the lock is held and non-blocking behavior was requested:


> The lockfx and lockf subroutines fail if one of the following is true:
Item
> 
> EACCES 	The Command parameter is F_SETLK, the l_type field is F_RDLCK, and the segment of the file to be locked is already write-locked by another process.
> EACCES 	The Command parameter is F_SETLK, the l_type field is F_WRLCK, and the segment of a file to be locked is already read-locked or write-locked by another process.

https://www.ibm.com/docs/en/aix/7.1?topic=l-lockfx-lockf-flock-lockf64-subroutine

(Note the docs are a bit wonky referring to lockf/lockfx but talking about parameters and fields which apply to fcntl instead)

The lockf/flock APIs provided by AIX handle this appropriately, mapping EACCES to EWOULDBLOCK, but while Python calls the libbsd flock API, it uses its own lockf implementation which calls fcntl directly: https://github.com/python/cpython/blob/main/Modules/fcntlmodule.c#L426
History
Date User Action Args
2021-05-10 21:30:57vstinnersetkeywords: patch, patch
nosy: - vstinner
2021-05-07 01:57:38kadlersetnosy: + kadler
messages: + msg393161
2019-03-11 21:44:55Michael.Feltsetmessages: + msg337701
2019-03-10 18:34:12Michael.Feltsetmessages: + msg337630
versions: + Python 3.6
2019-02-18 11:05:07Michael.Feltsetpull_requests: + pull_request11940
2019-02-14 18:41:41miss-islingtonsetmessages: + msg335553
2019-02-14 18:25:02Mariattasetstatus: open -> closed


keywords: patch, patch
nosy: + Mariatta
messages: + msg335552
resolution: fixed
stage: patch review -> resolved
2019-02-14 18:22:48miss-islingtonsetpull_requests: + pull_request11890
2019-02-14 18:22:39miss-islingtonsetnosy: + miss-islington
messages: + msg335550
2019-01-21 15:31:56Michael.Feltsetmessages: + msg334147
2019-01-18 22:54:54Michael.Feltsetmessages: + msg334021
2019-01-15 10:23:14vstinnersetkeywords: patch, patch

messages: + msg333662
2019-01-15 10:07:38Michael.Feltsetmessages: + msg333658
2019-01-07 17:44:27Michael.Feltsetmessages: + msg333179
2019-01-07 14:46:53vstinnersetkeywords: patch, patch

messages: + msg333157
2019-01-05 16:26:53Michael.Feltsetmessages: + msg333064
title: test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX -> test_eintr fails on AIX since fcntl functions were modified
2019-01-05 08:25:56Michael.Feltsetmessages: + msg333048
2019-01-04 23:03:13vstinnersetkeywords: patch, patch

messages: + msg333020
2019-01-04 22:43:26vstinnersetkeywords: patch, patch
title: test_eintr fails on AIX since fcntl functions were modified -> test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX
2019-01-04 22:42:50vstinnersetkeywords: patch, patch
nosy: + vstinner
messages: + msg333015

2019-01-03 17:34:55Michael.Feltsetcomponents: - IO
2019-01-03 17:29:56Michael.Feltsetkeywords: + patch
stage: patch review
pull_requests: + pull_request10841
2019-01-03 17:29:54Michael.Feltsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10840
2019-01-03 17:04:54Michael.Feltsetmessages: + msg332936
2019-01-01 18:09:08Michael.Feltcreate