test_eintr fails on AIX since fcntl functions were modified
In issue35189 the fnctl() module was modified so that the EINTR interruption should be retried automatically.

On AIX the test for flock() passes, but the test for lockf() fails:

 ======================================================================
> ERROR: test_lockf (__main__.FNTLEINTRTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 522, in test_lockf
>     self._lock(fcntl.lockf, "lockf")
>   File "/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py", line 507, in _lock
>     lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
> PermissionError: [Errno 13] Permission denied
>

Researching...

After reading the PEP I realized it is much simpler. The test is for interrupts that occur at a low-level - and not for permission issues. The test is failing because there is a permission issue, not a missed interrupt issue. Modifying the code to: (see line 510)

  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except (BlockingIOError, PermissionError):
  +511                          break
  +512                  # the child locked the file just a moment ago for 'sleep_time' seconds
  +513                  # that means that the lock below will block for 'sleep_time' minus some
  +514                  # potential context switch delay
  +515                  lock_func(f, fcntl.LOCK_EX)
  +516                  dt = time.monotonic() - start_time
  +517                  self.assertGreaterEqual(dt, self.sleep_time)
  +518                  self.stop_alarm()

fixes this.

Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode?

I'm talking about this line:

open(support.TESTFN, 'wb')

Note: if you want to test, you have the modify the mode twice:
  "with open('%s', 'wb') as f:" % support.TESTFN,
and
  with open(support.TESTFN, 'wb') as f:

Previous discussion at:
https://bugs.python.org/issue35189#msg332580

I had tried “wb+”, not “w+b”. Is there a difference? I forget if I tried just “w+”. 

But I’ll do them anyway/again to be sure. 

Sent from my iPhone

> On 4 Jan 2019, at 23:43, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> Change by STINNER Victor <vstinner@redhat.com>:
> 
> 
> ----------
> title: test_eintr fails on AIX since fcntl functions were modified -> test_eintr: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>

On 04/01/2019 23:42, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Does the test pass if you open the file in read+write ("w+b") mode rather than write-only ("wb") mode?
>
> I'm talking about this line:
>
> open(support.TESTFN, 'wb')
>
> Note: if you want to test, you have the modify the mode twice:
>   "with open('%s', 'wb') as f:" % support.TESTFN,
> and
>   with open(support.TESTFN, 'wb') as f:

Without except containing PermissionError:

  +500          with kill_on_error(proc):
  +501              with open(support.TESTFN, 'w+b') as f:
  +502                  while True:  # synchronize the subprocess
  +503                      dt = time.monotonic() - start_time
  +504                      if dt > 60.0:
  +505                          raise Exception("failed to sync child in
%.1f sec" % dt)
  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except BlockingIOError:
  +511                          break
FAILS on test_lockf, passes on test_flock -> with open(support.TESTFN,
'w+b') as f:

FAILS on both test_lockf and test_flock (as expected I am guessing) ->
with open(support.TESTFN, 'r+b') as f:

I am not python savvy enough to get it tested using the syntax: ' "with
open('%s', 'wb') as f:" % support.TESTFN, '

  +500          with kill_on_error(proc):
  +501              "with open('%s', 'wb') as f:" % support.TESTFN,
  +502                  while True:  # synchronize the subprocess
  +503                      dt = time.monotonic() - start_time
  +504                      if dt > 60.0:
  +505                          raise Exception("failed to sync child in
%.1f sec" % dt)
  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except BlockingIOError:
  +511                          break

--- run eintr_tester.py ---
  File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 502
    while True:  # synchronize the subprocess
    ^
IndentationError: unexpected indent
--- eintr_tester.py completed: exit code 1 ---

>
> ----------
> nosy: +vstinner
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>

Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?

diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py
index 25c169bde5..4db5dc9045 100644
--- a/Lib/test/eintrdata/eintr_tester.py
+++ b/Lib/test/eintrdata/eintr_tester.py
@@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest):
         self.addCleanup(support.unlink, support.TESTFN)
         code = '\n'.join((
             "import fcntl, time",
-            "with open('%s', 'wb') as f:" % support.TESTFN,
+            "with open('%s', 'w+b') as f:" % support.TESTFN,
             "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
             "   time.sleep(%s)" % self.sleep_time))
         start_time = time.monotonic()
         proc = self.subprocess(code)
         with kill_on_error(proc):
-            with open(support.TESTFN, 'wb') as f:
+            with open(support.TESTFN, 'w+b') as f:
                 while True:  # synchronize the subprocess
                     dt = time.monotonic() - start_time
                     if dt > 60.0:

On 07/01/2019 15:46, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?

I was clearly a bit blind.. Now with:

  +490  class FNTLEINTRTest(EINTRBaseTest):
  +491      def _lock(self, lock_func, lock_name):
  +492          self.addCleanup(support.unlink, support.TESTFN)
  +493          code = '\n'.join((
  +494              "import fcntl, time",
  +495              "with open('%s', 'w+b') as f:" % support.TESTFN,
  +496              "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
  +497              "   time.sleep(%s)" % self.sleep_time))
  +498          start_time = time.monotonic()
  +499          proc = self.subprocess(code)
  +500          with kill_on_error(proc):
  +501              with open(support.TESTFN, 'w+b') as f:
  +502                  while True:  # synchronize the subprocess
  +503                      dt = time.monotonic() - start_time
  +504                      if dt > 60.0:
  +505                          raise Exception("failed to sync child in
%.1f sec" % dt)
  +506                      try:
  +507                          lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
  +508                          lock_func(f, fcntl.LOCK_UN)
  +509                          time.sleep(0.01)
  +510                      except BlockingIOError:
  +511                          break

I get - PermissionError - as before.

root@x066:[/data/prj/python/python3-3.8]./python -m test -v test_eintr
== CPython 3.8.0a0 (heads/master-dirty:b4ea8bb080, Jan 1 2019, 18:24:36) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/python3-3.8/build/test_python_3342514
== CPU count: 8
== encodings: locale=ISO8859-1, FS=iso8859-1
Run tests sequentially
0:00:00 [1/1] test_eintr
test_all (test.test_eintr.EINTRTests) ...
--- run eintr_tester.py ---
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ERROR
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... skipped 'need select.epoll'
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

======================================================================
ERROR: test_lockf (__main__.FNTLEINTRTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 522, in test_lockf
    self._lock(fcntl.lockf, "lockf")
  File
"/data/prj/python/git/python3-3.8/Lib/test/eintrdata/eintr_tester.py",
line 507, in _lock
    lock_func(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
PermissionError: [Errno 13] Permission denied

----------------------------------------------------------------------
Ran 24 tests in 9.098s

FAILED (errors=1, skipped=3)
--- eintr_tester.py completed: exit code 1 ---
FAIL

======================================================================
FAIL: test_all (test.test_eintr.EINTRTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/python/git/python3-3.8/Lib/test/test_eintr.py", line
31, in test_all
    self.fail("eintr_tester.py failed")
AssertionError: eintr_tester.py failed

----------------------------------------------------------------------

Ran 1 test in 9.674s

FAILED (failures=1)
test test_eintr failed
test_eintr failed

== Tests result: FAILURE ==

>
> diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py
> index 25c169bde5..4db5dc9045 100644
> --- a/Lib/test/eintrdata/eintr_tester.py
> +++ b/Lib/test/eintrdata/eintr_tester.py
> @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest):
>          self.addCleanup(support.unlink, support.TESTFN)
>          code = '\n'.join((
>              "import fcntl, time",
> -            "with open('%s', 'wb') as f:" % support.TESTFN,
> +            "with open('%s', 'w+b') as f:" % support.TESTFN,
>              "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
>              "   time.sleep(%s)" % self.sleep_time))
>          start_time = time.monotonic()
>          proc = self.subprocess(code)
>          with kill_on_error(proc):
> -            with open(support.TESTFN, 'wb') as f:
> +            with open(support.TESTFN, 'w+b') as f:
>                  while True:  # synchronize the subprocess
>                      dt = time.monotonic() - start_time
>                      if dt > 60.0:
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>

On 07/01/2019 15:46, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Since you are getting indentation error, I'm not sure about your test. Can you please apply the patch below and run again test_eintr? Does it still fail with PermissionError?
Answer - Yes. Still get PermissionError
>
> diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py
> index 25c169bde5..4db5dc9045 100644
> --- a/Lib/test/eintrdata/eintr_tester.py
> +++ b/Lib/test/eintrdata/eintr_tester.py
> @@ -492,13 +492,13 @@ class FNTLEINTRTest(EINTRBaseTest):
>          self.addCleanup(support.unlink, support.TESTFN)
>          code = '\n'.join((
>              "import fcntl, time",
> -            "with open('%s', 'wb') as f:" % support.TESTFN,
> +            "with open('%s', 'w+b') as f:" % support.TESTFN,
>              "   fcntl.%s(f, fcntl.LOCK_EX)" % lock_name,
>              "   time.sleep(%s)" % self.sleep_time))
>          start_time = time.monotonic()
>          proc = self.subprocess(code)
>          with kill_on_error(proc):
> -            with open(support.TESTFN, 'wb') as f:
> +            with open(support.TESTFN, 'w+b') as f:
>                  while True:  # synchronize the subprocess
>                      dt = time.monotonic() - start_time
>                      if dt > 60.0:
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>

> On AIX the test for flock() passes, but the test for lockf() fails: (...)

I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?

There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.

I’ll make a new PR and delete the news entry. 

Sent from my iPhone

> On 15 Jan 2019, at 11:23, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@redhat.com> added the comment:
> 
>> On AIX the test for flock() passes, but the test for lockf() fails: (...)
> 
> I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?
> 
> There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>

Done.

> On 1/15/2019 11:23 AM, STINNER Victor wrote:
> I would prefer to simply skip the lockf() test rather than ignoring PermissionError for flock() and lockf() on all platforms. Can you please write a PR for that?
> 
> There is no need to add a NEWS entry, I will add "skip news". If you want to add a NEWS entry, mention at least the fixed test and AIX.

New changeset b94d4bed67c60834b1a5a0936b8c13934cf3b872 by Miss Islington (bot) (Michael Felt) in branch 'master':
bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424)
https://github.com/python/cpython/commit/b94d4bed67c60834b1a5a0936b8c13934cf3b872

Thanks all!

New changeset 7e618f3154404613cd9774fb8ffb5b48e7233a9e by Miss Islington (bot) in branch '3.7':
bpo-35633: test_lockf() fails with "PermissionError: [Errno 13] Permission denied" on AIX (GH-11424)
https://github.com/python/cpython/commit/7e618f3154404613cd9774fb8ffb5b48e7233a9e

Could this also be backported to version 3.6?

On 10/03/2019 19:34, Michael Felt wrote:
> Michael Felt <aixtools@felt.demon.nl> added the comment:
>
> Could this also be backported to version 3.6?
>
> ----------
> versions: +Python 3.6
Not to worry: As 3.6 is in security mode.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35633>
> _______________________________________
>

FYI, the problem here is that AIX fcntl returns EACCES in the case that the lock is held and non-blocking behavior was requested:


> The lockfx and lockf subroutines fail if one of the following is true:
Item
> 
> EACCES 	The Command parameter is F_SETLK, the l_type field is F_RDLCK, and the segment of the file to be locked is already write-locked by another process.
> EACCES 	The Command parameter is F_SETLK, the l_type field is F_WRLCK, and the segment of a file to be locked is already read-locked or write-locked by another process.

https://www.ibm.com/docs/en/aix/7.1?topic=l-lockfx-lockf-flock-lockf64-subroutine

(Note the docs are a bit wonky referring to lockf/lockfx but talking about parameters and fields which apply to fcntl instead)

The lockf/flock APIs provided by AIX handle this appropriately, mapping EACCES to EWOULDBLOCK, but while Python calls the libbsd flock API, it uses its own lockf implementation which calls fcntl directly: https://github.com/python/cpython/blob/main/Modules/fcntlmodule.c#L426