Title: test_posix fails ERRNO 0
Type: Stage:
Components: ctypes, macOS, Tests Versions: Python 3.8, Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, nitishch, ronaldoussoren, wizofe
Priority: normal Keywords:

Created on 2018-04-04 14:13 by wizofe, last changed 2019-06-13 07:49 by ned.deily.

Messages (5)
msg314934 - (view) Author: Ioannis Valasakis (wizofe) Date: 2018-04-04 14:13
I am on mac OSX high Sierra 10.13.4. I am using the Apple LLVM version 9.1.0 (clang-902.0.37.1) from the XCode toolchain. 

My tests fail on posix and if run the interpreter I get

--- pylog ---
Python 3.8.0a0 (heads/master:da58533ac6, Apr  3 2018, 16:54:53)
[Clang 9.1.0 (clang-902.0.37.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getgrouplist('wizofe', 20)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 0] Error
--- pylog ---

My system is defined as case-sensitive in the setup.
msg314937 - (view) Author: Ioannis Valasakis (wizofe) Date: 2018-04-04 15:24
Note: I get the same error with the version of python installed via homebrew:

(Python 3.6.4 (default, Jan  6 2018, 11:51:59)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin

Here is also the output of the id command, if that is helpful:

(box3) λ /Volumes/pythontesting/cpython/ master id
uid=501(wizofe) gid=20(staff) groups=20(staff),1(daemon),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),204(_developer),450(_xcs),701(1),33(_appstore),100(_lpoperator),250(_analyticsusers),395(,398(,399(

I am also attaching a dtrace log of the two liner that causes the bug.
msg322576 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-28 16:20
I get the same error on High Sierra, but not on the Mojave beta.  I also regularly get a FileNotFound error instead (which is why I ended up at this issue in the first place).

I'm not sure yet what causes this, a small reproducer in C works correctly and the code in posixmodule.c looks correct. This might be a platform bug (hence my attempt at writing a reproducer in C).

BTW. There is no attached file
msg322577 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-28 16:46
I'm slightly closer to understanding the problem. The value of errno is that of the last error *before* getgrouplist(3) was called, that is, getgrouplist return -1 but does not update errno.

And looking at Apple source code this is a bug in CPython after all: getgrouplist(3) return -1 when the buffer is too small, but does not set errno (see <>). The manpage also doesn't mention setting errno.

I'm pretty sure that the use of posix_error() is wrong here. 

We're running into a similar issue as with getgroups: on macOS a user can be a member of more than NGROUPS_MAX groups.  Because of this I'm not yet sure of the correct fix for this issue.

The easy part is that the call to posix_error in the implementation of getgrouplist should be replaced by raising some other error (or possibly just setting errno to EOVERFLOW before calling posix_error()).

The harder part is what to do about the definition of MAX_GROUPS for this function. Either force it to a larger default on macOS, or add some code to increase the buffer size (because the user cannot select the size of buffer to use). The latter is annoyingly hard because the system gives no indication on what the correct buffer size should be.

BTW. The same is also true on other platforms with getgrouplist(), this is not a macOS specific issue other than that NGROUPS_MAX is bogus there.
msg345471 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-13 07:49
See also the discussion and fix merged in Issue35070.
Date User Action Args
2019-06-13 07:49:03ned.deilysetmessages: + msg345471
2018-07-28 16:46:22ronaldoussorensetmessages: + msg322577
2018-07-28 16:20:49ronaldoussorensetmessages: + msg322576
versions: + Python 3.7
2018-04-05 05:01:18nitishchsetnosy: + nitishch
2018-04-04 15:24:16wizofesetmessages: + msg314937
2018-04-04 14:13:39wizofecreate