This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Large C stack usage of os.getgroups() and os.setgroups()
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: corona10, methane, vstinner
Priority: normal Keywords: patch

Created on 2022-02-02 03:06 by methane, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 31073 merged methane, 2022-02-02 04:54
PR 31561 merged methane, 2022-02-25 04:07
PR 31569 merged vstinner, 2022-02-25 11:30
Messages (6)
msg412335 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2022-02-02 03:06
I checked stack usage for bpo-46600 and found this two functions use a lot of stack.

os_setgroups: 262200 bytes
os_getgroups_impl: 262184 bytes

Both function has local variable like this:

    gid_t grouplist[MAX_GROUPS];

MAX_GROUPS is defined as:

```
#ifdef NGROUPS_MAX
#define MAX_GROUPS NGROUPS_MAX
#else
    /* defined to be 16 on Solaris7, so this should be a small number */
#define MAX_GROUPS 64
#endif
```

NGROUPS_MAX is 65536 and sizeof(gid_t) is 4 on Ubuntu 20.04, so grouplist is 262144bytes.

It seems this grouplist is just for avoid allocation:

```
    } else if (n <= MAX_GROUPS) {
        /* groups will fit in existing array */
        alt_grouplist = grouplist;
    } else {
        alt_grouplist = PyMem_New(gid_t, n);
        if (alt_grouplist == NULL) {
            return PyErr_NoMemory();
        }
```

How about just using `#define MAX_GROUPS 64`?
Or should we remove this grouplist because os.grouplist() is not called so frequently?
msg413689 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2022-02-22 02:59
New changeset 74127b89a8224d021fc76f679422b76510844ff9 by Inada Naoki in branch 'main':
bpo-46606: Reduce stack usage of getgroups and setgroups (GH-31073)
https://github.com/python/cpython/commit/74127b89a8224d021fc76f679422b76510844ff9
msg413864 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-23 22:53
>  n++; // Avoid malloc(0)
> grouplist = PyMem_New(gid_t, n+1);

FYI PyMem_New(0) is well specified and doesn't return NULL:
https://docs.python.org/dev/c-api/memory.html#c.PyMem_Malloc
msg413963 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2022-02-25 05:13
New changeset ad6c7003e38a9f8bdf8d865fb5fa0f3c03690315 by Inada Naoki in branch 'main':
bpo-46606: Remove redundant +1. (GH-31561)
https://github.com/python/cpython/commit/ad6c7003e38a9f8bdf8d865fb5fa0f3c03690315
msg414116 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-26 23:14
New changeset e02c47528b31f513d5f5d6eb91b8c9714134cea2 by Victor Stinner in branch 'main':
bpo-46606: os.getgroups() doesn't overallocate (GH-31569)
https://github.com/python/cpython/commit/e02c47528b31f513d5f5d6eb91b8c9714134cea2
msg414117 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-26 23:16
> NGROUPS_MAX is 65536 and sizeof(gid_t) is 4 on Ubuntu 20.04, so grouplist is 262144bytes.

Oops, that's a lot! Nicely spotted! Yeah, it's perfectly fine to allocate a temporary array on the heap memory. There is no need to micro-optimize this function.

Maybe the code was written when NGROUPS_MAX was way smaller (64?).
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90764
2022-02-26 23:16:35vstinnersetmessages: + msg414117
2022-02-26 23:14:35vstinnersetmessages: + msg414116
2022-02-25 11:30:18vstinnersetpull_requests: + pull_request29691
2022-02-25 05:13:48methanesetmessages: + msg413963
2022-02-25 04:07:51methanesetpull_requests: + pull_request29684
2022-02-23 22:53:16vstinnersetnosy: + vstinner
messages: + msg413864
2022-02-22 02:59:43methanesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2022-02-22 02:59:35methanesetmessages: + msg413689
2022-02-02 05:16:27corona10setnosy: + corona10
2022-02-02 04:54:36methanesetkeywords: + patch
stage: patch review
pull_requests: + pull_request29257
2022-02-02 03:06:44methanecreate