classification
Title: Running python test suite fails on macOS 10.14.4 with resource.RLIMIT_STACK error
Type: behavior Stage: resolved
Components: macOS Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: python3 resource.setrlimit strange behaviour under macOS
View: 34602
Assigned To: ned.deily Nosy List: berker.peksag, dimpase, inada.naoki, ned.deily, remi.lapeyre, ronaldoussoren, yan12125
Priority: critical Keywords:

Created on 2019-03-26 07:09 by ned.deily, last changed 2019-07-02 07:57 by ned.deily. This issue is now closed.

Messages (9)
msg338858 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-26 07:09
After upgrading my first macOS system to the newly released macOS 10.14.4 update, attempts to run the Python test suite via regrtest fail:

$ /usr/local/bin/python3.7 -m test -uall -j3 -w
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/__main__.py", line 2, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 640, in main
    Regrtest().main(tests=tests, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 586, in main
    self._main(tests, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 607, in _main
    setup_tests(self.ns)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/setup.py", line 77, in setup_tests
    resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
ValueError: current limit exceeds maximum limit

The error is during regrtest initialization when it is trying to increase the process stack size to avoid previously seen problems when running tests.  This code has been unchanged for a long time.

Stepping through the code in the REPL on 10.14.4:
>>> import resource
>>> resource.getrlimit(resource.RLIMIT_STACK)
(8388608, 67104768)
>>> soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
>>> newsoft = min(hard, max(soft, 1024*2048))
>>> newsoft
8388608
>>> resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: current limit exceeds maximum limit

The same code run on a macOS system still running 10.14.3 gives the same values from getrlimit but does not fail when calling setrlimit.

I will investigate further tomorrow; if this is a general problem with macOS 10.14.4, we'll need to provide a workaround and possibly open an incident report with Apple.
msg338862 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-03-26 08:41
My mac is also 10.14.4.  I don't have older macOS now.  Could someone test this?

$ ulimit -Sa
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4864
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1418
virtual memory          (kbytes, -v) unlimited

$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) unlimited
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65532
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2128
virtual memory          (kbytes, -v) unlimited
msg338865 - (view) Author: Kumar Akshay (kakshay) * Date: 2019-03-26 09:25
I'm on macOS 10.14.3 and running those commands gives me this ->

~ ulimit -Sa
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                4864
➜  ~ ulimit -Ha
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             65532
-c: core file size (blocks)         unlimited
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1064
-n: file descriptors                unlimited
msg338867 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-03-26 09:29
I created simple program calling setrlimit and it succeeds.
I confirmed setrlimit argument is exactly same.
It's very curious why same Python code fails...


== c code
#include <sys/resource.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
    struct rlimit rl;
    int err;

    err = getrlimit(RLIMIT_STACK, &rl);
    if (err < 0) {
        perror("getrlimit");
        return err;
    }

    printf("%d, soft=%llu, hard=%llu\n", RLIMIT_STACK, rl.rlim_cur, rl.rlim_max);

    err = setrlimit(RLIMIT_STACK, &rl);
    if (err < 0) {
        perror("setrlimit");
        return err;
    }

    return 0;
}

== Python code
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
print("limits=", soft, hard)
resource.setrlimit(resource.RLIMIT_STACK, (soft, hard))

== fails
Traceback (most recent call last):
  File "x.py", line 4, in <module>
    resource.setrlimit(resource.RLIMIT_STACK, (soft, hard))
ValueError: current limit exceeds maximum limit
msg338941 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-03-27 06:39
I think this issue is duplicate of https://bugs.python.org/issue34602
msg339307 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-04-01 22:58
I confirm that reverting 335ab5b66f432ae3713840ed2403a11c368f5406 work as a workaround, and does not seem to create regressions.
msg340369 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2019-04-16 22:29
I can also confirm that reverting 335ab5b66f432ae3713840ed2403a11c368f5406 fixes the problem.
msg341116 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-04-29 19:53
Thanks, Inada-san, for the pointer.  Closing this as a duplicate of Issue34602
msg347120 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-07-02 07:57
See updated Issue34602 discussion for why reverting 335ab5b66f432ae3713840ed2403a11c368f5406 causes other problems and for a different fix for this issue.
History
Date User Action Args
2019-07-02 07:57:03ned.deilysetmessages: + msg347120
2019-04-29 19:53:25ned.deilysetstatus: open -> closed
superseder: python3 resource.setrlimit strange behaviour under macOS
messages: + msg341116

resolution: duplicate
stage: test needed -> resolved
2019-04-16 22:29:30berker.peksagsettype: behavior

messages: + msg340369
nosy: + berker.peksag
2019-04-11 20:08:39dimpasesetnosy: + dimpase
2019-04-01 22:58:40remi.lapeyresetmessages: + msg339307
2019-04-01 22:23:36remi.lapeyresetnosy: + remi.lapeyre
2019-03-30 13:50:43yan12125setnosy: + yan12125
2019-03-27 06:39:07inada.naokisetmessages: + msg338941
2019-03-26 09:29:25inada.naokisetnosy: - kakshay
messages: + msg338867
2019-03-26 09:25:43kakshaysetnosy: + kakshay
messages: + msg338865
2019-03-26 08:41:38inada.naokisetnosy: + inada.naoki
messages: + msg338862
2019-03-26 07:09:11ned.deilycreate