Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running python test suite fails on macOS 10.14.4 with resource.RLIMIT_STACK error #80613

Closed
ned-deily opened this issue Mar 26, 2019 · 9 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes OS-mac type-bug An unexpected behavior, bug, or error

Comments

@ned-deily
Copy link
Member

BPO 36432
Nosy @ronaldoussoren, @ned-deily, @methane, @berkerpeksag, @yan12125, @remilapeyre, @dimpase
Superseder
  • bpo-34602: python3 resource.setrlimit strange behaviour under macOS
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ned-deily'
    closed_at = <Date 2019-04-29.19:53:25.019>
    created_at = <Date 2019-03-26.07:09:11.590>
    labels = ['OS-mac', '3.8', 'type-bug', '3.7']
    title = 'Running python test suite fails on macOS 10.14.4 with resource.RLIMIT_STACK error'
    updated_at = <Date 2019-07-02.07:57:03.169>
    user = 'https://github.com/ned-deily'

    bugs.python.org fields:

    activity = <Date 2019-07-02.07:57:03.169>
    actor = 'ned.deily'
    assignee = 'ned.deily'
    closed = True
    closed_date = <Date 2019-04-29.19:53:25.019>
    closer = 'ned.deily'
    components = ['macOS']
    creation = <Date 2019-03-26.07:09:11.590>
    creator = 'ned.deily'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36432
    keywords = []
    message_count = 9.0
    messages = ['338858', '338862', '338865', '338867', '338941', '339307', '340369', '341116', '347120']
    nosy_count = 7.0
    nosy_names = ['ronaldoussoren', 'ned.deily', 'methane', 'berker.peksag', 'yan12125', 'remi.lapeyre', 'dimpase']
    pr_nums = []
    priority = 'critical'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '34602'
    type = 'behavior'
    url = 'https://bugs.python.org/issue36432'
    versions = ['Python 3.7', 'Python 3.8']

    @ned-deily
    Copy link
    Member Author

    After upgrading my first macOS system to the newly released macOS 10.14.4 update, attempts to run the Python test suite via regrtest fail:

    $ /usr/local/bin/python3.7 -m test -uall -j3 -w
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/__main__.py", line 2, in <module>
        main()
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 640, in main
        Regrtest().main(tests=tests, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 586, in main
        self._main(tests, kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/main.py", line 607, in _main
        setup_tests(self.ns)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/test/libregrtest/setup.py", line 77, in setup_tests
        resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
    ValueError: current limit exceeds maximum limit

    The error is during regrtest initialization when it is trying to increase the process stack size to avoid previously seen problems when running tests. This code has been unchanged for a long time.

    Stepping through the code in the REPL on 10.14.4:
    >>> import resource
    >>> resource.getrlimit(resource.RLIMIT_STACK)
    (8388608, 67104768)
    >>> soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
    >>> newsoft = min(hard, max(soft, 1024*2048))
    >>> newsoft
    8388608
    >>> resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: current limit exceeds maximum limit

    The same code run on a macOS system still running 10.14.3 gives the same values from getrlimit but does not fail when calling setrlimit.

    I will investigate further tomorrow; if this is a general problem with macOS 10.14.4, we'll need to provide a workaround and possibly open an incident report with Apple.

    @ned-deily ned-deily added 3.7 (EOL) end of life 3.8 only security fixes labels Mar 26, 2019
    @ned-deily ned-deily self-assigned this Mar 26, 2019
    @methane
    Copy link
    Member

    methane commented Mar 26, 2019

    My mac is also 10.14.4. I don't have older macOS now. Could someone test this?

    $ ulimit -Sa
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    file size               (blocks, -f) unlimited
    max locked memory       (kbytes, -l) unlimited
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 4864
    pipe size            (512 bytes, -p) 1
    stack size              (kbytes, -s) 8192
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1418
    virtual memory          (kbytes, -v) unlimited
    
    $ ulimit -Ha
    core file size          (blocks, -c) unlimited
    data seg size           (kbytes, -d) unlimited
    file size               (blocks, -f) unlimited
    max locked memory       (kbytes, -l) unlimited
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) unlimited
    pipe size            (512 bytes, -p) 1
    stack size              (kbytes, -s) 65532
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 2128
    virtual memory          (kbytes, -v) unlimited

    @kakshay21
    Copy link
    Mannequin

    kakshay21 mannequin commented Mar 26, 2019

    I'm on macOS 10.14.3 and running those commands gives me this ->

    ~ ulimit -Sa
    -t: cpu time (seconds) unlimited
    -f: file size (blocks) unlimited
    -d: data seg size (kbytes) unlimited
    -s: stack size (kbytes) 8192
    -c: core file size (blocks) 0
    -v: address space (kbytes) unlimited
    -l: locked-in-memory size (kbytes) unlimited
    -u: processes 709
    -n: file descriptors 4864
    ➜ ~ ulimit -Ha
    -t: cpu time (seconds) unlimited
    -f: file size (blocks) unlimited
    -d: data seg size (kbytes) unlimited
    -s: stack size (kbytes) 65532
    -c: core file size (blocks) unlimited
    -v: address space (kbytes) unlimited
    -l: locked-in-memory size (kbytes) unlimited
    -u: processes 1064
    -n: file descriptors unlimited

    @methane
    Copy link
    Member

    methane commented Mar 26, 2019

    I created simple program calling setrlimit and it succeeds.
    I confirmed setrlimit argument is exactly same.
    It's very curious why same Python code fails...

    == c code
    #include <sys/resource.h>
    #include <stdio.h>
    
    int
    main(int argc, char *argv[])
    {
        struct rlimit rl;
        int err;
    
        err = getrlimit(RLIMIT_STACK, &rl);
        if (err < 0) {
            perror("getrlimit");
            return err;
        }
    
        printf("%d, soft=%llu, hard=%llu\n", RLIMIT_STACK, rl.rlim_cur, rl.rlim_max);
    
        err = setrlimit(RLIMIT_STACK, &rl);
        if (err < 0) {
            perror("setrlimit");
            return err;
        }
    
        return 0;
    }

    == Python code
    import resource
    soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
    print("limits=", soft, hard)
    resource.setrlimit(resource.RLIMIT_STACK, (soft, hard))

    == fails
    Traceback (most recent call last):
      File "x.py", line 4, in <module>
        resource.setrlimit(resource.RLIMIT_STACK, (soft, hard))
    ValueError: current limit exceeds maximum limit

    @methane
    Copy link
    Member

    methane commented Mar 27, 2019

    I think this issue is duplicate of https://bugs.python.org/issue34602

    @remilapeyre
    Copy link
    Mannequin

    remilapeyre mannequin commented Apr 1, 2019

    I confirm that reverting 335ab5b work as a workaround, and does not seem to create regressions.

    @berkerpeksag
    Copy link
    Member

    I can also confirm that reverting 335ab5b fixes the problem.

    @berkerpeksag berkerpeksag added the type-bug An unexpected behavior, bug, or error label Apr 16, 2019
    @ned-deily
    Copy link
    Member Author

    Thanks, Inada-san, for the pointer. Closing this as a duplicate of bpo-34602

    @ned-deily
    Copy link
    Member Author

    See updated bpo-34602 discussion for why reverting 335ab5b causes other problems and for a different fix for this issue.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes OS-mac type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants