Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a new helper function that enable to test that an operation don't hang more than a given timeout. #56619

Closed
mouad mannequin opened this issue Jun 25, 2011 · 6 comments
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@mouad
Copy link
Mannequin

mouad mannequin commented Jun 25, 2011

BPO 12410
Nosy @vstinner
Files
  • operation_timeout.patch: Add a helper method to make sure that an operation will not last more than a given timeout.
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2012-02-05.15:31:10.352>
    created_at = <Date 2011-06-25.14:49:35.433>
    labels = ['type-feature', 'tests']
    title = "Create a new helper function that enable to test that an operation don't hang more than a given timeout."
    updated_at = <Date 2012-02-05.15:31:10.351>
    user = 'https://bugs.python.org/mouad'

    bugs.python.org fields:

    activity = <Date 2012-02-05.15:31:10.351>
    actor = 'neologix'
    assignee = 'none'
    closed = True
    closed_date = <Date 2012-02-05.15:31:10.352>
    closer = 'neologix'
    components = ['Tests']
    creation = <Date 2011-06-25.14:49:35.433>
    creator = 'mouad'
    dependencies = []
    files = ['22469']
    hgrepos = []
    issue_num = 12410
    keywords = ['patch']
    message_count = 6.0
    messages = ['139075', '139091', '139099', '139123', '139127', '152692']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'neologix', 'mouad']
    pr_nums = []
    priority = 'normal'
    resolution = 'rejected'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue12410'
    versions = ['Python 3.2']

    @mouad
    Copy link
    Mannequin Author

    mouad mannequin commented Jun 25, 2011

    While working on issue bpo-12157 [http://bugs.python.org/issue12157], I needed a function that make sure that an operation will not hang forever, for this reason i have create this helper function that support the context manager protocol and accept a timeout as an argument and raise an IOError if the operation didn't terminate before that timeout.

    @mouad mouad mannequin added tests Tests in the Lib/test dir type-feature A feature request or enhancement labels Jun 25, 2011
    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Jun 25, 2011

    It's a little bit more complicated than that:

    • signals and threads don't mix well together
    • this will make syscalls fail with EINTR
    • the old SIGALRM handler is lost
    • etc

    In short, don't use signals.
    I'm not sure there's a reliable way to write such a general-purpose wrapper (usually one can use select() with a timeout or spawn a subprocess and use communicate's timeout to achieve this kind of things).
    In your use case (issue bpo-12157), I think that letting the test block is fine, since:

    • there's no easy way to add a timeout (but you could spawn a new interpreter and use communicate with a timeout if you really wanted to)
    • it will be caught by the faulthandler module
    • a test killed by faulthandler's timeout is more interesting to fix that a "common" failed test ;-)

    @mouad
    Copy link
    Mannequin Author

    mouad mannequin commented Jun 25, 2011

    Thanks for the instructive feedback :)

    I totally agree i guess there is a lot of issues that i didn't think of :-(, my first thinking was to use "Pool.join" timeout argument but it was removed in 3.2 (by the way i didn't find the issue or the rational that lead to this change).

    And now that i know about "faulthandler" module i guess that will make also my life easier :), i will rewrite the patch in the issue bpo-12157 to not use any *fancy* way to check if it will hang.

    cheers,

    @vstinner
    Copy link
    Member

    alarm() is one possible implementation, but Charles-François listed some drawbacks.

    You can also use resource.setrlimit(RLIMIT_CPU), but the timeout is the CPU time (e.g. you cannot stop a sleep) and it is not portable (e.g. resource is not available on Windows).

    Another possible implementation is a thread. faulthandler uses an "hidden" thread (implemented in C): a thread ignoring all signals using pthread_sigmask. Python threads are not reliable for a timeout because of the GIL, and it is not easy to "interrupt" another thread from the "timeout" thread. For example, you cannot (easily) raise an exception in another thread.

    I'm not sure there's a reliable way to write such a general-purpose
    wrapper

    I agree, but it doesn't mean that it is not possible :-)

    I think that you should try to implement in C a thread ignoring all signals. It becomes more complex when you have to implement the "interrupt the current thread" (current thread, or maybe the thread using the operation_timeout context manager?) part.

    I suppose that you will have to use low-level "tricks" and you will have to experiment your tool on different platform.

    You should start this project outside CPython (as a third party module), and then ask for an integration when your work is well tested. You have to know that a module "dies" when it enters CPython: you have to wait something like 18 months to modify it, so you have to be sure that your code is "correct" ;-)

    @vstinner
    Copy link
    Member

    Oh, there is another possible implementation: use a subprocess. But if the timeout is implemented using a subprocess, the syntax cannot be:

    with timeout(5):
       do_something()

    It should be something like:

    timeout(5, """if 1:
       import os, sys
       ...
       do_something()
       ...
       sys.exit(0)
    """)

    Some tests are already doing that manually.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Feb 5, 2012

    Closing, since it's hard to write correctly, and apparently not that useful.

    @neologix neologix mannequin closed this as completed Feb 5, 2012
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    tests Tests in the Lib/test dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant