Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imap from ThreadPool hangs by an exception in a generator function #72882

Closed
lev-veshnyakov mannequin opened this issue Nov 15, 2016 · 10 comments
Closed

imap from ThreadPool hangs by an exception in a generator function #72882

lev-veshnyakov mannequin opened this issue Nov 15, 2016 · 10 comments
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@lev-veshnyakov
Copy link
Mannequin

lev-veshnyakov mannequin commented Nov 15, 2016

BPO 28696
Nosy @applio, @zhangyangyu, @lev-veshnyakov
Superseder
  • bpo-28699: Imap from ThreadPool behaves unexpectedly
  • Files
  • test.py: Example of code leading to the issue
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2016-11-25.22:06:42.835>
    created_at = <Date 2016-11-15.12:52:50.751>
    labels = ['type-bug', 'library']
    title = 'imap from ThreadPool hangs by an exception in a generator function'
    updated_at = <Date 2016-11-25.22:06:42.833>
    user = 'https://github.com/lev-veshnyakov'

    bugs.python.org fields:

    activity = <Date 2016-11-25.22:06:42.833>
    actor = 'davin'
    assignee = 'none'
    closed = True
    closed_date = <Date 2016-11-25.22:06:42.835>
    closer = 'davin'
    components = ['Library (Lib)']
    creation = <Date 2016-11-15.12:52:50.751>
    creator = 'lev-veshnyakov'
    dependencies = []
    files = ['45486']
    hgrepos = []
    issue_num = 28696
    keywords = []
    message_count = 10.0
    messages = ['280833', '280838', '280840', '280842', '280863', '280866', '280867', '280868', '281032', '281743']
    nosy_count = 3.0
    nosy_names = ['davin', 'xiang.zhang', 'lev-veshnyakov']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '28699'
    type = 'behavior'
    url = 'https://bugs.python.org/issue28696'
    versions = ['Python 3.4']

    @lev-veshnyakov
    Copy link
    Mannequin Author

    lev-veshnyakov mannequin commented Nov 15, 2016

    It's only in imap, in map it's ok. The following code explains the issue:

    from multiprocessing.pool import ThreadPool
    
    pool = ThreadPool(10)
    
    def gen():
        yield 1 + '1' # here is an error

    try:
    next((pool.imap(str, gen())))
    except:
    # Will be catched using pool.map normally, but using pool.imap will be not.
    # Instead it hangs. This is the same for ThreadPool and Pool.
    print('this will not be printed because thread is hanging')

    @lev-veshnyakov lev-veshnyakov mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Nov 15, 2016
    @applio
    Copy link
    Member

    applio commented Nov 15, 2016

    Using the supplied example, I was unable to reproduce what you described using either 3.5 or 3.6-beta on OS X 10.11.

    What platform are you using? (Perhaps it is platform specific.)

    @lev-veshnyakov
    Copy link
    Mannequin Author

    lev-veshnyakov mannequin commented Nov 15, 2016

    Ubuntu 14.04 LTS, 3.13.0-83-generic, x86_64

    @lev-veshnyakov
    Copy link
    Mannequin Author

    lev-veshnyakov mannequin commented Nov 15, 2016

    So, I've checked twice, it's presented by me on python 3.4.3, and not by 3.5.2. So I will go deaper

    @lev-veshnyakov
    Copy link
    Mannequin Author

    lev-veshnyakov mannequin commented Nov 15, 2016

    I've reproduced it on 2 different machines:

    • on a MacBook in Docker (debian:jessie, python 3.4.2)
    • on another desktop (Ubuntu 14.04.1, 3.16.0-77-generic, x86_64, python 3.4.3)

    @lev-veshnyakov
    Copy link
    Mannequin Author

    lev-veshnyakov mannequin commented Nov 15, 2016

    It's hanging in a while loop in _handle_workers, /usr/lib/python3.4/multiprocessingpool.py:365.
    I can't figure out what is the reason.

    @staticmethod
    def _handle_workers(pool):
        thread = threading.current_thread()
    
        # Keep maintaining workers until the cache gets drained, unless the pool
        # is terminated.
        while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
            pool._maintain_pool()
            time.sleep(0.1)
        # send sentinel to stop workers
        pool._taskqueue.put(None)
        util.debug('worker handler exiting')

    @applio
    Copy link
    Member

    applio commented Nov 15, 2016

    If it only occurs in 3.4.x, is moving to 3.5 an option for you?

    @lev-veshnyakov
    Copy link
    Mannequin Author

    lev-veshnyakov mannequin commented Nov 15, 2016

    Yes, I'm free to move to 3.5, now I'm seeing isn't there any problems in 3.5 according to this issue.

    @zhangyangyu
    Copy link
    Member

    In Py3.6, it raises error:

    >>> next((pool.imap(str, gen())))
    Traceback (most recent call last):
      File "/opt/lib/python3.7/multiprocessing/pool.py", line 684, in next
        item = self._items.popleft()
    IndexError: pop from an empty deque
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/lib/python3.7/multiprocessing/pool.py", line 690, in next
        item = self._items.popleft()
    IndexError: pop from an empty deque
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/opt/lib/python3.7/multiprocessing/pool.py", line 693, in next
        raise StopIteration
    StopIteration

    But this doesn't looks good. I think it should raise TypeError. It can be fixed in bpo-28699.

    @applio
    Copy link
    Member

    applio commented Nov 25, 2016

    Closing this issue -- all further discussion moves to bpo-28699

    @applio applio closed this as completed Nov 25, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants