Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing.Pool from concurrent threads failure on 3.9.0rc1 #85739

Closed
drougge mannequin opened this issue Aug 17, 2020 · 6 comments
Closed

multiprocessing.Pool from concurrent threads failure on 3.9.0rc1 #85739

drougge mannequin opened this issue Aug 17, 2020 · 6 comments
Labels
3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@drougge
Copy link
Mannequin

drougge mannequin commented Aug 17, 2020

BPO 41567
Nosy @atsuoishimoto, @drougge, @aeros, @iritkatriel, @cebtenzzre
Superseder
  • bpo-43517: Fix false positives in circular import detection with from-imports
  • Files
  • pool_error_on_3.9.py: repro example
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-03-16.16:16:54.899>
    created_at = <Date 2020-08-17.11:37:30.505>
    labels = ['type-bug', 'library', '3.9', '3.10']
    title = 'multiprocessing.Pool from concurrent threads failure on 3.9.0rc1'
    updated_at = <Date 2021-03-16.16:16:54.899>
    user = 'https://github.com/drougge'

    bugs.python.org fields:

    activity = <Date 2021-03-16.16:16:54.899>
    actor = 'pitrou'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-03-16.16:16:54.899>
    closer = 'pitrou'
    components = ['Library (Lib)']
    creation = <Date 2020-08-17.11:37:30.505>
    creator = 'drougge'
    dependencies = []
    files = ['49401']
    hgrepos = []
    issue_num = 41567
    keywords = []
    message_count = 6.0
    messages = ['375542', '375544', '375556', '379400', '379401', '388771']
    nosy_count = 6.0
    nosy_names = ['ishimoto', 'drougge', 'aeros', 'iritkatriel', 'cebtenzzre', 'doublex']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '43517'
    type = 'behavior'
    url = 'https://bugs.python.org/issue41567'
    versions = ['Python 3.9', 'Python 3.10']

    @drougge
    Copy link
    Mannequin Author

    drougge mannequin commented Aug 17, 2020

    If several threads try to start a multiprocessing.Pool at the same time when no pool has been started before this often fails with an exception like this (the exact import varies):

    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "/tmp/py3.9.0rc1/lib/python3.9/threading.py", line 950, in _bootstrap_inner
        self.run()
      File "/tmp/py3.9.0rc1/lib/python3.9/threading.py", line 888, in run
        self._target(*self._args, **self._kwargs)
      File "/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/context.py", line 118, in Pool
        from .pool import Pool
    ImportError: cannot import name 'Pool' from partially initialized module 'multiprocessing.pool' (most likely due to a circular import) (/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/pool.py)

    This happens even if Pool was imported before starting the threads and is new in 3.9. It's easy to work around by starting a pool in the main thread before starting the other threads.

    I have attached a minimal example that triggers it. Tested on Debian stable and FreeBSD 11.3.

    @drougge drougge mannequin added 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Aug 17, 2020
    @iritkatriel
    Copy link
    Member

    I see the same in Python 3.10 on windows 10.

    If I change the relative imports to absolute imports in a couple of functions in multiprocessing.context as below, the attached (pool_error_on_3.9.py) script not longer raises the exception.

        def SimpleQueue(self):
            '''Returns a queue object'''
            from multiprocessing.queues import SimpleQueue
            return SimpleQueue(ctx=self.get_context())
    
        def Pool(self, processes=None, initializer=None, initargs=(),
                 maxtasksperchild=None):
            '''Returns a process pool object'''
            from multiprocessing.pool import Pool
            return Pool(processes, initializer, initargs, maxtasksperchild,
                        context=self.get_context())

    @iritkatriel iritkatriel added 3.10 only security fixes labels Aug 17, 2020
    @iritkatriel
    Copy link
    Member

    I think this is no a bug, on the basis that multiprocessing.Pool is not thread-safe.

    @atsuoishimoto
    Copy link
    Mannequin

    atsuoishimoto mannequin commented Oct 23, 2020

    multiprocessing module used to work multithread environment until https://bugs.python.org/issue35943 merged.

    I guess we can make multiprocessing thread-safe again if we move local imports to global.
    Does it make sense?

    @atsuoishimoto
    Copy link
    Mannequin

    atsuoishimoto mannequin commented Oct 23, 2020

    @doublex
    Copy link
    Mannequin

    doublex mannequin commented Mar 15, 2021

    Example code (fails):

    import os, concurrent.futures
    
    def parallel_callback( arg ):
        return os.getpid()
    
    def parallel( *args ):
        def thread_callback( param ):
            with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
                future = executor.submit( parallel_callback, param )
                pid = future.result()
                print( 'pid:', pid )
                return pid
        with concurrent.futures.ThreadPoolExecutor(max_workers=len(args)) as executor:
            future = executor.map( thread_callback, args )
            results = list(future)
        print( 'DONE' )
    
    parallel( 1, 2, 3 )

    @pitrou pitrou closed this as completed Mar 16, 2021
    @pitrou pitrou closed this as completed Mar 16, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants