Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_multiprocessing failure (test_pool_worker_lifetime) #55975

Closed
pitrou opened this issue Apr 4, 2011 · 6 comments
Closed

test_multiprocessing failure (test_pool_worker_lifetime) #55975

pitrou opened this issue Apr 4, 2011 · 6 comments
Labels
stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@pitrou
Copy link
Member

pitrou commented Apr 4, 2011

BPO 11766
Nosy @pitrou, @vstinner

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-04-06.21:03:06.044>
created_at = <Date 2011-04-04.21:31:27.177>
labels = ['tests', 'type-bug', 'library']
title = 'test_multiprocessing failure (test_pool_worker_lifetime)'
updated_at = <Date 2011-04-06.21:03:06.042>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2011-04-06.21:03:06.042>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2011-04-06.21:03:06.044>
closer = 'pitrou'
components = ['Library (Lib)', 'Tests']
creation = <Date 2011-04-04.21:31:27.177>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 11766
keywords = []
message_count = 6.0
messages = ['132974', '133161', '133164', '133166', '133167', '133168']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'jnoller', 'asksol', 'neologix', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11766'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']
@pitrou
Copy link
Member Author

pitrou commented Apr 4, 2011

This sometimes happens on the buildbots:

test test_multiprocessing failed -- Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_multiprocessing.py", line 1191, in test_pool_worker_lifetime
    self.assertNotIn(None, finalworkerpids)
AssertionError: None unexpectedly found in [1788, 3984, None]

e.g. http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/4334/steps/test/logs/stdio

@pitrou pitrou added stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Apr 4, 2011
@neologix
Copy link
Mannequin

neologix mannequin commented Apr 6, 2011

Does this only happen on Cygwin buildbots ?
If yes, then it might simply be an issue with Cygwin's fork implementation, which is much slower than natively.
Right now, the test waits 0.5s before checking that the processes are started, after repopulating the pool. While 0.5s is normally way enough for forking a couple processes, it seems that under Cygwin this can take a surprising amount of time, see http://old.nabble.com/Slow-fork-issue---Win-x64-td19538601.html and also http://superuser.com/questions/133313/can-i-speed-up-cygwins-fork
Unless I misunderstand their benchmarks, the fork (+exec) rate of "date" from a shell can be as low as 5/sec, so I can only guess what forking cpython would take.
Maybe we could try to increase the timeout before checking the PIDs:

+    countdown = 10
-    countdown = 10
    while countdown and not all(w.is_alive() for w in p._pool):
        countdown -= 1
        time.sleep(DELTA)

@pitrou
Copy link
Member Author

pitrou commented Apr 6, 2011

There is no Cygwin buildbot, this is a regular Windows buildbot (despite the path in the traceback).
That said, spawning processes is quite slow under Windows anyway (much slower than under Linux). So we could up the countdown.

@python-dev
Copy link
Mannequin

python-dev mannequin commented Apr 6, 2011

New changeset c4a514199dba by Antoine Pitrou in branch '3.2':
Issue bpo-11766: increase countdown waiting for a pool of processes to start
http://hg.python.org/cpython/rev/c4a514199dba

New changeset 3eac8302a448 by Antoine Pitrou in branch 'default':
Issue bpo-11766: increase countdown waiting for a pool of processes to start
http://hg.python.org/cpython/rev/3eac8302a448

@python-dev
Copy link
Mannequin

python-dev mannequin commented Apr 6, 2011

New changeset 2e4cdaffe493 by Antoine Pitrou in branch '2.7':
Issue bpo-11766: increase countdown waiting for a pool of processes to start
http://hg.python.org/cpython/rev/2e4cdaffe493

@pitrou
Copy link
Member Author

pitrou commented Apr 6, 2011

Hopefully fixed now, thanks for your diagnosis!

@pitrou pitrou closed this as completed Apr 6, 2011
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pitrou