Issue 30198: distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel

➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

This issue has been migrated to GitHub: https://github.com/python/cpython/issues/74384

classification

Title:	distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel
Type:	performance	Stage:	resolved
Components:	Distutils	Versions:	Python 3.7

process

Status:	closed	Resolution:	out of date
Dependencies:		Superseder:
Assigned To:		Nosy List:	brett.cannon, dstufft, eric.araujo, pitrou, vstinner
Priority:	normal	Keywords:

Created on 2017-04-28 12:17 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL	Status	Linked	Edit
PR 1342	closed	vstinner, 2017-04-28 12:20

Messages (1)
msg292530 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-04-28 12:17
Since Python 3.5, distutils is able to build extensions in parallel, nice enhancement! But setup.py of CPython is 2x slower in parallel mode when all modules are already built: (1 sec vs 500 ms). Building extensions calls newer_group() which calls os.stat() 6,856 times. I wrote a cache for os.stat() but it has no impact on performance. It seems like threads are fighting to death for the GIL in the os.stat() race... Attached pull request calls newer_group() before spawning threads in parallel mode, so "setup.py build" takes the same time with and without parallel module, when all extensions are already built. I didn't measure performance when all extensions must be built.

msg292530 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2017-04-28 12:17

Since Python 3.5, distutils is able to build extensions in parallel, nice enhancement! But setup.py of CPython is 2x slower in parallel mode when all modules are already built: (1 sec vs 500 ms).

Building extensions calls newer_group() which calls os.stat() 6,856 times. I wrote a cache for os.stat() but it has no impact on performance.

It seems like threads are fighting to death for the GIL in the os.stat() race...

Attached pull request calls newer_group() before spawning threads in parallel mode, so "setup.py build" takes the same time with and without parallel module, when all extensions are already built.

I didn't measure performance when all extensions must be built.

History
Date	User	Action	Args
2022-04-11 14:58:45	admin	set	github: 74384
2018-09-19 23:22:57	vstinner	set	status: open -> closed resolution: out of date stage: resolved
2017-04-28 12:22:25	vstinner	set	nosy: + brett.cannon, pitrou
2017-04-28 12:20:58	vstinner	set	pull_requests: + pull_request1453
2017-04-28 12:17:50	vstinner	create