classification
Title: distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel
Type: performance Stage: resolved
Components: Distutils Versions: Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, dstufft, eric.araujo, pitrou, vstinner
Priority: normal Keywords:

Created on 2017-04-28 12:17 by vstinner, last changed 2018-09-19 23:22 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1342 closed vstinner, 2017-04-28 12:20
Messages (1)
msg292530 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-04-28 12:17
Since Python 3.5, distutils is able to build extensions in parallel, nice enhancement! But setup.py of CPython is 2x slower in parallel mode when all modules are already built: (1 sec vs 500 ms).

Building extensions calls newer_group() which calls os.stat() 6,856 times. I wrote a cache for os.stat() but it has no impact on performance.

It seems like threads are fighting to death for the GIL in the os.stat() race...

Attached pull request calls newer_group() before spawning threads in parallel mode, so "setup.py build" takes the same time with and without parallel module, when all extensions are already built.

I didn't measure performance when all extensions must be built.
History
Date User Action Args
2018-09-19 23:22:57vstinnersetstatus: open -> closed
resolution: out of date
stage: resolved
2017-04-28 12:22:25vstinnersetnosy: + brett.cannon, pitrou
2017-04-28 12:20:58vstinnersetpull_requests: + pull_request1453
2017-04-28 12:17:50vstinnercreate