classification
Title: Windows IocpProactor: CreateIoCompletionPort 4th arg 0xffffffff -- why is this value the default?
Type: Stage:
Components: asyncio Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, jeffr@livedata.com, vstinner, yselivanov
Priority: normal Keywords:

Created on 2018-12-24 23:41 by jeffr@livedata.com, last changed 2019-02-16 22:00 by jeffr@livedata.com.

Messages (6)
msg332498 - (view) Author: Jeff Robbins (jeffr@livedata.com) * Date: 2018-12-24 23:41
By default, the __init__ function of IocpProactor in windows_events.py calls CreateIoCompletionPort with a 4th argument of 0xffffffff, yet MSDN doesn't document this as a valid argument.    https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport

It looks like the 4th arg (NumberOfConcurrentThreads) is meant to be either a positive integer or 0.  0 is a special value meaning "If this parameter is zero, the system allows as many concurrently running threads as there are processors in the system."

Why does asyncio use 0xffffffff instead as the default value?
msg332499 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-12-24 23:59
The value type is DWORD.

0xffffffff is a maximum allowed value for DWORD.

0 implies something like 8 for Intel i7.

I don't remember the reason (the value was present is very first proactor implementation and never changed after it).

Maybe giving Windows a privilegy to decide what number of IO threads is optimal is not a bad idea, I don't know.
msg332502 - (view) Author: Jeff Robbins (jeffr@livedata.com) * Date: 2018-12-25 01:35
Per https://stackoverflow.com/questions/38133870/how-the-parameter-numberofconcurrentthreads-is-used-in-createiocompletionport, it seems that `NumberOfConcurrentThreads` controls what happens when multiple threads call `GetQueuedCompletionStatus`.  

But since a given instance of `IocpProactor` only calls `GetQueuedCompletionStatus` from a single thread, probably this arg doesn't matter, and the value `1` would be more explicit about the pattern asyncio is using?  A huge number is, presumably, either not relevant or, at worst, wasteful of some kernel resource.  

Am I correct that only one thread calls `GetQueuedCompletionStatus` on a given `iocp` object in asyncio under Windows `IocpProactor`?
msg333695 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-15 12:34
> Am I correct that only one thread calls `GetQueuedCompletionStatus` on a given `iocp` object in asyncio under Windows `IocpProactor`?

An asyncio event loop must only run in a single thread at the same time. It doesn't make sense to run the same event loop multiple times in parallel and it is not supported (most asyncio classes are not thread-safe, there is no need to be thread-safe).
msg333697 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-01-15 12:41
From my understanding the question is: replace 0xffffffff with 0 or 1?

I don't know much about IOCP, 0 sounds safer for me.
msg335724 - (view) Author: Jeff Robbins (jeffr@livedata.com) * Date: 2019-02-16 22:00
I don't understand why 0 would be safer.  Since asyncio can only service this IOCP from its single threaded event loop, I would have thought 1 would express the intent better.  

Why not convey to the OS what we're up to, in case that helps it do a better job or reduces resource footprint?
History
Date User Action Args
2019-02-16 22:00:08jeffr@livedata.comsetmessages: + msg335724
2019-01-15 12:41:12asvetlovsetmessages: + msg333697
2019-01-15 12:34:29vstinnersetnosy: + vstinner
messages: + msg333695
2018-12-25 01:35:56jeffr@livedata.comsetmessages: + msg332502
2018-12-24 23:59:53asvetlovsetmessages: + msg332499
2018-12-24 23:41:44jeffr@livedata.comcreate