classification
Title: C Extension import limit
Type: resource usage Stage:
Components: Distutils Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Kevin Schlossser, Yurii Leonov, dstufft, eric.araujo, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-10-26 19:48 by Kevin Schlossser, last changed 2019-11-18 17:54 by Yurii Leonov.

Messages (5)
msg355425 - (view) Author: Kevin Schlossser (Kevin Schlossser) Date: 2019-10-26 19:48
System
Windows 7 x64 SP2
Ram 16GB
6 Core AMD @ 3.2ghz

CPython 3.7.2


C Extension (pyd) import cap.

There seems to be a cap on the number of extensions that a package is able to contain. I am able to import 123 extension modules that my package has but when i go to import number 124 i get the following traceback

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

these extension modules are part of my package, importing an extension module from another package does not change this behavior. it is only when I import the 124th extension that is in my package does it occur. 

When I change the order of the imports the error does not follow the import. I end up getting the same error when the 124th extension gets loaded doesn't matter what extension it is. 

I have tried to see if maybe it was a module limit and I spread the imports across multiple files and it still fails when the 124th gets loaded. I also tried imp.load_dynamic and importlib.import_module and the same error occurs.

If there is a way to work around this limitation it would be very helpful.
msg355578 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-10-28 17:52
There is an implicit cap due to the C runtime (specifically vcruntime140.dll) allocating fibre local storage on load, which seems like the one you are hitting. However, we discovered this before the first 3.5 release and fixed it.

How are you compiling your packages? Or where are you getting them from?

It's possible that someone else is building and statically linking the C runtime, which will cause this, but distutils (and hence CPython) should not be doing it by default.
msg356824 - (view) Author: Yurii Leonov (Yurii Leonov) Date: 2019-11-17 18:32
Looks like I have same problem for Windows 10 (version 1809, build - 17763.864).

I created repository with steps for reproducing - https://github.com/Yuriy-Leonov/cython_imports_limit_issue
msg356887 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-18 17:19
Could you share just one of your .pyd files?

Without being able to see whether they are compiled incorrectly, it's hard to be sure whether this is the same cause as before. It certainly looks like distutils is still going to link correctly.
msg356892 - (view) Author: Yurii Leonov (Yurii Leonov) Date: 2019-11-18 17:54
Done:
.pyd files are added in https://github.com/Yuriy-Leonov/cython_imports_limit_issue/commit/2f9e7c02798fb52185dabfe6ce3811c439ca2839

folder with name "dist_example_with_error"
History
Date User Action Args
2019-11-18 17:54:23Yurii Leonovsetmessages: + msg356892
versions: + Python 3.6, - Python 3.7, Python 3.8, Python 3.9
2019-11-18 17:19:38steve.dowersetmessages: + msg356887
versions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.6
2019-11-17 18:32:20Yurii Leonovsetversions: + Python 3.6, - Python 3.7
nosy: + Yurii Leonov

messages: + msg356824

components: - Interpreter Core, Windows
2019-10-28 17:52:12steve.dowersetnosy: + eric.araujo, dstufft
messages: + msg355578
components: + Distutils
2019-10-26 20:16:46serhiy.storchakasetcomponents: - Extension Modules
2019-10-26 20:16:34serhiy.storchakasetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Interpreter Core, Windows
2019-10-26 19:48:20Kevin Schlosssercreate