classification
Title: C Extension import limit
Type: resource usage Stage: patch review
Components: Distutils Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Kevin Schlossser, P12 Pfx, Yurii Leonov, dstufft, eric.araujo, miss-islington, p-ganssle, paul.moore, scoder, steve.dower, tim.golden, xinfazhu, zach.ware
Priority: normal Keywords: patch

Created on 2019-10-26 19:48 by Kevin Schlossser, last changed 2020-05-16 18:43 by P12 Pfx.

Files
File name Uploaded Description Edit
setup.py xinfazhu, 2020-02-27 00:01
Pull Requests
URL Status Linked Edit
PR 18724 merged steve.dower, 2020-03-01 15:05
PR 18758 merged miss-islington, 2020-03-03 00:04
PR 18759 merged steve.dower, 2020-03-03 00:08
Messages (22)
msg355425 - (view) Author: Kevin Schlossser (Kevin Schlossser) Date: 2019-10-26 19:48
System
Windows 7 x64 SP2
Ram 16GB
6 Core AMD @ 3.2ghz

CPython 3.7.2


C Extension (pyd) import cap.

There seems to be a cap on the number of extensions that a package is able to contain. I am able to import 123 extension modules that my package has but when i go to import number 124 i get the following traceback

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

these extension modules are part of my package, importing an extension module from another package does not change this behavior. it is only when I import the 124th extension that is in my package does it occur. 

When I change the order of the imports the error does not follow the import. I end up getting the same error when the 124th extension gets loaded doesn't matter what extension it is. 

I have tried to see if maybe it was a module limit and I spread the imports across multiple files and it still fails when the 124th gets loaded. I also tried imp.load_dynamic and importlib.import_module and the same error occurs.

If there is a way to work around this limitation it would be very helpful.
msg355578 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-10-28 17:52
There is an implicit cap due to the C runtime (specifically vcruntime140.dll) allocating fibre local storage on load, which seems like the one you are hitting. However, we discovered this before the first 3.5 release and fixed it.

How are you compiling your packages? Or where are you getting them from?

It's possible that someone else is building and statically linking the C runtime, which will cause this, but distutils (and hence CPython) should not be doing it by default.
msg356824 - (view) Author: Yurii Leonov (Yurii Leonov) Date: 2019-11-17 18:32
Looks like I have same problem for Windows 10 (version 1809, build - 17763.864).

I created repository with steps for reproducing - https://github.com/Yuriy-Leonov/cython_imports_limit_issue
msg356887 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-18 17:19
Could you share just one of your .pyd files?

Without being able to see whether they are compiled incorrectly, it's hard to be sure whether this is the same cause as before. It certainly looks like distutils is still going to link correctly.
msg356892 - (view) Author: Yurii Leonov (Yurii Leonov) Date: 2019-11-18 17:54
Done:
.pyd files are added in https://github.com/Yuriy-Leonov/cython_imports_limit_issue/commit/2f9e7c02798fb52185dabfe6ce3811c439ca2839

folder with name "dist_example_with_error"
msg359314 - (view) Author: Kevin Schlossser (Kevin Schlossser) Date: 2020-01-05 01:12
Thank you msg356892 for spear heading this for me. I family things to attend to so I apologize for opening this bug report and then walking away..

As far as recreating this issue. It simple to do. you can either use cython or you can put together a quick script that will make say 150 extensions. put one  line of code in it and compile it.. The results always end up being the same 123 extensions can be loaded on the 124th a TB happens.. It does not matter where the files are inside the package. it is always the 124th one that gets loaded for any one package. importing a pyd from another package does not change the number. The extensions have to be a direct descendant of one single package.
msg359395 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-01-06 02:34
I haven't looked into _why_ yet, but the first PYD I grabbed from the GitHub link above has had the CRT statically linked. This is not the default (or it should not be), because when we made it the default this exact issue occurred :)

If somehow the default linking mode in distutils has changed, we should fix that. If it is being overridden by Setuptools or Cython then we should get those projects fixed.

Stefan/Paul - do Cython or Setuptools override compiler/linker settings like this at all? Most likely it's the /MT vs /MD option
msg359403 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-01-06 06:17
Cython doesn't interfere with the C compiler setup in any way, that's left
to distutils/setuptools (and the user).
msg362529 - (view) Author: Xinfa Zhu (xinfazhu) * Date: 2020-02-23 17:54
I have a similar issue. Do we have an estimate how long it may take to fix this bug? Thanks. I can help but would need some mentoring.
msg362549 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-23 23:10
Okay, looking at _find_vcvarsall in distutils, I'm guessing that something about how your machines are set up means that the vcredist search is failing.

First, could you specify which versions of Visual Studio you have installed, and if possible which one is being found and used by your builds.

Then, if you can search your install to find both vcvarsall.bat and vcruntime140.dll (there will be a few of these) and post the paths, that may indicate if the layout isn't consistent.

Distutils will try and dynamically link to the runtime if you have the redist, and statically link it if you don't (though I don't remember why it doesn't just rely on the copy included with Python... probably future-proofing or licencing).
msg362557 - (view) Author: Xinfa Zhu (xinfazhu) * Date: 2020-02-24 01:57
Thanks Steve. Here is what you requested.

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ ./Installer/vswhere.exe
Visual Studio Locator version 2.7.1+180c706d56 [query version 2.3.2200.14893]
Copyright (C) Microsoft Corporation. All rights reserved.

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ find . -name "vcvarsall.bat"
./2019/BuildTools/VC/Auxiliary/Build/vcvarsall.bat

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ find . -name "vcruntime140.dll"
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/onecore/x64/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/onecore/x86/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/x64/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/x86/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x86/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx86/x64/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx86/x86/vcruntime140.dll
./Installer/resources/app/ServiceHub/Services/Microsoft.VisualStudio.Setup.Service/vcruntime140.dll
./Installer/vcruntime140.dll

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio


I want mention that I have 301 extension modules. I used setuptools in my setup.py
    from setuptools import setup
    from setuptools.extension import Extension
python setup.py build_ext
python setup.py bdist_wheel
I bring the whl file to another computer, pip install, then launch the app and I get this error
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

I tried using distutils instead, but it says: error: invalid command 'bdist_wheel'
    from distutils.core import setup
    from distutils.extension import Extension
msg362653 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-25 17:41
You should be able to install "wheel" without setuptools to get the bdist_wheel command.

Can you confirm that the build process is actually using that Visual Studio install? If it's going through the regular distutils detection process then it ought to be finding the right files.
msg362756 - (view) Author: Xinfa Zhu (xinfazhu) * Date: 2020-02-27 00:01
I have had wheel installed. Following this SO answer (https://stackoverflow.com/a/50314071/7269441), I added "import setuptools" before distutils (my setup.py attached).

Following is the log, where we see the VS path it is using. BTW, I am trying to reduce my package to <120 modules and test it.

(base) C:\Users\xinfa\Documents\code\ezcad-dev\beta>C:\Users\xinfa\AppData\Local\Continuum\anaconda3\python.exe setup.py build_ext
Include directory: C:\Users\xinfa\Documents\code\ezcad-dev\beta
Number of ext modules = 225
running build_ext
building 'ezcad.__main__' extension
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT -IC:\Users\xinfa\Documents\code\ezcad-dev\beta -I. -IC:\Users\xinfa\AppData\Loca
l\Continuum\anaconda3\include -IC:\Users\xinfa\AppData\Local\Continuum\anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits
\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.1836
2.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcezcad\__main__.c /Fobuild\temp.win-amd64-3.7\Release\ezcad\__main__.obj
__main__.c
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /nodefaultlib:libucrt.lib ucrt.lib /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPAT
H:C:\Users\xinfa\AppData\Local\Continuum\anaconda3\libs /LIBPATH:C:\Users\xinfa\AppData\Local\Continuum\anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\l
ib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" build\temp.win-amd64-3.7\Release\ezcad\__main__.obj /OUT:build\lib.win
-amd64-3.7\ezcad\__main__.cp37-win_amd64.pyd
   Creating library build\lib.win-amd64-3.7\ezcad\__main__.cp37-win_amd64.lib and object build\lib.win-amd64-3.7\ezcad\__main__.cp37-win_amd64.exp
Generating code
Finished generating code
version = 0.1.9
msg362793 - (view) Author: Xinfa Zhu (xinfazhu) * Date: 2020-02-27 12:55
FYI when I reduced my package to 106 extension modules, I could run without the DLL error.
msg362795 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-27 13:16
Thanks, it does seem like it's finding the correct MSVC, but is not finding the redistributable DLL:

> cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT ...

The "/MT" means to statically the CRT, and then we use link settings later to dynamically link the part installed in the OS (see https://stevedower.id.au/blog/building-for-python-3-5-part-two for more details).

The problem seems to be that this glob in distutils can't find the path listed below it:

"2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll"

"2019\BuildTools\VC\Redist\MSVC\14.24.28127\x64\Microsoft.VC142.CRT\vcruntime140.dll"

We swallow a lot of errors while doing this glob, so I assume there's something about your install that's affecting it (maybe related to it being BuildTools rather than Community or higher?)

Could you try running this code and see what result/error you get:

import glob
glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll", recursive=True)
msg362796 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-27 13:19
Hmm, just ran it on my own and it's finding the OneCore file first, which is not a problem (yet) but could become one.

So we at least need to replace the "**" with a "*" in Lib/distutils/_msvccompiler.py#L109. Not sure if that will help in this other case or not.
msg363067 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-01 15:03
In thinking about this, I think the best way forward is to just remove the logic that might statically link the initialization code, and instead commit to CPython releases always including vcruntime140.dll even if we switch to a newer version one day.

Hopefully third party distributions will do the same, though it should only matter for ABI3 modules that are not recompiled for the newer versions.
msg363080 - (view) Author: Xinfa Zhu (xinfazhu) * Date: 2020-03-01 19:20
Steve, don't know if you still need it but here is what you requested. Sorry for the slow move (I was working on something else). Seems mine is finding the x64 before the OneCore, though I don't know the significance.

Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] on win32
In[2]: import glob
In[3]: glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll", recursive=True)
Out[3]: 
['C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\onecore\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll']
msg363157 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-02 11:20
> don't know if you still need it but here is what you requested.

Thanks. That basically confirms that something is interfering with distutils. Skipping the check entirely should avoid it.
msg363222 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-03 00:04
New changeset ce3a4984089b8e0ce5422ca32d75ad057b008074 by Steve Dower in branch 'master':
bpo-38597: Never statically link extension initialization code on Windows (GH-18724)
https://github.com/python/cpython/commit/ce3a4984089b8e0ce5422ca32d75ad057b008074
msg363225 - (view) Author: miss-islington (miss-islington) Date: 2020-03-03 00:26
New changeset 8a5f7ad5e423b74ea612e25472e5bff3adf1ea87 by Steve Dower in branch '3.7':
[3.7] bpo-38597: Never statically link extension initialization code on Windows (GH-18724) (GH-18759)
https://github.com/python/cpython/commit/8a5f7ad5e423b74ea612e25472e5bff3adf1ea87
msg363226 - (view) Author: miss-islington (miss-islington) Date: 2020-03-03 00:30
New changeset 0d20364b132014eec609b900997c34779a4d548c by Miss Islington (bot) in branch '3.8':
bpo-38597: Never statically link extension initialization code on Windows (GH-18724)
https://github.com/python/cpython/commit/0d20364b132014eec609b900997c34779a4d548c
History
Date User Action Args
2020-05-16 18:43:09P12 Pfxsetnosy: + P12 Pfx
2020-03-03 00:30:24miss-islingtonsetmessages: + msg363226
2020-03-03 00:26:30miss-islingtonsetmessages: + msg363225
2020-03-03 00:08:31steve.dowersetpull_requests: + pull_request18114
2020-03-03 00:04:24miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request18113
2020-03-03 00:04:18steve.dowersetmessages: + msg363222
2020-03-02 11:20:48steve.dowersetmessages: + msg363157
versions: + Python 3.8, Python 3.9
2020-03-01 19:20:55xinfazhusetmessages: + msg363080
2020-03-01 15:05:59steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request18080
2020-03-01 15:03:44steve.dowersetmessages: + msg363067
2020-02-27 13:19:36steve.dowersetmessages: + msg362796
2020-02-27 13:16:35steve.dowersetmessages: + msg362795
2020-02-27 12:55:22xinfazhusetmessages: + msg362793
2020-02-27 00:01:12xinfazhusetfiles: + setup.py

messages: + msg362756
versions: + Python 3.7, - Python 3.6
2020-02-25 17:41:33steve.dowersetmessages: + msg362653
2020-02-24 01:57:32xinfazhusetmessages: + msg362557
2020-02-23 23:10:39steve.dowersetmessages: + msg362549
2020-02-23 17:54:50xinfazhusetnosy: + xinfazhu
messages: + msg362529
2020-01-06 06:17:15scodersetmessages: + msg359403
2020-01-06 02:34:37steve.dowersetnosy: + scoder, p-ganssle
messages: + msg359395
2020-01-05 01:12:07Kevin Schlosssersetmessages: + msg359314
2019-11-18 17:54:23Yurii Leonovsetmessages: + msg356892
versions: + Python 3.6, - Python 3.7, Python 3.8, Python 3.9
2019-11-18 17:19:38steve.dowersetmessages: + msg356887
versions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.6
2019-11-17 18:32:20Yurii Leonovsetversions: + Python 3.6, - Python 3.7
nosy: + Yurii Leonov

messages: + msg356824

components: - Interpreter Core, Windows
2019-10-28 17:52:12steve.dowersetnosy: + eric.araujo, dstufft
messages: + msg355578
components: + Distutils
2019-10-26 20:16:46serhiy.storchakasetcomponents: - Extension Modules
2019-10-26 20:16:34serhiy.storchakasetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Interpreter Core, Windows
2019-10-26 19:48:20Kevin Schlosssercreate