Enable better DLL resolution #80266
Comments
|
So the fundamental problem is that the default DLL search path on Windows changes in various contexts, and the only consistent approach is the most difficult to support with current packaging tools. The result is .pyd files that need to resolve .dll dependencies from directories *other* than where the .pyd file is located. Here's a generic scenario:
While there are various ways for my_modules.pyd to locate HelperLib.dll, the only totally reliable way is to put HelperLib.dll alongside my_module.pyd. However, because it is needed twice, this means two copies of the DLL, which is unacceptable. With Python 3.8, we are *nearly* dropping support for Windows 7, and I believe we can justify dropping support for Windows 7 without KB2533625 1, which will have been released over eight years by the time 3.8 releases. This means the DLL search path enhancements are available. Proposal #1: CPython calls SetDefaultDllDirectories() 2 on startup and exposes AddDllDirectory() 3 via the sys or os module. This would ensure consistency in DLL search order regardless of security settings, and modules that have their own ".libs" directory have a supported API for adding it to the search path. Past experience of forcing a consistent search path like this is that it has broken many users who expect features like %PATH% to locate DLL dependencies to work. For security reasons, this feature is already deprecated and often disabled (see 4), so it can't be relied upon, but it makes it impossible for a single package to modify this setting or use the supported method for adding more DLL search directories. Proposal #2: Resolve extension modules by full name Without this proposal, the directory structure looks like: my_package\ After this proposal, it could look like: my_package\ Essentially, when searching for modules, allow going up the package hierarchy and locating a fully-qualified name at any level of the import tree. Note that since "import my_package.subpackage1.my_module" implies both "import my_package" and "import my_package.subpackage1", those have to succeed, but then the final part of the import would use subpackage1.__path__ to look for "my_module.pyd" and my_package.__path__ to look for "my_package.subpackage1.my_module.pyd". This allows all extension modules to be co-located in the one (importable) directory, along with a single copy of any shared dependencies. |
|
I nosied both Windows and import experts, and I'm about to go ping relevant numpy/scipy people on numpy/numpy#13019 Personally, I would prefer option #1, despite the pain it would cause. It is the better long-term supported option, and Anaconda has already adopted a patch that does this. However, I think it's most appropriate to be a change in CPython at a major release boundary so that we can provide proper porting information for users. Option #2 is kind of neat, and honestly I thought this already worked when the fully-qualified .pyd was in a folder on sys.path. However, it's going to mess big time with all of our existing build tools. So I'm not thrilled about passing on that kind of pain - then again, most people don't need this, and those who do can do their own hacks to make it work (on the theory that they're already applying their own hacks anyway). I'm totally open to other suggestions on how to make these situations workable, though I will (continue to) push back hard against ideas that simply bring back the security concerns that led us to this point :) |
SetDefaultDllDirectories() affects the entire process, so it would needlessly break the world -- especially for embedding applications and ctypes users that have relied on adding directories to PATH. When loading an extension module, we can simply replace LOAD_WITH_ALTERED_SEARCH_PATH in the LoadLibraryExW flags with LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR and LOAD_LIBRARY_SEARCH_DEFAULT_DIRS (application directory, System32 directory, and directories added via SetDllDirectoryW and AddDllDirectory). Writers of extension modules are constrained by our API. We'll simply mandate that PATH is no longer searched. It's cumbersome to require packages to have to manually call AddDllDirectory before being able to import an extension module with dependencies. We could create a protocol to store relative paths as an embedded resource in the extension module, which would be similar to the RPATH/RUNPATH $ORIGIN field in POSIX. We'd first map the extension module as a data file via LOAD_LIBRARY_AS_DATAFILE | LOAD_LIBRARY_AS_IMAGE_RESOURCE. Load and resolve the relative paths, add them via AddDllDirectory. Call LoadLibraryExW with the above-mentioned flags. Then remove the directories via RemoveDllDirectory. |
|
I'm very against doing magic to extract the names from the DLL, but maybe we could have a search path in the parent package? Then add/remove it around importing the module. I think you're right that we just need to update the LoadLibrary flags, but will those also apply to dependencies of the loaded module? I thought not... |
That works, too. I'm happy either way. We still can't load multiple DLLs with the same name with this technique. That requires private assembly packages, which is doable (in Windows 7+), but a bit complex and requires modifying the embedded #2 manifest of the extension module. The alternative is to rewrite the PE import tables of all DLLs to reference unique DLL names. Neither is necessary if everything is built against unique, versioned DLL names.
The DLL search path is computed once per LoadLibraryExW call based on either the call flags or the process default flags. We shouldn't mess with the process default, since there's no way to restore the legacy DLL search path, in particular this includes the Windows directory (%SystemRoot%), 16-bit System directory (%SystemRoot%\System), current directory, and PATH. Should we support a convenient syntax for including the current value of PATH at extension-module load time? This could be an entry that's exactly equal to "<PATH>". (Less-than and greater-than are reserved as wildcard characters by all Windows file systems that I can think of.) It would iterate over PATH, adding each directory via AddDllDirectory. Of course, all added directories would subsequently be removed via RemoveDllDirectory after the LoadLibraryExW call. |
No. This is the bit that I refuse to add back, at least in CPython itself (if someone does it themselves then it's their bug). Private directories only. |
|
Clear documentation would go a long way toward onboarding package providers. Of course this does not solve the problem for packages with no active ongoing support for windows, and will annoy developers whose code base is full of It would be very helpful if failure to import a pyd (or for that matter to open a DLL with ctypes) due to missing support dlls could be easily debugged. Right now we get a message from windows that seems to suggest the file was not found.
Even better would be official python/Microsoft support for a CLI version of depends.exe like ldd on linux, but that seems much harder and is out of scope for this issue. |
The dumpbin.exe tool with /IMPORTS is a good start, and I've definitely wrapped it in Python before to do this kind of analysis (not reproducibly, yet...). Doing this kind of analysis live is oddly tough, but there may be an ETW provider that a debug hook could enable to get more of a trace. Again, we're deep in third-party tool territory, not a change to core CPython. Certainly if we can drop support for base Windows 7 we will document how to use more recent OS support via whatever we add. Though to a certain extent those hitting problems are going deep enough that reading the MSDN docs will have to be mandatory (at least for those who want to know "why"). I really don't want to have to reproduce those docs and make them guaranteed Python semantics.
Yeah, this is the downside of changing anything at all, though of course since resolving DLLs via PATH is already broken, those developers are already annoyed ;) And we can't add warning that wouldn't annoy those who legitimately modify PATH for process launches. So I think it's mostly about providing a supported path for those developers to be able to port their code when they discover it's broken. |
Correct me if I'm wrong, don't process launches use the |
If you don't provide env, it'll use the current process's environment, and if you do provide it without copying at least some entries, chances are your process won't start (and in general, you copy the current value and add to it). I've never seen anyone try to reset PATH here, nor would I recommend it. |
|
As a note in favour of the "Allow package nesting to be encoded in names, not just directories" approach, we actually have a similar problem affecting builtin modules: they're currently limited to top-level modules, with no way for the module to indicate that it's actually part of the parent package. Details at https://bugs.python.org/issue1644818 (yes, that's a SourceForge era issue number). The solutions may not overlap in practice, but they're conceptually related (i.e. outside frozen modules, the package topology is pretty tightly coupled to the underlying filesystem layout) |
|
Adding Łukasz for his RM opinion on Win7 support for 3.8. According to PEP-11, we've already dropped support for Win7 without Service Pack 1. Win7 SP1 would be dropped ~2-3 months after 3.8 releases, which means we still have to support it for all of 3.8. My concern is the KB2533623 I mentioned in the original post, which adds the ability to control the search path properly. I *think* it might be already included in Win7 SP1, in which case we're fine (I'm confirming this with some colleagues), but if it's optional on top of SP1 then I want to make it required for Python. Alternatively, I'm totally happy to make a three month exception to PEP-11 and just drop Win7 completely for 3.8. But I think that needs to be made official as early as possible, and should get python-dev exposure. Łukasz - thoughts? (Yes, I incorrectly typed the KB number at the top. Apparently I regularly fail to type numbers into bpo for some reason... doesn't happen elsewhere?) |
Windows 7 is still used on about 40% of Windows desktops, and will likely remain popular for a few years after its scheduled end of life in January 2020. Would this be a hard drop, i.e. would installing 3.8 be prevented in Windows 7? Or would it install but require users to manually install KB2533623? |
|
As someone whose work environment is still Windows 7, I'd prefer it if it were a soft desupport (i.e., we require users to manually ensure that the KB fix is installed, but we don't do anything specific to refuse to install Python on Win7). I'd rather not drop Win7 support in 3.8 completely - I feel like that's a bit too aggressive, as Eryk says, there's still a *lot* of Windows 7 usage. |
That's the question I'm asking :) Python 3.9 is currently going to be a hard drop, according to our policy, and if Python 3.8 slips by 3 months then it will also be a hard drop unless we make an exception to the policy. Paul's comment suggests we should avoid slipping/make the exception, and that it's okay to require the update, which is basically where I'm at too (provided the buildbot team are willing to keep an EOL'd OS running for as long as 3.8 is supported). |
|
In the absence of any other comments, here's my proposal.
Any thoughts? The only one I'm not 100% committed to is the SetDefaultDllDirectories call, but I'd rather ship it in alpha/beta releases and pull it out later if necessary. Passing the flags to LoadLibrary should have the same effect anyway, so I don't think changing the defaults in python.exe will make the current scenarios worse. |
That will require rewriting many scripts and packages that use ctypes or cffi to load DLLs. It would also break DLLs that internally rely on modifying PATH for a delayed load, though I hope that's uncommon. I think it's easier for everyone else if we implement this just for extension-module loading with the LoadLibraryExW flags. Also, if I'm understanding your intention, loading an extension may fail when Python is embedded if the process is using the legacy DLL search path. So, like with ctypes, we'll be forcing embedding applications to update how they load DLLs in order to comply with us, else they'll have to accept that some packages won't work without the SetDefaultDllDirectories call.
ctypes calls LoadLibraryW, which uses the default that's set by SetDefaultDllDirectories, if that's what we eventually decide is the best course of action. If we decide to not call SetDefaultDllDirectories, then we should provide a way for ctypes packages to update to using the secure search path instead of relying on the legacy search path. We could rewrite the ctypes LoadLibrary wrapper to call LoadLibraryExW instead of LoadLibraryW and support the flags in the CDLL
I'd prefer some way for scripts and packages to configure their shared-library search paths as static data. The implementation would be kept private in the interpreter. I know there's debate about removing ".pth" files. But maybe we could implement something similar for the DLL search path with package and script ".pthext" files. These would contain a list of directories (relative to the script or package) that extend the shared-library search path.
Also, at runtime we can raise a SystemError if AddDllDirectory isn't found via GetProcAddress. This supports portable Python installations. |
|
Correct me if I'm wrong, but once SetDefaultDllDirectories() is used, there is no going back: PATH no longer can change the search path no matter what flags are used with LoadLibrary* calls (see the recent Anaconda issue when they did this and broke NumPy). Assuming that is true, then
Why is this CPython-specific and "private"? It seems like
There should be some kind of debugging tool for when LoadLibraryEx fails, to indicate what might have gone wrong, perhaps os.getdlldirectory() would be helpful |
This bothers me - how will backward compatibility work in that case? Actually, as a more general point, I have been following this |
Only if they're loading them via PATH. If they're using full paths they'll be fine, and if they're using system DLLs they'll be fine. In both cases, the fix will work (better) with existing versions.
That's true. "import" will always use the secure flags, and so if you were relying on PATH to locate dependencies of the extension module (note that extension modules themselves are loaded by full path, so it doesn't apply to them), you need to stop doing that.
This adds a lot of complexity for very old Windows 7 installs. I'm not inclined to care that much for them - installing security updates isn't a big ask for a nearly EOL operating system.
Correct. But this is already the case if your sysadmin has enabled certain policies or if you're running via Store Python. So you can't rely on PATH anyway.
Not every implementation has to support Windows. We can certainly recommend that those that do copy it, but I'm not going to force MicroPython to declare an exception from a standard Python API.
Fair point. It can go into os. :)
I'd love to have this. Now someone just has to invent one that can be used from Python :) It's unrelated to this discussion - in fact, this change will make it so that you get the failure on _all_ machines, not just on some random user's machine. We can't retrieve the true search path, only the ones that were added through an API that we control and making assumptions based on the documentation. I think this would be more of a distraction. The best way to diagnose actual LoadLibrary failures is to use a debugger, at which point simply getting the search paths doesn't add anything.
The new search order is compatible with the old search order, so you can update all your layouts to have DLL dependencies in suitable locations and you'll be fine. And if you're still writing code for Windows 7 with no security updates installed, Python 3.8 isn't going to save you anyway.
Since we're not going to change the default search directories for the entire process when embedding, the only practical impact is that your extension modules need to have their dependent DLLs either:
And if the embedding application is already calling SetDefaultDllDirectories, as has been recommended for years, then they're already experiencing this change and won't have to update a thing. |
|
Since I just dug enough to find it, the best way to diagnose problems with dependent DLLs not being found is probably to run Process Monitor 1 while doing the import and checking its logs. It should show the paths that were attempted to be accessed. |
OK, cool. But one thing I'm not clear on, will this change just affect Note that this is all on the basis of "I don't understand the
Nobody's suggesting that it will. But maintaining *existing* code that
OK, if that's the case, then that alleviates most of my concerns. But
That seems fine, so let's just state that and keep things simple for
Sadly, in my experience, an awful lot of projects (specifically, open Anyhow, if as you say the only impact is that when a pyd file depends |
SetDefaultDllDirectories affects the whole process and cannot be reverted back to the legacy search path that includes "%SystemRoot%", "%SystemRoot%\System" (the old 16-bit directory), the current working directory, and the PATH directories. Also, there's no LoadLibraryExW flag to use the legacy search path, either, so scripts and packages that use ctypes or cffi will have to be updated if they depend on PATH or changing the working directory. The alternative is to modify Python's importer to use the secure LoadLibraryExW flags (i.e. LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS), without affecting the rest of the process. LOAD_LIBRARY_SEARCH_DEFAULT_DIRS includes the application directory, the user directories added with AddDlldirectory or SetDllDirectoryW, and the System32 directory. LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR prepends the directory of the target DLL, which must be passed as a fully-qualified path.
SetDllDirectoryW still works after calling SetDefaultDllDirectories, as long as we include either LOAD_LIBRARY_SEARCH_USER_DIRS or LOAD_LIBRARY_SEARCH_DEFAULT_DIRS. It only allows adding a single directory, so it's of limited use anyway. |
Is what I proposed in the first place. Adding the SetDefaultDllDirectories call to Py_Main (that is, NOT for embedders) is to ensure consistency throughout the process. It'll only affect extension module dependencies that do their own delay loading and currently rely on unsupported resolve paths. Since everyone seems to have misunderstood at least part of the proposal, I'm not going to explain any more until I have a patch. Excluding boilerplate and docs, it'll be about ten lines of code. |
|
OK, I don't really follow enough of the details here to comment properly. But clearly Steve and Eryk are not yet in agreement. My personal view is that this is something where we should be trying *very* hard to preserve backward compatibility. The proposal here is intended to solve the problem of making it easier for .pyd files to reliably load helper DLLs from shared locations. That's fine, and while it's an important use case (AIUI, it matters for a lot of the scientific stack) IMO it's *not* important enough to warrant breaking working scripts or embedding applications (particularly as this is a fairly obscure detail of how Windows works, so it's unlikely that people carefully follow "best practices" here). I'm very concerned that comments I've seen here, specifically
imply that it's OK to break working code "because they are doing things wrongly". That's not how backward compatibility works - we should avoid breaking *any* working code, no matter how ill-advised it seems to be. If it's necessary to break code that (say) uses ctypes to load a DLL via PATH, or an embedding application that relies on getting DLLs using PATH, then we need to follow PEP-387 and go through a deprecation cycle for the existing behaviour. For the ctypes case I assume we can detect where we found the DLL being loaded, so warning that behaviour will change is certainly possible. For the embedding case, we could (for example) add an API Py_UseSecureSearchPath(bool) that embedders should call to opt into the new search semantics. With an explicit opt-in, we can then migrate that to be the default over time - have the Python API warn for a release if called without the opt-in, and then switch the default to be the secure search path, with applications that want to use the old search path being able to opt out using Py_UseSecureSearchPath(FALSE) for a release or two. That proposal is very much off the top of my head. But the point is that it's not impossible to make the transition follow the normal backward compatibility rules, and so we should do so. Of course, far simpler would be to choose a solution which *doesn't* break existing code :-) |
+1 on that. Code is much harder to misunderstand :-) Paul |
|
The For instance, in complex builds with CMake pybind11 artifacts using deeply nested, temporary directories one would like to temporarily hint some dependent DLLs for testing prior to an install in a fixed installation structure. For such workflows, it would be super useful if we could define additional paths without injecting Python code or going down the rabbit hole manipulating the Do you think you could add support for an environment variable that adds safe DLL directories? |
|
I'd rather not, because that opens the door to all sorts of code injection/search path hijacking attacks. Are you really not running any of your own code that deep? You could use your own environment variable and add them yourself. |
|
That's what I considered as well, but I do not yet see how this is different in terms of risk to a standardized environment variable. (I'll have to implement such a migration path for a series of nested projects in superbuilds, so I thought standardization might be good.) |
|
Just curious: What's the attack vector (own user rights? rights escalation?) that you guard against with respect to search path hijacking? On Linux, we have LD_LIBRARY_PATH and even LD_PRELOAD - as long as one is in control of one's own environment that's usually not a problem. Is that different on Windows (besides that PATH controls apps and dlls)? |
It only applies to people actually running your code, not all arbitrary Python code in the entire world. And builds generally have a different threat model that allows for these kinds of overrides (and so your security teams will protect at a different level).
It's basically just a threat model thing. Windows allows defenders to assume that DLLs won't be hijacked, and so anything that enables it is considered a bug to be fixed - the change to not seacrh Linux has a broader threat model because of Along similar lines, because Python tends to be preinstalled with Linux distros, but isn't preinstalled on Windows, there are different approaches to prevent/detect attackers using Python as part of their attack. They're just both examples of different platform assumptions that lead to different threat models, and so Python works to fit into both without unnecessarily breaking regular expectations. |
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
* CI: Support Shared AMReX Lib Add a Linux and Windows build that each build a shared AMReX library. * Windows: Add PATH to .dll directories Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory * dumpbin -> llvm-objdump -p * Use CDB Debugger https://ten0s.github.io/blog/2022/07/01/debugging-dll-loading-errors * PATH: Pwsh Syntax * Search Last * gflags * Skip gflags (need to install debug tools) * path cdb? actions/runner-images#942 * Remove (Working) Debugger :-) * Clean-Up: Print * CI: Rename Jobs * Remove `llvm-objdump -p`
Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
* CI: Windows + Python Run Python scripts in Windows CI tests. * [Draft] Only Py tests * CMake Python Tests: Multi-Config Hints Make sure that we can find the extra build config suffix directory that is appended in multi-config generators. * Python: Test After Install Calling `os.add_dll_directory()` injection is too complicated for now. https://bugs.python.org/issue43173 * Python DLL Loading Note * Lib Symlink: Allow Static, too * pyAMReX: .dll support * Static: Missing Env for Install * DLL: Debug Loader Logic * Windows: Add PATH to .dll directories Python 3.8+ on Windows: DLL search paths for dependent shared libraries. Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory * CI: Debug * CMake: Update Comments * CMake: Fix Install * Win PATH: Add ABLASTR & AMReX DLLs * MSVC: w/ shared (dll) libs * CI: Cleanup * Test Before Install Make sure we can run tests from the build tree.
Python 3.8+ on Windows: DLL search paths for dependent shared libraries Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory
* pyAMReX: Build System * CI Updates (Changed Options) * Callback modernization (#4) * refactor callbacks.py * added binding code in `pyWarpX.cpp` to add or remove keys from the callback dictionary * minor PR cleanups Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Added Python level reference to fetch the multifabs (#3) * pyAMReX: Build System * Added Python level reference to Ex_aux * Now uses the multifab map * Fix typo Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Add initialization and finalize routines (#5) A basic PICMI input file will now run to completion. * Regression Tests: WarpX_PYTHON=ON * Update Imports to nD pyAMReX * IPO/LTO Control Although pybind11 relies heavily on IPO/LTO to create low-latency, small-binary bindings, some compilers will have troubles with that. Thus, we add a compile-time option to optionally disable it when needed. * Fix: Link Legacy WarpXWrappers.cpp * Wrap WarpX instance and start multi particle container * Fix test Python_pass_mpi_comm * Start wrapper for multiparticle container * Add return policy Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update fields to use MultiFabs directly Remove EOL white space Removed old routines accessing MultiFabs Update to use "node_centered" * Fix compilation with Python * Update fields.py to use modified MultiFab tag names * Remove incorrect, unused code * Add function to extract the WarpX MultiParticleContainer * Complete class WarpXParticleContainer * Wrap functions getNprocs / getMyProc * restore `install___` callback API - could remove later if we want but should maintain backward compatibility for now * add `gett_new` and `getistep` functions wrappers; fix typos in `callbacks.py`; avoid error in getting `rho` from `fields.py` * Update callback call and `getNproc`/`getMyProc` function * Replace function add_n_particles * Fix setitem in fields.py for 1d and 2d * also update `gett_new()` in `_libwarpx.py` in case we want to keep that API * added binding for `WarpXParIter` - needed to port `libwarpx.depositChargeDensity()` which is an ongoing effort * Wrap function num_real_comp * added binding for `TotalNumberOfParticles` and continue progress on enabling 1d MCC test to run * add `SyncRho()` binding and create helper function in `libwarpx.depositChargeDensity` to manage scope of the particle iter * Clean up issues in fields.py * update bindings for `get_particle_structs` * Fix setitem in fields.py * switch order of initialization for particle container and particle iterator * switch deposit_charge loop to C++ code; bind `ApplyInverseVolumeScalingToChargeDensity` * move `WarpXParticleContainer.cpp` and `MultiParticleContainer.cpp` to new Particles folder * added binding for `ParticleBoundaryBuffer` * More fixes for fields.py * Fix: Backtraces from Python Add the Python executable name with an absolute path, so backtraces in AMReX work. See linked AMReX issue for details. * Cleaning * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix: Backtraces from Python Part II Do not add Python script name - it confuses the AMReX ParmParser to build its table. * Fix: CMake Dependencies for Wheel This fixes a racecondition during `pip_install`: it was possible that not all dimensions where yet build from pybind before we start packing them in the wheel for pip install. * MCC Test: Install Callbacks before Run Otherwise hangs in aquiring the gil during shutdown. * addition of `Python/pywarpx/particle_containers.py` and various associated bindings * Fix: CMake Superbuild w/ Shared AMReX We MUST build AMReX as a shared (so/dll/dylib) library, otherwise all the global state in it will cause split-brain situations, where our Python modules operate on different stacks. * add `clear_all()` to callbacks in order to remove all callbacks at finalize * add `-DWarpX_PYTHON=ON` to CI tests that failed to build * add `get_comp_index` and continue to port particle data bindings * Add AMReX Module as `libwarpx_so.amr` Attribute to pass through the already loaded AMReX module with the matching dimensionality to the simulation. * Fix for fields accounting for guard cells * Fix handling of ghost cells in fields * Update & Test: Particle Boundary Scraping * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * CI: Python Updates - modernize Python setups - drop CUDA 11.0 for good and go 11.3+ as documented already ``` Error #3246: Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!" ``` * CI: Python Updates (chmod) * Add support for cupy in fields.py * Add MultiFab reduction routines * CI: CUDA 11.3 is <= Ubuntu 20.04 * changed `AddNParticles` to take `amrex::Vector` arguments * setup.py: WarpX_PYTHON=ON * update various 2d and rz tests with new APIs * add `-DWarpX_PYTHON_IPO=OFF` to compile string for 2d and 3d Python CI tests to speed up linking * CI: -DpyAMReX_IPO=OFF * CI: -DpyAMReX_IPO=OFF actually adding `=OFF` * CI: Intel Python * CI: macOS Python Executable Ensure we always use the same `python3` executable, as specified by the `PATH` priority. * CMake: Python Multi-Config Build Add support for multi-config generators, especially on Windows. * __init__.py: Windows DLL Support Python 3.8+ on Windows: DLL search paths for dependent shared libraries Refs.: - python/cpython#80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory * CI: pywarpx Update our setup.py cannot install pyamrex yet as a dependency. * ABLASTR: `ablastr/export.H` Add a new header to export public globals that are not covered by `WINDOWS_EXPORT_ALL_SYMBOLS`. https://stackoverflow.com/questions/54560832/cmake-windows-export-all-symbols-does-not-cover-global-variables/54568678#54568678 * WarpX: EXPORT Globals in `.dll` files WarpX still uses a lot of globals: - `static` member variables - `extern` global variables These globals cannot be auto-exported with CMake's `WINDOWS_EXPORT_ALL_SYMBOLS` helper and thus we need to mark them manually for DLL export (and import) via the new ABLASTR `ablastr/export.H` helper macros. This starts to export symbols in the: - WarpX and particle container classes - callback hook database map - ES solver * CI: pywarpx Clang CXXFLAGS Down Move CXXFLAGS (`-Werror ...`) down until deps are installed. * GNUmake: Generate `ablastr/export.H` * CMake: More Symbol Exports for Windows * `WarpX-tests.ini`: Simplify Python no-IPO Also avoids subtle differences in compilation that increase compile time. * Update PICMI_inputs_EB_API.py for embedded_boundary_python_API CI test * Fix Python_magnetostatic_eb_3d * Update: Python_restart_runtime_components New Python APIs * Windows: no dllimport for now * CI: Skip `PYINSTALLOPTIONS` For Now * CMake: Dependency Bump Min-Versions for external packages picked up by `find_package`. * Fix F and G_fp names in fields.py * Tests: Disable `Python_pass_mpi_comm` * Wrappers: Cleanup * pyWarpX: Include Cleaning * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fields.py: Fix F and G Wrappers Correct MultiFab names (w/o components). * Remove unused in fields.py * Windows: New Export Headers - ABLASTR: `ablastr/export.H` - WarpX: `Utils/export.H` * removed `WarpInterface.py` since that functionality is now in `particle_containers.py`; removed parts of `WarpXWrappers.cpp` that have been ported to pyamrex * CMake: Link OBJECT Target PRIVATE * CMake: Remove OBJECT Target Simplify and make `app` link `lib` (default: static). Remove OBJECT target. * Fix in fields.py for the components index * Update get_particle_id/cpu As implemented in pyAMReX with AMReX-Codes/pyamrex#165 * WarpX: Update for Private Constructor * Import AMReX Before pyd DLL Call Importing AMReX will add the `add_dll_directory` to a potentially shared amrex DLL on Windows. * Windows CI: Set PATH to amrex_Nd.dll * CMake: AMReX_INSTALL After Python In superbuild, Python can modify `AMReX_BUILD_SHARED_LIBS`. * Clang Win CI: Manually Install requirements Sporadic error is: ``` ... Installing collected packages: pyparsing, numpy, scipy, periodictable, picmistandard ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\hostedtoolcache\\windows\\Python\\3.11.4\\x64\\Lib\\site-packages\\numpy\\polynomial\\__init__.py' Consider using the `--user` option or check the permissions. ``` * Hopefully final fixes to fields.py * Update getProbLo/getProbHi * Set plasma length strength Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> * Fix fields method to remove CodeQL notice * Update Comments & Some Finalize * Move: set_plasma_lens_strength to MPC --------- Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Co-authored-by: David Grote <dpgrote@lbl.gov> Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> Co-authored-by: Dave Grote <grote1@llnl.gov> Co-authored-by: Roelof Groenewald <regroenewald@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: