New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
windows: cross compilation fails due to headers with uppercase #78398
Comments
When cross compiling python on a linux host with a case sensitive file system for a windows target. The compilation will fail due to header |
Attached a small helper script to detect camelcase windows headers, Modules/socketmodule.c versionhelpers.h 306 #include <VersionHelpers.h> |
Are those bugs? On Windows, they are spelled with that casing (and while NTFS is case-insensitive, it *is* case-preserving). Whatever package you have used for cross-compiling may have renamed them (similarly Shlwapi.h). In which case, perhaps you could request that the package you are using preserve the case of the files it is cloning, rather than changing it? |
@inada.naoki : thank you ! @steve.dower : I would not consider those bugs per se, but they are number of "Windows.h" includes : 2 whatever might be the 'right' solution, it's always an advantage to |
New windows will support case sensitive filesystem too. |
If this is the case, then we need to normalise the casing to match the actual files (which are mixed-case, not lowercase). You just broke this with your PR :) So which is it? Do we match the casing of the real files? Or do we change to match someone's unofficial versions of the files? |
Not when you are consistently wrong, it's not. Let's figure out the right answer here before making many changes to the codebase. First, can you tell me where you got your lowercased named files from? Because it wasn't the official SDK... |
NTFS in Windows 10 supports a flag to mark a directory as case sensitive [*] via NtSetInformationFile: FileCaseSensitiveInformation, which can be set from the command line via fsutil.exe. In recent builds this flag was made inheritable by child directories, so it doesn't have to be set individually on every directory in a tree. This feature was added for inter-operation with (WSL) Linux programs. It's not intended for system directories such as "Windows Kits".
IIRC, MinGW lower-cases the filenames in its modified version of the SDK. --- [*] Prior to Windows XP, case-sensitive file access was possible via FILE_FLAG_POSIX_SEMANTICS (CreateFile) and FIND_FIRST_EX_CASE_SENSITIVE (FindFirstFileEx). Windows XP (the first consumer version of NT) broke those flags by adding a (default-enabled) global flag in the kernel object manager that makes all device and file access case-insensitive -- regardless of the system call's OBJ_CASE_INSENSITIVE flag. Now in Windows 10 they're bringing back selective case sensitivity in NTFS, but this time using a per-directory flag that overrides the API. It seems they don't care that this breaks the existing OBJ_CASE_INSENSITIVE flag in the NT API. IMO, this could have been handled more cleanly by combining the per-directory flag with one or two new API flags. |
The lowercase headers are indeed part of the mingw-w64 SDK, which is the most convenient option to cross compile on a linux host for a windows target. |
Have you posted to their mailing list to ask why they use different casing from the official SDK and whether they would consider supporting both (via symlink, presumably)? I think it would be strange for CPython to add a style rule for a non-supported toolset. The supported tooling will default to using the correct case for the header files. |
As I understand it, if mingw-w64 would change to camelcase headers, it would break backwards compatibility, since a lot of windows source code uses file names in includes that differ from those on this, which works in MSVC, so it goes unnoticed. There is no good solution. Even if camelcase headers would be available, the CPython source code would break in much more places. I agree that it would be strange to have a style rule for a specific toolset, however mixing different styles is even stranger. The current situation is that 100% of the windows includes are lowercase and allow cross compilation. I would tend to see that as an advantage. If a more comprehensive solution would be available, that would be ideal of course. |
Please be precise with what you are saying - the "current situation" is only so because you proposed changes and got them merged without proper discussion, and "allow cross compilation" with only one alternate toolset. Pretending this is the status quo is dishonest, so please don't insult me like that.
I'm not recommending camel case - I'm recommending matching the casing used for each file in the official SDK (which I agree is a weird mix, but there are certainly compatibility reasons involved in keeping them this way). And yes, it would be a compatibility break for the clones of the SDK, but to a large extent that is their fault for choosing different casing in the first place, so I'm only a little bit sympathetic to that argument. What I'm mostly opposed to is the very weak bug report and the random selection of changes that came with it. Failing to compile on an unsupported toolset is not an immediate "must-fix" issue, and it really ought to go through the affected platform experts so they can confirm acceptable impact on developers who are using the supported tools. There also needs to be this central point so that, assuming we decide to keep them this way, the next person who comes in and complains that the casing doesn't match the actual files is given the correct explanation - otherwise we'll keep switching them back and forth forever. You're also likely to face a maintenance burden for this change, since there is currently no rule or validation that will prevent people from adding properly cased includes in the future (as I hinted, most IDEs on Windows will autocorrect filename casing for headers). If you want one, propose a change to PEP-7 on python-dev, and if it's approved then you can add a build step to validate. Otherwise, you'll have to track changes that are made and fix them as necessary. Without an explicit rule, our default position is "whatever the native target platform/tools prefer". I hope that explains the position a bit better, and why I push back against changes with insufficient justification being provided. At this point, I'm not going to revert these particular changes, as the cases where they will affect native developers are startlingly few these days, but I'm also explicitly saying that this does not mean open-season for all the fixes required for mingw to be happy. Each one will be taken on its merits (primarily compatibility and maintainability), at least until someone has committed to fully support the toolset. |
I googled and checked what MSDN uses before I merged these PRs. |
I'll try to be more precise :
I'm not enough of an expert on the different tools to come up with a comprehensive solution for this issue, but am willing to validate such solution for the use case of cross compiling on linux. |
As someone who has written MSDN samples, I'll spoil the secret and let you know that there's no consistency there, so you can find examples to prove either direction ;) I wasn't expecting expertise or research. I was only expecting a description of the change that clearly specifies who benefits and who loses, and leaves a chance for the affected experts to confirm impact before being merged. There can be a surprising amount of subtlety around cross-platform issues, which is why we have designated people who are responsible for tracking them. |
I'm sorry about that. I won't merge PR like that next time. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: