Based on a bug report (https://bitbucket.org/openpyxl/openpyxl/issues/1266/locale) from a user of the openpyxl library I've identified a bug in the zipfile module that causes the Python process to crash on Windows. Currently tested with Python 3.7.3 (32-bit on Windows 10).

Sample code

import faulthandler
import locale
from zipfile import ZipFile

faulthandler.enable()

locale.setlocale(locale.LC_ALL, 'de_DE')
out = open("out.zip", "wb") 
archive = ZipFile(out, "w")
archive.writestr("properties.xml", b"<workbookPr/>")

faulthandler fingers line 1757 as the culprit but running this line locally does not cause the crash. The issue seems to be limited to Windows.

I can't reproduce this error on my system by using the same code.

Could you share the system info with us?

Or would you share the Exception with us ?

I guess it's caused by system setting

Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32

Same environment.

But I still can not reproduce this exception. I guess maybe it's about the local time or timezone problem. I will find a way to figure it out

That's what we thought when we looked at it, but as I said, I couldn't reproduce it with just the `time` call or the `ZInfo` instantiation, so something odd is happening. I do have a German version of Windows as I suspect the original reporter does. You'd think that sort of thing wouldn't matter…

I can reproduce it on Python 3.7.3 german Windows10 enterprise

Windows fatal exception: code 0xc0000374

Current thread 0x00003bc8 (most recent call first):
  File "C:\Python37\lib\zipfile.py", line 1757 in writestr

Hi  Dominik Geldmacher

May I get your system version? 1809 or 1903?

I guess maybe it's microsoft error

Enterprise 1809 and Professional 1803 (german localized) both reproduce the issue

copy that

I will reset my locale setting to figure it out

winver tells me I have 1809. I'm only using Windows in a VM so I'm not that familiar with its innards.

Also get the error with WinPython 3.6:

Windows fatal exception: code 0xc0000374

Current thread 0x000010c0 (most recent call first):
  File "C:\Users\charlieclark\WPy64-3680\python-3.6.8.amd64\lib\zipfile.py", line 1658 in writestr
  File "<stdin>", line 1 in <module>

Related to issue bpo-36319

I can confirm the error is caused by time.localtime(time.time()) as indicated by the related bug.

I've also found the crash log. It's in C:\ProgramData\Microsoft\Windows\WER\ReportQueue on my machine. Well, at least that's all I can find.

So does that mean that simply doing:

import locale, time
locale.setlocale(locale.LC_ALL, 'de_DE')
time.localtime(time.time())

is enough to trigger the heap corruption?  If yes, then what is the output of:

import locale, time
locale.setlocale(locale.LC_ALL, 'de_DE')
time.strftime('%Z')

import time, locale
locale.setlocale(locale.LC_ALL, 'de_DE')
'de_DE'
time.strftime("%Z")
aborted (disconnected)

Ok, now let's try it using the C runtime directly:

import ctypes, struct
libc = ctypes.cdll.msvcrt
buf = ctypes.create_string_buffer(1024)
tm = struct.pack('9i', 2019, 5, 6, 9, 50, 4, 0, 126, 1)
print('count:', libc.strftime(buf, 1024, b'%Z', tm))
print('value:', buf.value)

import ctypes, struct
libc = ctypes.cdll.msvcrt
buf = ctypes.create_string_buffer(1024)
tm = struct.pack('9i', 2019, 5, 6, 9, 50, 4, 0, 126, 1)
print('count:', libc.strftime(buf, 1024, b'%Z', tm))
print('value:', buf.value)

count: 28
value: b'Mitteleurop\xe4ische Sommerzeit'

Oops, I forgot to add in my snippet, the setlocale() call prior to calling the C strftime() function.  So an updated test:

import locale
locale.setlocale(locale.LC_ALL, 'de_DE')
import ctypes, struct
libc = ctypes.cdll.msvcrt
buf = ctypes.create_string_buffer(1024)
tm = struct.pack('9i', 2019, 5, 6, 9, 50, 4, 0, 126, 1)
print('count:', libc.strftime(buf, 1024, b'%Z', tm))
print('value:', buf.value)
wbuf = ctypes.create_unicode_buffer(1024)
print('count:', libc.wcsftime(wbuf, 1024, '%Z', tm))
print('value:', wbuf.value)
print('count:', libc.mbstowcs(wbuf, buf, 1024))
print('value:', wbuf.value)

import ctypes, struct
libc = ctypes.cdll.msvcrt
buf = ctypes.create_string_buffer(1024)
tm = struct.pack('9i', 2019, 5, 6, 9, 50, 4, 0, 126, 1)
print('count:', libc.strftime(buf, 1024, b'%Z', tm))
print('value:', buf.value)
wbuf = ctypes.create_unicode_buffer(1024)
print('count:', libc.wcsftime(wbuf, 1024, '%Z', tm))
print('value:', wbuf.value)
print('count:', libc.mbstowcs(wbuf, buf, 1024))
print('value:', wbuf.value)

count: 28
value: b'Mitteleurop\xe4ische Sommerzeit'
count: 28
value: Mitteleuropäische Sommerzeit
count: 28
value: Mitteleuropäische Sommerzeit

> libc = ctypes.cdll.msvcrt

That's the private CRT of Windows, not the Universal CRT for applications. In a release build (python.exe), use ctypes.CDLL('ucrtbase', use_errno=True). In a debug build (python_d.exe), use ctypes.CDLL('ucrtbased', use_errno=True).

I suppose we should use API sets [1] for the release build, such as "api-ms-win-crt-locale-l1-1-0" and 
"api-ms-win-crt-time-l1-1-0". But they resolve to "ucrtbase". 

[1]: https://docs.microsoft.com/en-us/uwp/win32-and-com/win32-extension-apis

Thanks for the reminder Eryk Sun.  This means the test needs to be run yet one more time :)

import ctypes, locale, struct
crt_time = ctypes.CDLL('api-ms-win-crt-time-l1-1-0', use_errno=True)
locale.setlocale(locale.LC_ALL, 'de_DE')
buf = ctypes.create_string_buffer(1024)
tm = struct.pack('9i', 2019, 5, 6, 9, 50, 4, 0, 126, 1)
print('count:', crt_time.strftime(buf, 1024, b'%Z', tm))
print('value:', buf.value)
wbuf = ctypes.create_unicode_buffer(1024)
print('count:', crt_time.wcsftime(wbuf, 1024, '%Z', tm))
print('value:', wbuf.value)
crt_convert = ctypes.CDLL('api-ms-win-crt-convert-l1-1-0', use_errno=True)
print('count:', crt_convert.mbstowcs(wbuf, buf, 1024))
print('value:', wbuf.value)

The code crashes on this line:
print('count:', crt_time.strftime(buf, 1024, b'%Z', tm))

Thanks for your patience with this Charlie, but please try another run this time without the strftime() and mbstowcs() calls.  Honest, we are getting closer!

If the process crashes at the first print statement, I'm not sure how I can run the tests. Or should I try them separately?

You can safely execute each line individually (omitting the aforementioned count/value pairs) or depending on how the copy/paste is being done, just paste the script into a text editor (notepad) and comment out those lines.  Then copy-paste that modified script.

This time I'm really attempting to see if the internal encoding/decoding done in the CRT strftime() is failing somehow (it calls wcsftime).

print('count:', crt_time.wcsftime(wbuf, 1024, '%Z', tm)) also fails
but
crt_convert = ctypes.CDLL('api-ms-win-crt-convert-l1-1-0', use_errno=True)
print('count:', crt_convert.mbstowcs(wbuf, buf, 1024))
seems to work okay.

Here is another test, this time removing Python from the equation (mostly :)

import ctypes, struct
crt_locale = ctypes.CDLL('api-ms-win-crt-locale-l1-1-0', use_errno=True)
crt_time = ctypes.CDLL('api-ms-win-crt-time-l1-1-0', use_errno=True)
crt_locale._wsetlocale.restype = ctypes.c_wchar_p
print('old locale:', crt_locale._wsetlocale(0, 0))
tm = struct.pack('9i', 2019, 5, 6, 9, 50, 4, 0, 126, 1)
wbuf = ctypes.create_unicode_buffer(1024)
print('count:', crt_time.wcsftime(wbuf, 1024, '%Z', tm))
print('value:', wbuf.value)
print('new locale:', crt_locale._wsetlocale(0, 'de_DE'))
print('count:', crt_time.wcsftime(wbuf, 1024, '%Z', tm))
print('value:', wbuf.value)

And this is the result.

old locale: C
count: 28
value: Mitteleuropäische Sommerzeit
new locale: de_DE
count: -1
value:
Windows fatal exception: code 0xc0000374

Looks like

print('new locale:', crt_locale._wsetlocale(0, 'de_DE'))
print('count:', crt_time.wcsftime(wbuf, 1024, '%Z', tm))

is the culprit.

Final test, this time, no Python what so ever.  I've added a zip containing a simple C program (source and .exe) that performs the same test.

The output should be similar to:

--------
The current locale is now: C
The time zone is: 'Mountain Daylight Time' (22 characters)
--------
The updated locale is now: de_DE
The time zone is: 'Mountain Daylight Time' (22 characters)
--------

If this crashes, the problem is then within the CRT itself :(

This is the result
\issue36792>test.exe
--------
The current locale is now: C
The time zone is: 'Mitteleuropõische Sommerzeit' (28 characters)
--------
The updated locale is now: de_DE
The time zone is: '' (-1 characters)
--------
NB something is wrong with that string. Should be Mitteleuropäsiche Sommerzeit but that could just be an encoding thing.

Thanks again!  I will have some more tests for you to try tomorrow as I am out of time for today.

I'm currently of the belief that there is something Python is going to have to do to work around an issue within the CRT, but more testing will prove that theory.

I've added another test executable (issue36792-2.zip) which should bring some insight into where things are going wrong.  Please run and post the results.

I have managed to setup a VM that can reproduce the error.  Unfortunately, the error (heap corruption) is coming from within the UCRT.  Attempting to work around that, I came across another error in the UCRT.

Due to these errors in all available UCRT versions, the only available solution is to re-implement %Z handling in Python's implementation of strftime().

@steve.dower: If you could forward the following information to the interested parties for faster response, it would be great.

Everyone else, what follows contains UCRT internal details which is of no interest to Python.

In wcsftime() [ucrt\time\wcsftime.cpp, line 1018], the call to _mbstowcs_s_l() doesn't handle the error case of EILSEQ.  This error happens when the tzname (originally set by __tzset() from the Unicode name in the registry encoded to the ACP) contains an undecodable character.  In the 'de_DE' case, \u00E4.  _mbstowcs_s_l returns without changing `wnum` (which is 0).  The 'Z' case then blindly subtracts 1 (wnum=-1) and advances the string pointer, now pointing *before* the start when the format is just L"%Z" (like Python has).  Now when wcsftime() is returning it stores a '\0' at that current position thus causing a heap corruption.

Ok, no deal breaker, we can just update the _tzname array when we call setlocale().  Alas, another error :(  If _tzset() is called a second time, it fails when encoding the Unicode name due to an uninitialized variable.  In tzset_from_system_nolock() [ucrt\time\tzset.cpp, line 158], the function __acrt_WideCharToMultiByte() [appcrt\locale\widechartomultibyte.cpp] does not set the `lpUsedDefaultChar` flag when FALSE.  This is only an issue, it seems, on subsequent calls as `used_default_char` can be non-zero prior to the call.  A non-zero value there, causes the tzname to be set to the empty string even if the conversion succeeded.

I've received a detailed response from the UCRT team, and there are a few pieces here.

* the fact that tzname is cached in ACP is known and will be fixed
* the decoding bug is real, but it's due to the experimental UTF-8 support
* the experimental UTF-8 support was enabled because "de_DE" is not a known Windows locale name - try with "de-DE"

Perhaps it would be easy to do the replacement of underscores with hyphens on Windows in this function? I think that's safe enough, yes?

I can confirm that using "de-DE" does indeed avoid the crash.

> * the experimental UTF-8 support was enabled because "de_DE" is not a
> known Windows locale name - try with "de-DE"
>
> Perhaps it would be easy to do the replacement of underscores with hyphens
> on Windows in this function? I think that's safe enough, yes?
>

Even some well known locale names still use the utf-8 code page.  Most seem
to uncommon, but at least es-BR (Brazil) does and would still fall victim
to these UCRT bugs.

>

> Perhaps it would be easy to do the replacement of underscores with 
> hyphens on Windows in this function? I think that's safe enough, yes?

Recent releases of ucrt implement this translation from underscore to hyphen for us, so this suggestion is no longer always necessary. However, I don't know what the PEP-11 stance is regarding the modern lifecycle policy of Windows 10. Will Python support a release for as long as it's supported by the enterprise version? For example, support for 1709 enterprise ends on 2020-04-14, so if we followed that, then Python 3.9 would require Windows 10 1803 or higher. That seems wrong while we're still supporting Windows 8.1, but what will the stance be when Python supports only Windows versions that use the modern lifecycle?

> Even some well known locale names still use the utf-8 code page.  Most 
> seem to uncommon, but at least es-BR (Brazil) does and would still 
> fall victim to these UCRT bugs.

es-BR is a custom locale for the Spanish language in Brazil, as opposed to the common Portuguese locale (pt-BR). It's a Unicode-only locale, which means its ANSI codepage is 0. Since 0 is CP_ACP, its effective ANSI codepage is the system or process ANSI codepage. 

For example:

    >>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    >>> buf = (ctypes.c_wchar * 10)()

Portuguese in Brazil uses codepage 1252 as its ANSI codepage:

    >>> n = kernel32.GetLocaleInfoEx('pt-BR', 0x1004, buf, 10)
    >>> buf.value
    '1252'

Spanish in Brazil uses CP_ACP:

    >>> n = kernel32.GetLocaleInfoEx('es-BR', 0x1004, buf, 10)
    >>> buf.value
    '0'

hi-IN (Hindi, India) is a common Unicode-only locale:

    >>> n = kernel32.GetLocaleInfoEx('hi-IN', 0x1004, buf, 10)
    >>> buf.value
    '0'

ucrt has switched to using UTF-8 for Unicode-only locales:

    >>> locale.setlocale(locale.LC_CTYPE, 'hi_IN')
    'hi_IN'
    >>> ucrt = ctypes.CDLL('ucrtbase', use_errno=True)
    >>> ucrt.___lc_codepage_func()
    65001

Note that ucrt uses UTF-8 for Unicode-only locales only when using an explicitly named locale such as "hi_IN", "Hindi_India" or even just "Hindi". On the other hand, if a Unicode-only locale is used implicitly, ucrt instead uses the system ANSI codepage:

    >>> locale.setlocale(locale.LC_CTYPE, '')
    'Hindi_India.1252'
    >>> ucrt.___lc_codepage_func()
    1252

I suppose this is for backwards compatibility. Windows 10 at least supports setting the system ANSI codepage to UTF-8, or overriding the process ANSI codepage to UTF-8 via the application manifest "actveCodePage" setting. For the latter, I modified the manifest in a "python_utf8.exe" copy of the normal "python.exe" binary, which is simpler than having to reboot to change the system locale:

    C:\>python_utf8 -q

    >>> import locale
    >>> locale.setlocale(locale.LC_CTYPE, '')
    'Hindi_India.utf8'

That the CRT caches the tzname strings as ANSI multibyte strings is frustrating -- whether or not it's buggy. I would expect there to be a _wtzname cache of the native OS strings that wcsftime uses directly, with no potential for failed encodings (e.g. empty strings or mojibake).

It's also strange that it encodes the time-zone name using the system ANSI codepage in the C locale. Normally LC_CTYPE in the C locale uses Latin-1, due to simple casting between WCHAR and CHAR. This leads to mojibake when the ANSI time-zone name gets decoded as Latin-1 by an internal mbstowcs call in wcsftime. I'm not saying one or the other is necessarily right, but more care should haven gone into this. At the very least, if we're stuck with system ANSI tzname strings in the C locale, then a flag should be set that tells wcsftime to decode them as system ANSI strings instead of via mbstowcs.

Also, the timezone name is determined by the preferred UI language of the current user, which is not necessarily compatible with the system ANSI codepage. It's not even necessarily compatible with the user-locale ANSI codepage, as used by setlocale(LC_CTYPE, ""). Windows 10 at least provides an option to sync the user locale with the user preferred UI language. IMO, this is a strong argument in favor of using _wtzname wide-character strings. UI Language (MUI) and locale are not tightly coupled in Windows NLS.

Here's an example where the user's preferred language is Hindi, and the time zone name is "समन्वित वैश्विक समय" (i.e. Coordinated Universal Time), but the system locale is English with codepage 1252 (for Western European languages). This is a normal configuration if the system locale doesn't have beta UTF-8 support enabled, or if the process ANSI codepage isn't overridden to UTF-8 via the "activeCodePage" manifest setting.

The tzname strings normally get set by a one-time _tzset call, and they're only reset if tzset is called manually. tzset uses the system ANSI encoding if LC_CTYPE is the "C" locale (again, normally ucrt uses Latin-1 in the "C" locale). Since the encoding of the Hindi timezone name to codepage 1252 contains the default character ("?"), which is not allowed, tzset sets the tzname strings to empty strings.

    import ctypes, locale, time
    ucrt = ctypes.CDLL('ucrtbase', use_errno=True)
    ucrt.__tzname.restype = ctypes.POINTER(ctypes.c_char_p)
    tzname = ucrt.__tzname()

    >>> locale.setlocale(locale.LC_CTYPE, 'C')
    'C'
    >>> ucrt._tzset()
    0
    >>> tzname[0], tzname[1]
    (b'', b'')
    >>> time.strftime('%Z')
    ''

If we update the LC_CTYPE category to use UTF-8, the cached tzname value doesn't get automatically updated, and strftime still returns an empty string:

    >>> locale.setlocale(locale.LC_CTYPE, '.utf8')
    'Hindi_India.utf8'

    >>> tzname[0], tzname[1]
    (b'', b'')
    >>> time.strftime('%Z')
    ''

The tzname values get updated if we manually call tzset:

    >>> ucrt._tzset()
    0
    >>> tzname[0].decode('utf-8'), tzname[1].decode('utf-8')
    ('समन्वित वैश्विक समय', 'समन्वित वैश्विक समय')

However, LC_TIME is still in the "C" locale. strftime uses system ANSI (1252) in this case, so the encoded result from the CRT strftime call ends up using the default character (?):

    >>> time.strftime('%Z')
    '??????? ??????? ???'

If we set LC_TIME to UTF-8, we finally get a valid result:

    >>> locale.setlocale(locale.LC_TIME, '.utf8')
    'Hindi_India.utf8'
    >>> time.strftime('%Z')
    'समन्वित वैश्विक समय'

We wouldn't have to worry about LC_TIME here if Python called C wcsftime instead of C strftime. The problem that bpo-10653 was trying to work around is a design flaw in the C runtime library, and calling strftime is not a solution. 

Here's a variation on my example in msg243660, continuing with the current Hindi example. The setup in this example uses UTF-8 as the system ANSI codepage (via python_utf8.exe) and sets LC_CTYPE to the "C" locale. This yields the following monstrosity:

    >>> time.strftime('%Z')
    'Ã\xa0¤¸Ã\xa0¤®Ã\xa0¤¨Ã\xa0Â¥Â\x8dÃ\xa0¤µÃ\xa0¤¿Ã\xa0¤¤ Ã\xa0¤µÃ\xa0Â¥Â\x88Ã\xa0¤¶Ã\xa0Â¥Â\x8dÃ\xa0¤µÃ\xa0¤¿Ã\xa0¤Â\x95 Ã\xa0¤¸Ã\xa0¤®Ã\xa0¤¯'

It's due to the following sequence of encoding and decoding operations:

    >>> mbs_lcctype_utf8 = 'समन्वित वैश्विक समय'.encode('utf-8')
    >>> wcs_lcctype_latin1 = mbs_lcctype_utf8.decode('latin-1')
    >>> mbs_lctime_utf8 = wcs_lcctype_latin1.encode('utf-8')

This last one is from PyUnicode_DecodeLocaleAndSize and mbstowcs:

    >>> py_str_lcctype_latin1 = mbs_lctime_utf8.decode('latin-1')
    >>> py_str_lcctype_latin1 == time.strftime('%Z')
    True

> Will Python support a release for as long as it's supported by the enterprise version? For example, support for 1709 enterprise ends on 2020-04-14, so if we followed that, then Python 3.9 would require Windows 10 1803 or higher. That seems wrong while we're still supporting Windows 8.1, but what will the stance be when Python supports only Windows versions that use the modern lifecycle?

This needs a separate discussion on python-dev, but for issues like this I think it's fine to blame the underlying platform and recommend getting their updates. We don't have to work around every Windows bug for all time.

I'll read the rest of what's been happening on this issue when I'm less tired, but not quite up to it right now ;)