This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author asturm
Recipients asturm
Date 2018-04-05.14:49:04
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1522939744.33.0.682650639539.issue33232@psf.upfronthosting.co.za>
In-reply-to
Content
This issue can be reproduced using Python 3.5.2 - 3.5.5 compiled with VS2015 (tested both the official Python builds as well as local builds using VS2015 Version 14.0.25431.01 Update 3) on 64-bit Windows.

To reproduce, run the attached file (or python -c "import operator; operator.attrgetter('x'*1000000)").

The segfault itself occurs in attrgetter_new [1], while scanning for dots inside a unicode string. From my reading of the code, the code itself is actually correct - instead, it triggers a compiler bug in VS2015, which causes it to miscompile the code by applying an incorrect optimization.

The optimization VS2015 applies is twofold: It uses SIMD instructions to avoid looking at every byte of the input string individually, and it tries to eliminate the condition in PyUnicode_READ [2] by executing all three branches (UCS1, UCS2 and UCS4) at once and then in a separate step discarding the results for the two branches that would not be taken.

When combining these optimizations, the generated code incorrectly only checks the precondition of the UCS1 branch to decide whether it can use the optimized SIMD version. Since the UCS1 branch looks at 8 bytes of the buffer in each iteration, it checks whether the buffer contains at least 8 bytes. However, each iteration of the UCS2/UCS4 branches look at 16/32 bytes, respectively.

As a result, when passing a UCS1 string to attrgetter(), we end up reading 24 bytes past the end of the buffer for every 8 bytes the string contains. In the reproduction example, we thus read about 24 MB past the buffer, almost guaranteeing a crash.

The issue does not occur when compiling Python using VS2017 (i.e. PlatformToolset v141, MSVC version 14.13.26128). Presumably this means the compiler bug has been fixed upstream.

[1]: https://github.com/python/cpython/blob/v3.5.5/Modules/_operator.c#L587
[2]: https://github.com/python/cpython/blob/v3.5.5/Include/unicodeobject.h#L521
History
Date User Action Args
2018-04-05 14:49:04asturmsetrecipients: + asturm
2018-04-05 14:49:04asturmsetmessageid: <1522939744.33.0.682650639539.issue33232@psf.upfronthosting.co.za>
2018-04-05 14:49:04asturmlinkissue33232 messages
2018-04-05 14:49:04asturmcreate