test_unaligned_buffers (test.test_hash.HashEqualityTestCase) ... Fatal Python error: Bus error #67974
Comments
|
I compiled Python 3.4.3 on Solaris 11, and when I ran the regression test, I get a core dump when the hash() function was being used. I went through the bug database looking for something similar but couldn't find anything. Tests like: Current thread 0x00000001 (most recent call first): and test_hash_equality (test.datetimetester.TestTime_Fast) ... Fatal Python error: Bus error Current thread 0x00000001 (most recent call first): I then ran the same test through gdb: (gdb) list Further info: $ gcc --verbose
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/sparc-sun-solaris2.11/4.6.2/lto-wrapper
Target: sparc-sun-solaris2.11
Configured with: ../gcc-4.6.2/configure --prefix=/usr/local --enable-languages=c,c++ --disable-nls --with-gnu-as --with-gnu-ld --target=sparc-sun-solaris2.11
Thread model: posix
gcc version 4.6.2 (GCC) |
petriborg
mannequin
added
interpreter-core
|
I went and recompiled with: But this crashed as well. test_unaligned_buffers (test.test_hash.HashEqualityTestCase) ... Fatal Python error: Bus error Current thread 0x00000001 (most recent call first): test_unaligned_buffers (test.test_hash.HashEqualityTestCase) ... (gdb) list |
|
Siphash24 implementation is not designed to work on platforms that require aligned access. But I'm surprised that fnv implementation crashes. I don't see anything wrong. May be gcc needs some special options to produce correct binaries on this platform? |
|
I've compiled Python 3.3.6 using the same options (./configure --prefix=/usr/local --enable-shared) and build system and that passes almost all the tests (test_uuid fails for an ignorable reason). Specifically test_hash passes fully: $ LD_LIBRARY_PATH=/usr/local/src/Python-3.3.6 ./python -m test -v test_hash
== CPython 3.3.6 (default, Mar 26 2015, 15:35:36) [GCC 4.6.2]
== Solaris-2.11-sun4v-sparc-32bit-ELF big-endian
== /usr/local/src/Python-3.3.6/build/test_python_4539
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1)
[1/1] test_hash
test_empty_string (test.test_hash.BytesHashRandomizationTests) ... ok
test_fixed_hash (test.test_hash.BytesHashRandomizationTests) ... ok
test_null_hash (test.test_hash.BytesHashRandomizationTests) ... ok
test_randomized_hash (test.test_hash.BytesHashRandomizationTests) ... ok
test_randomized_hash (test.test_hash.DatetimeDateTests) ... ok
test_randomized_hash (test.test_hash.DatetimeDatetimeTests) ... ok
test_randomized_hash (test.test_hash.DatetimeTimeTests) ... ok
test_hashes (test.test_hash.HashBuiltinsTestCase) ... ok
test_coerced_floats (test.test_hash.HashEqualityTestCase) ... ok
test_coerced_integers (test.test_hash.HashEqualityTestCase) ... ok
test_numeric_literals (test.test_hash.HashEqualityTestCase) ... ok
test_unaligned_buffers (test.test_hash.HashEqualityTestCase) ... ok
test_default_hash (test.test_hash.HashInheritanceTestCase) ... ok
test_error_hash (test.test_hash.HashInheritanceTestCase) ... ok
test_fixed_hash (test.test_hash.HashInheritanceTestCase) ... ok
test_hashable (test.test_hash.HashInheritanceTestCase) ... ok
test_not_hashable (test.test_hash.HashInheritanceTestCase) ... ok
test_empty_string (test.test_hash.MemoryviewHashRandomizationTests) ... ok
test_fixed_hash (test.test_hash.MemoryviewHashRandomizationTests) ... ok
test_null_hash (test.test_hash.MemoryviewHashRandomizationTests) ... ok
test_randomized_hash (test.test_hash.MemoryviewHashRandomizationTests) ... ok
test_empty_string (test.test_hash.StrHashRandomizationTests) ... ok
test_fixed_hash (test.test_hash.StrHashRandomizationTests) ... ok
test_null_hash (test.test_hash.StrHashRandomizationTests) ... ok
test_randomized_hash (test.test_hash.StrHashRandomizationTests) ... okRan 25 tests in 1.356s OK So any ideas what I should look for? Or perhaps it would be helpful if I posted config.log, etc. |
|
Yes, 3.3 uses less efficient implementation. Try to compile Python with gcc option -mno-unaligned-access. |
|
That's not a valid option on SPARC, (see https://gcc.gnu.org/onlinedocs/gcc/SPARC-Options.html ) the flag is only available on ARM it seems. |
|
I'm puzzled about the segfault. I'm pretty sure that I've compiled and tested the code on big endian machines as well as a SPARC machines that doesn't allow unaligned memory access. The FNV code copies the blocks to an aligned buffer to prevent exactly this bug. Serhiy spent days to optimize the code as humanly possible. Please recompile Python again. This time create a debug build (--with-pydebug) and use the default hash function FNV for your platform. The debug build helps to pin point the bug. |
|
What if add double field in the block union? (And may be compile with -munaligned-doubles) diff -r a417d89fbc38 Python/pyhash.c
--- a/Python/pyhash.c Thu Mar 26 09:37:23 2015 +0100
+++ b/Python/pyhash.c Fri Mar 27 11:52:25 2015 +0200
@@ -247,6 +247,7 @@ fnv(const void *src, Py_ssize_t len)
union {
Py_uhash_t value;
unsigned char bytes[SIZEOF_PY_UHASH_T];
+ double double_value;
} block;
#ifdef Py_DEBUGWhat if add the __aligned__ attribute? diff -r a417d89fbc38 Python/pyhash.c
--- a/Python/pyhash.c Thu Mar 26 09:37:23 2015 +0100
+++ b/Python/pyhash.c Fri Mar 27 11:58:30 2015 +0200
@@ -247,7 +247,7 @@ fnv(const void *src, Py_ssize_t len)
union {
Py_uhash_t value;
unsigned char bytes[SIZEOF_PY_UHASH_T];
- } block;
+ } block __attribute__ ((__aligned__(SIZEOF_PY_UHASH_T)));
#ifdef Py_DEBUG
assert(_Py_HashSecret_Initialized); |
|
Or may be better to use explicit alignment 8 if this is 32-bit platform. diff -r a417d89fbc38 Python/pyhash.c
--- a/Python/pyhash.c Thu Mar 26 09:37:23 2015 +0100
+++ b/Python/pyhash.c Fri Mar 27 12:02:55 2015 +0200
@@ -247,7 +247,7 @@ fnv(const void *src, Py_ssize_t len)
union {
Py_uhash_t value;
unsigned char bytes[SIZEOF_PY_UHASH_T];
- } block;
+ } block __attribute__ ((__aligned__(8)));
#ifdef Py_DEBUG
assert(_Py_HashSecret_Initialized); |
|
The union trick is not portable, using memcpy() is safer. Compilers should be able to optimize it. Use Py_MEMCPY() for the stupid Visual Studio compiler unable to optimize memcpy() for small sizes. |
|
AFAIK it conforms the C standard and should be portable (but weird compiler bugs or misconfiguration can make it broken). With Py_MEMCPY() the code is less optimal. Don't add regression on common platforms. |
|
OK I recompiled with "./configure --prefix=/usr/local --enable-shared --with-pydebug" and reran the test, unfortunately... $ LD_LIBRARY_PATH=/usr/local/src/Python-3.4.3 ./python -m test test_hash
[1/1] test_hash
1 test OK.I then applied the patch in msg239385, this resulted in the same crash, I couldn't see any difference in gdb. I then tried each of the patches in msg239384, with -munaligned-doubles and with -mnounaligned-doubles, again no difference. I did notice that two tests failed in test_hash prior to the core dump, so here are the outputs from those in case that helps. <3.4.3 ./python -m test -v test_hash.DatetimeDatetimeTests test_hash.DatetimeTimeTests
== CPython 3.4.3 (default, Mar 27 2015, 08:45:04) [GCC 4.6.2]
== Solaris-2.11-sun4v-sparc-32bit-ELF big-endian
== hash algorithm: fnv 32bit
== /usr/local/src/Python-3.4.3/build/test_python_10340
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[1/2] test_hash.DatetimeDatetimeTests
test test_hash.DatetimeDatetimeTests crashed -- Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 2218, in _find_and_load_unlocked
AttributeError: 'module' object has no attribute '__path__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/src/Python-3.4.3/Lib/test/regrtest.py", line 1271, in runtest_inner
the_module = importlib.import_module(abstest)
File "/usr/local/src/Python-3.4.3/Lib/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2221, in _find_and_load_unlocked
ImportError: No module named 'test.test_hash.DatetimeDatetimeTests'; 'test.test_hash' is not a package
[2/2/1] test_hash.DatetimeTimeTests
test test_hash.DatetimeTimeTests crashed -- Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 2218, in _find_and_load_unlocked
AttributeError: 'module' object has no attribute '__path__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/src/Python-3.4.3/Lib/test/regrtest.py", line 1271, in runtest_inner
the_module = importlib.import_module(abstest)
File "/usr/local/src/Python-3.4.3/Lib/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2221, in _find_and_load_unlocked
ImportError: No module named 'test.test_hash.DatetimeTimeTests'; 'test.test_hash' is not a package2 tests failed: Thanks guys! |
|
Sorry I copied the wrong term buffer :-) LD_LIBRARY_PATH=/usr/local/src/Python-3.4.3 ./python -m test -v test_hash ====================================================================== Traceback (most recent call last):
File "/usr/local/src/Python-3.4.3/Lib/test/test_hash.py", line 193, in test_randomized_hash
run1 = self.get_hash(self.repr_, seed='random')
File "/usr/local/src/Python-3.4.3/Lib/test/test_hash.py", line 187, in get_hash
**env)
File "/usr/local/src/Python-3.4.3/Lib/test/script_helper.py", line 106, in assert_python_ok
return _assert_python(True, *args, **env_vars)
File "/usr/local/src/Python-3.4.3/Lib/test/script_helper.py", line 92, in _assert_python
err.decode('ascii', 'ignore')))
AssertionError: Process return code is -10, command line was: ['/usr/local/src/Python-3.4.3/python', '-X', 'faulthandle
r', '-c', 'import datetime; print(hash(datetime.datetime(1, 2, 3, 4, 5, 6, 7)))'], stderr follows:
Fatal Python error: Bus errorCurrent thread 0x00000001 (most recent call first): ====================================================================== Traceback (most recent call last):
File "/usr/local/src/Python-3.4.3/Lib/test/test_hash.py", line 193, in test_randomized_hash
run1 = self.get_hash(self.repr_, seed='random')
File "/usr/local/src/Python-3.4.3/Lib/test/test_hash.py", line 187, in get_hash
**env)
File "/usr/local/src/Python-3.4.3/Lib/test/script_helper.py", line 106, in assert_python_ok
return _assert_python(True, *args, **env_vars)
File "/usr/local/src/Python-3.4.3/Lib/test/script_helper.py", line 92, in _assert_python
err.decode('ascii', 'ignore')))
AssertionError: Process return code is -10, command line was: ['/usr/local/src/Python-3.4.3/python', '-X', 'faulthandle
r', '-c', 'import datetime; print(hash(datetime.time(0, 0)))'], stderr follows:
Fatal Python error: Bus errorCurrent thread 0x00000001 (most recent call first): ---------------------------------------------------------------------- FAILED (failures=2) |
|
Hi haypo, I just realized you had created a patch too, the fnv_memcpy.patch worked! $ LD_LIBRARY_PATH=/usr/local/src/Python-3.4.3 ./python -m test test_hash
[1/1] test_hash
1 test OK.Running the full regression test now, but I bet everything passes. |
|
So this morning I got around to rebuilding the tool chain using GCC 4.9.2 and I'm happy to report that this problem goes away (no patch required)! So it must be some sort of problem with the 4.6 GCC. I've still got the old tool chain around, and I'm happy to further patch / test if anyone at Python wants me to. $ gcc --verbose
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/sparc-sun-solaris2.11/4.9.2/lto-wrapper
Target: sparc-sun-solaris2.11
Configured with: ../gcc-4.9.2/configure --prefix=/usr/local --enable-languages=c,c++ --disable-nls --with-gnu-as --with-gnu-ld --target=sparc-sun-solaris2.11
Thread model: posix
gcc version 4.9.2 (GCC) |
|
Just interesting, what is the result of str(memoryview(b'abcdefghijklmnopqrstuvwxyz')[1:], 'ascii') with old and new toolchains. |
Test 1
Python 3.4.3 built by GCC 4.9.2 is:
>>> str(memoryview(b'abcdefghijklmnopqrstuvwxyz')[1:], 'ascii')
'bcdefghijklmnopqrstuvwxyz'
Test 2
Python 3.4.3 built by GCC 4.6.2 is (no patches applied)
This build will core dump if I run -m test test_hash.
>>> str(memoryview(b'abcdefghijklmnopqrstuvwxyz')[1:], 'ascii')
'bcdefghijklmnopqrstuvwxyz' |
|
Can this be close (as third party?) or is there anything left to do? |
|
We've migrated our python process off Solaris. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: