classification
Title: Undefined behavior flagged by Clang 3.4 (Python 3.5 from hg)
Type: compile error Stage:
Components: Build Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Jeffrey.Walton, benjamin.peterson, martin.panter, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-03-15 02:37 by Jeffrey.Walton, last changed 2016-07-20 13:27 by martin.panter.

Files
File name Uploaded Description Edit
python-3.5-make.txt Jeffrey.Walton, 2014-03-15 02:37 Results of 'make' with Clang's undefined behavior sanitizer.
python-3.5-make-test.txt Jeffrey.Walton, 2014-03-15 02:38 Results of 'make test' with Clang's undefined behavior sanitizer.
audioop_signed_left_shift.patch serhiy.storchaka, 2014-03-18 18:54 review
Messages (8)
msg213625 - (view) Author: Jeffrey Walton (Jeffrey.Walton) * Date: 2014-03-15 02:37
Checked out Python-3.5 from mercurial. Compiled with Clang 3.4, added -fsanitzie=undefined to CFLAGS. Ran 'make' and 'make check'. A few issues during `make`:

Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
Objects/listobject.c:2046:22: runtime error: index 623 out of bounds for type 'PyObject *[256]'
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'

Similar issues with `make test`, with the addition of some mis-aligned pointers. A digest is below, and the tagCDataObject alignment issue dominates the issues. The tests ended early due to a segfault.

$ make test
...
[ 19/389/1] test_struct
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
/home/jwalton/Desktop/cpython-checkout/Objects/floatobject.c:2028: runtime error: value 3.40282e+38 is outside the range of representable values of type 'float'
/home/jwalton/Desktop/cpython-checkout/Modules/_struct.c:792:15: runtime error: left shift of 72057594037927935 by 8 places cannot be represented in type 'long long'
...
[ 43/389/2] test_cmd_line
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
/home/jwalton/Desktop/cpython-checkout/Modules/_ctypes/_ctypes.c:2890:10: runtime error: member access within misaligned address 0x2b54033f1cc8 for type 'CDataObject' (aka 'struct tagCDataObject'), which requires 16 byte alignment
0x2b54033f1cc8: note: pointer points here
 ff ff ff ff  01 00 00 00 00 00 00 00  68 06 43 03 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 
test test_cmd_line failed -- multiple errors occurred; run in verbose mode for details
...
[ 45/389/2] test_datetime
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
/home/jwalton/Desktop/cpython-checkout/Python/pytime.c:218: runtime error: value -1e+200 is outside the range of representable values of type 'long'
/home/jwalton/Desktop/cpython-checkout/Python/pytime.c:218: runtime error: value 1e+200 is outside the range of representable values of type 'long'
...
test test_traceback failed -- Traceback (most recent call last):
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/test_traceback.py", line 155, in test_encoded_file
    do_test("", "foo", "ascii", 3)
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/test_traceback.py", line 132, in do_test
    stdout = stdout.decode(output_encoding).splitlines()
LookupError: unknown encoding: Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
...
[130/389/5] test_capi
Objects/longobject.c:40:42: runtime error: index -3 out of bounds for type 'PyLongObject [262]'
Fatal Python error: Segmentation fault

Current thread 0x00002b6eaf8c7b20 (most recent call first):
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/test_capi.py", line 466 in test__testcapi
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/case.py", line 574 in run
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/case.py", line 622 in __call__
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/suite.py", line 125 in run
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/suite.py", line 87 in __call__
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/suite.py", line 125 in run
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/suite.py", line 87 in __call__
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/suite.py", line 125 in run
  File "/home/jwalton/Desktop/cpython-checkout/Lib/unittest/suite.py", line 87 in __call__
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/support/__init__.py", line 1584 in run
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/support/__init__.py", line 1685 in _run_suite
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/support/__init__.py", line 1719 in run_unittest
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 1277 in <lambda>
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 1278 in runtest_inner
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 978 in runtest
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 532 in main
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 1562 in main_in_temp_cwd
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 1587 in <module>
  File "/home/jwalton/Desktop/cpython-checkout/Lib/runpy.py", line 86 in _run_code
  File "/home/jwalton/Desktop/cpython-checkout/Lib/runpy.py", line 171 in _run_module_as_main
Traceback (most recent call last):
  File "/home/jwalton/Desktop/cpython-checkout/Lib/runpy.py", line 171, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/jwalton/Desktop/cpython-checkout/Lib/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 1562, in main_in_temp_cwd
    main()
  File "/home/jwalton/Desktop/cpython-checkout/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_capi: Exit code -11
make: *** [test] Error 1
msg213630 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-03-15 03:34
This is just the same as #20929, I believe?
msg213631 - (view) Author: Jeffrey Walton (Jeffrey.Walton) * Date: 2014-03-15 03:53
Pulled the latest after BP cleared some more runtime errors:

  $ hg summary
  parent: 89662:7ce22d0899e4 tip
   merge 3.3
  branch: default
  commit: 2 modified
  update: (current)

Misaligned accesses dominate with over 11K:

  $ cat python-3.5-make-test.txt | grep misaligned | wc -l
  11320

*****

$ make test
...

[ 21/389] test_sort
Objects/listobject.c:1973:30: runtime error: index 257 out of bounds for type 'PyObject *[256]'
...
[ 31/389] test_xml_etree
/home/jwalton/Desktop/cpython-checkout/Modules/expat/xmltok.c:1396:11: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
/home/jwalton/Desktop/cpython-checkout/Modules/expat/xmltok.c:1398:16: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
...
[ 33/389] test_aifc
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1587:39: runtime error: left shift of negative value -24
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1546:19: runtime error: left shift of negative value -24
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1513:43: runtime error: left shift of negative value -24
...
[ 61/389] test_httplib
/home/jwalton/Desktop/cpython-checkout/Lib/socket.py:444: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.16.1.26', 51321), raddr=('82.94.164.164', 443)>
  self._sock = None
...
[ 99/389] test_audioop
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1060:20: runtime error: left shift of negative value -70
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1061:20: runtime error: left shift of negative value -70
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
...
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:422:11: runtime error: left shift of negative value -1
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1639:19: runtime error: left shift of negative value -69
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1639:19: runtime error: left shift of negative value -17767
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1639:19: runtime error: left shift of negative value -70
/home/jwalton/Desktop/cpython-checkout/Modules/audioop.c:1639:19: runtime error: left shift of negative value -4548489
...
[109/389] test_unicode
/home/jwalton/Desktop/cpython-checkout/Modules/_ctypes/_ctypes.c:2890:10: runtime error: member access within misaligned address 0x2ac5068c24f8 for type 'CDataObject' (aka 'struct tagCDataObject'), which requires 16 byte alignment
0x2ac5068c24f8: note: pointer points here
 ff ff ff ff  01 00 00 00 00 00 00 00  78 e4 af 02 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 
/home/jwalton/Desktop/cpython-checkout/Modules/_ctypes/_ctypes.c:2891:10: runtime error: member access within misaligned address 0x2ac5068c24f8 for type 'CDataObject' (aka 'struct tagCDataObject'), which requires 16 byte alignment
0x2ac5068c24f8: note: pointer points here
 ff ff ff ff  01 00 00 00 00 00 00 00  78 e4 af 02 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 
...
/home/jwalton/Desktop/cpython-checkout/Modules/_ctypes/_ctypes.c:3862:5: runtime error: member access within misaligned address 0x2ae9f1252e58 for type 'PyCFuncPtrObject', which requires 16 byte alignment
0x2ae9f1252e58: note: pointer points here
 00 00 00 00  01 00 00 00 00 00 00 00  58 0c 8e 02 00 00 00 00  a8 2e 25 f1 e9 2a 00 00  01 00 00 00
              ^ 
/home/jwalton/Desktop/cpython-checkout/Modules/_ctypes/_ctypes.c:3863:5: runtime error: member access within misaligned address 0x2ae9f1252e58 for type 'PyCFuncPtrObject', which requires 16 byte alignment
0x2ae9f1252e58: note: pointer points here
 00 00 00 00  01 00 00 00 00 00 00 00  58 0c 8e 02 00 00 00 00  a8 2e 25 f1 e9 2a 00 00  01 00 00 00
              ^ 
...
2 tests failed:
    test_faulthandler test_venv
2 tests altered the execution environment:
    test___all__ test_warnings
25 tests skipped:
    test_bz2 test_crypt test_curses test_dbm_gnu test_dbm_ndbm
    test_devpoll test_gdb test_gzip test_idle test_kqueue test_lzma
    test_msilib test_nis test_ossaudiodev test_readline test_sqlite
    test_startfile test_tcl test_tk test_ttk_guionly test_ttk_textonly
    test_winreg test_winsound test_zipfile64 test_zlib
msg213632 - (view) Author: Jeffrey Walton (Jeffrey.Walton) * Date: 2014-03-15 03:55
> This is just the same as #20929, I believe?

I think most of the issues in the 20929 report (Python 3.4-RC3) are present in this report. But under this report, I can re-test as you check in the fixes. (Python 3.4-RC3 is fixed in time from my perspective).
msg213634 - (view) Author: Jeffrey Walton (Jeffrey.Walton) * Date: 2014-03-15 04:22
Here's another one I missed. The first is a problem due to silent truncation when casting from the double 3.40282e+38 to a float (or Clang is wrong).

[ 10/389] test_struct
/home/jwalton/Desktop/cpython-checkout/Objects/floatobject.c:2028: runtime error: value 3.40282e+38 is outside the range of representable values of type 'float'
/home/jwalton/Desktop/cpython-checkout/Modules/_struct.c:792:15: runtime error: left shift of 72057594037927935 by 8 places cannot be represented in type 'long long'
msg214004 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-03-18 18:54
Please try this patch. Perhaps it fixes audioop issues.
msg214006 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-03-18 19:01
No, it still gives

/home/benjamin/dev/python/3.4/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/benjamin/dev/python/3.4/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/benjamin/dev/python/3.4/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/benjamin/dev/python/3.4/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/benjamin/dev/python/3.4/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/benjamin/dev/python/3.4/Modules/audioop.c:1811:9: runtime error: left shift of negative value -22
/home/benjamin/dev/python/3.4/Modules/audioop.c:1587:39: runtime error: left shift of negative value -5504
/home/benjamin/dev/python/3.4/Modules/audioop.c:1546:19: runtime error: left shift of negative value -5504
/home/benjamin/dev/python/3.4/Modules/audioop.c:1546:19: runtime error: left shift of negative value -1409024
/home/benjamin/dev/python/3.4/Modules/audioop.c:1639:19: runtime error: left shift of negative value -69
/home/benjamin/dev/python/3.4/Modules/audioop.c:1639:19: runtime error: left shift of negative value -17767
/home/benjamin/dev/python/3.4/Modules/audioop.c:1639:19: runtime error: left shift of negative value -4548489
/home/benjamin/dev/python/3.4/Modules/audioop.c:1546:19: runtime error: left shift of negative value -69
/home/benjamin/dev/python/3.4/Modules/audioop.c:1244:19: runtime error: left shift of negative value -69
/home/benjamin/dev/python/3.4/Modules/audioop.c:1244:19: runtime error: left shift of negative value -17767
/home/benjamin/dev/python/3.4/Modules/audioop.c:1244:19: runtime error: left shift of negative value -4548489
/home/benjamin/dev/python/3.4/Modules/audioop.c:1473:19: runtime error: left shift of negative value -69
/home/benjamin/dev/python/3.4/Modules/audioop.c:1473:19: runtime error: left shift of negative value -17767
/home/benjamin/dev/python/3.4/Modules/audioop.c:1473:19: runtime error: left shift of negative value -4548489
/home/benjamin/dev/python/3.4/Modules/audioop.c:1418:31: runtime error: left shift of negative value -69
/home/benjamin/dev/python/3.4/Modules/audioop.c:1418:31: runtime error: left shift of negative value -17767
/home/benjamin/dev/python/3.4/Modules/audioop.c:1418:31: runtime error: left shift of negative value -4548489
/home/benjamin/dev/python/3.4/Modules/audioop.c:1513:43: runtime error: left shift of negative value -32124

You, too, can build with the undefined behavior sanitizer using the helpful instructions at http://docs.python.org/devguide/clang.html
msg270870 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-20 13:27
Serhiy’s patch does fix about half the errors (1 whole screenful reduces to a bit over 1/2 a screenful). I do find it a bit ugly though (four casts in the one term of a macro!).
History
Date User Action Args
2016-07-20 13:27:29martin.pantersetmessages: + msg270870
2016-07-19 12:43:15serhiy.storchakasetnosy: + martin.panter
2014-03-18 19:01:30benjamin.petersonsetmessages: + msg214006
2014-03-18 18:54:24serhiy.storchakasetfiles: + audioop_signed_left_shift.patch

nosy: + serhiy.storchaka
messages: + msg214004

keywords: + patch
2014-03-15 04:22:17Jeffrey.Waltonsetmessages: + msg213634
2014-03-15 03:55:36Jeffrey.Waltonsetmessages: + msg213632
2014-03-15 03:53:10Jeffrey.Waltonsetmessages: + msg213631
2014-03-15 03:34:11benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg213630
2014-03-15 02:38:56Jeffrey.Waltonsetfiles: + python-3.5-make-test.txt
2014-03-15 02:37:32Jeffrey.Waltoncreate