classification
Title: Get the test suite passing with clang Memory Sanitizer enabled
Type: security Stage: patch review
Components: Build, Extension Modules, Interpreter Core, Tests Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: alex, benjamin.peterson, gregory.p.smith, izbyshev, miss-islington, pablogsal, serhiy.storchaka, twouters
Priority: normal Keywords: patch

Created on 2018-11-12 07:26 by gregory.p.smith, last changed 2018-12-31 06:14 by gregory.p.smith.

Pull Requests
URL Status Linked Edit
PR 10479 merged gregory.p.smith, 2018-11-12 07:29
PR 10492 merged gregory.p.smith, 2018-11-12 20:29
PR 10493 merged gregory.p.smith, 2018-11-12 21:58
PR 10499 merged gregory.p.smith, 2018-11-13 03:04
PR 10500 merged miss-islington, 2018-11-13 03:52
PR 10501 merged miss-islington, 2018-11-13 03:52
PR 10503 merged gregory.p.smith, 2018-11-13 05:43
PR 10504 merged miss-islington, 2018-11-13 06:01
PR 10505 merged miss-islington, 2018-11-13 06:01
PR 10506 merged gregory.p.smith, 2018-11-13 08:42
PR 10522 merged miss-islington, 2018-11-13 21:17
PR 10523 merged miss-islington, 2018-11-13 21:17
PR 10538 merged gregory.p.smith, 2018-11-14 01:27
PR 11375 merged gregory.p.smith, 2018-12-30 23:39
PR 11375 merged gregory.p.smith, 2018-12-30 23:39
PR 11375 merged gregory.p.smith, 2018-12-30 23:39
PR 11378 merged miss-islington, 2018-12-31 01:05
PR 11378 merged miss-islington, 2018-12-31 01:05
PR 11378 merged miss-islington, 2018-12-31 01:05
PR 11385 merged gregory.p.smith, 2018-12-31 02:58
PR 11385 merged gregory.p.smith, 2018-12-31 02:58
PR 11385 merged gregory.p.smith, 2018-12-31 02:58
PR 11388 merged miss-islington, 2018-12-31 04:18
PR 11388 merged miss-islington, 2018-12-31 04:18
PR 11388 merged miss-islington, 2018-12-31 04:18
PR 11389 merged gregory.p.smith, 2018-12-31 04:51
PR 11389 merged gregory.p.smith, 2018-12-31 04:51
PR 11389 merged gregory.p.smith, 2018-12-31 04:51
PR 11391 merged gregory.p.smith, 2018-12-31 05:53
PR 11391 merged gregory.p.smith, 2018-12-31 05:53
PR 11391 merged gregory.p.smith, 2018-12-31 05:53
Messages (26)
msg329723 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-12 07:26
clang's memory sanitizer (-fsanitize=memory) turns up useful problems in code.  I'm working on getting a CPython buildbot running it setup but would like our build to be cleaner to start with before I run that.

These are the initial fixes required for most of CPython to pass in an msan build.  We've been using these with our interpreters at Google.  (PR coming)
msg329750 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-12 20:07
New changeset 1584a0081500d35dc93ff88e5836df35faf3e3e2 by Gregory P. Smith in branch 'master':
bpo-35214: Initial clang MemorySanitizer support (GH-10479)
https://github.com/python/cpython/commit/1584a0081500d35dc93ff88e5836df35faf3e3e2
msg329759 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-12 21:48
New changeset 5f4d05d83fde05fed0b6554d0beee4f1c1f1d8f1 by Gregory P. Smith in branch '3.7':
[3.7] bpo-35214: Initial clang MemorySanitizer support (GH-10479) (GH-10492)
https://github.com/python/cpython/commit/5f4d05d83fde05fed0b6554d0beee4f1c1f1d8f1
msg329777 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-13 00:10
New changeset 3b5b1c0931cd32253bbf2f3bd74e90ff821e026e by Gregory P. Smith in branch '3.6':
[3.6] bpo-35214: Initial clang MemorySanitizer support (GH-10479) (GH-10493)
https://github.com/python/cpython/commit/3b5b1c0931cd32253bbf2f3bd74e90ff821e026e
msg329798 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-11-13 03:08
Can we prefix MEMORY_SANITIZER with _Py_?
msg329803 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-13 03:34
I cannot initialize the interpreter after compiling with --with-memory-sanitizer:

❯ CC=clang ./configure  --with-memory-sanitizer && make -j
❯ ./python
Python 3.8.0a0 (heads/master:1584a00815, Nov 13 2018, 03:29:18)
[Clang 7.0.0 (tags/RELEASE_700/final)] on linux
Type "help", "copyright", "credits" or "license" for more information.
==10989==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5592f18005c3  (/home/pablogsal/cpython/python+0x9a55c3)
    #1 0x5592f175c176  (/home/pablogsal/cpython/python+0x901176)
    #2 0x5592f17592da  (/home/pablogsal/cpython/python+0x8fe2da)
    #3 0x5592f1750f82  (/home/pablogsal/cpython/python+0x8f5f82)
    #4 0x5592f174a336  (/home/pablogsal/cpython/python+0x8ef336)
    #5 0x5592f174c906  (/home/pablogsal/cpython/python+0x8f1906)
    #6 0x5592f14ae214  (/home/pablogsal/cpython/python+0x653214)
    #7 0x5592f14a6915  (/home/pablogsal/cpython/python+0x64b915)
    #8 0x5592f14a293c  (/home/pablogsal/cpython/python+0x64793c)
    #9 0x5592f0f5ad88  (/home/pablogsal/cpython/python+0xffd88)
    #10 0x5592f0f5ce73  (/home/pablogsal/cpython/python+0x101e73)
    #11 0x5592f0f4d908  (/home/pablogsal/cpython/python+0xf2908)
    #12 0x7fd1a7381222  (/usr/lib/libc.so.6+0x24222)
    #13 0x5592f0ed3cdd  (/home/pablogsal/cpython/python+0x78cdd)

  Uninitialized value was created by a heap allocation
    #0 0x5592f0f02a0d  (/home/pablogsal/cpython/python+0xa7a0d)
    #1 0x7fd1a73cd790  (/usr/lib/libc.so.6+0x70790)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/pablogsal/github/cpython/python+0x9a55c3)
Exiting

❯ clang --version
clang version 7.0.0 (tags/RELEASE_700/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

❯ /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.28.
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 8.2.1 20180831.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.

I am missing something or is this expected (as there are more PRs to come)?
msg329804 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2018-11-13 03:37
All libraries that are linked against, including libc, need to be compiled with MSAN. MSAN is not for the faint of heart.
msg329805 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-13 03:39
Thanks @alex! I will try again after linking against ASAN-compiled version of my libraries.
msg329806 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-13 05:35
yeah, i've been surprised how far i was able to get from an oss CPython tree and pre-built clang 7 binary installation.  If you have headers installed for common libraries like libssl-dev and libreadline-dev you can't produce an interpreter that'll work as those both lead to quick crashes.  also, in my experience optimized builds tend to be more problematic (too early to say if what i'm seeing are actual problems).

also, make sure llvm-symbolize (no suffixes) is in your path for it to auto-symbolize the traces showing you where in the code it pointed out issues.  when it points within shared libraries instead of the Python source tree - that's a hint that the library probably may need msan compilation.

I'm plodding through things to see if I can get a _useful_ buildbot setup (i don't care of some extension modules can't be covered for now) out of all of this.
msg329807 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-13 05:44
> Can we prefix MEMORY_SANITIZER with _Py_?

Yes, I wondered if I should do that.  not that I expect anyone would ever define it to mean anything else, but that seems like the right thing to do.  https://github.com/python/cpython/pull/10503
msg329808 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-13 06:01
New changeset 3015fb8ce4d25603434b9b44bb7effb98a481532 by Gregory P. Smith in branch 'master':
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
https://github.com/python/cpython/commit/3015fb8ce4d25603434b9b44bb7effb98a481532
msg329809 - (view) Author: miss-islington (miss-islington) Date: 2018-11-13 06:20
New changeset f6602f92664b782125a1ec5009706f9300fe7e80 by Miss Islington (bot) in branch '3.7':
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
https://github.com/python/cpython/commit/f6602f92664b782125a1ec5009706f9300fe7e80
msg329810 - (view) Author: miss-islington (miss-islington) Date: 2018-11-13 06:27
New changeset 60cf26528b931215160a53fd61836dbe4f036009 by Miss Islington (bot) in branch '3.6':
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
https://github.com/python/cpython/commit/60cf26528b931215160a53fd61836dbe4f036009
msg329867 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-13 21:17
New changeset 746b2d35ea47005054ed774fecaed64fab803d7d by Gregory P. Smith in branch 'master':
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
https://github.com/python/cpython/commit/746b2d35ea47005054ed774fecaed64fab803d7d
msg329870 - (view) Author: miss-islington (miss-islington) Date: 2018-11-13 21:50
New changeset fdc485a5dff8508328b4f26265d0af1ba5004597 by Miss Islington (bot) in branch '3.6':
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
https://github.com/python/cpython/commit/fdc485a5dff8508328b4f26265d0af1ba5004597
msg329875 - (view) Author: miss-islington (miss-islington) Date: 2018-11-14 00:39
New changeset 9fbcb1402efab4e287f25145a69ba14c9c6dbce9 by Miss Islington (bot) in branch '3.7':
[3.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506) (GH-10522)
https://github.com/python/cpython/commit/9fbcb1402efab4e287f25145a69ba14c9c6dbce9
msg329929 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-11-14 19:55
New changeset b6f4472dc4190e2fd668490d86aeefd2ab0df935 by Gregory P. Smith in branch '2.7':
[2.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506) (GH-10538)
https://github.com/python/cpython/commit/b6f4472dc4190e2fd668490d86aeefd2ab0df935
msg331639 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-11 16:15
Is this issue completely fixed?
msg331642 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-11 18:30
I believe there are still some issues to deal with.  I don't want to close the issue until I've got my buildbot running.
msg332778 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-31 01:05
New changeset b474e6774d60fa67d5373e361a0ed53c18b24f53 by Gregory P. Smith in branch 'master':
bpo-35214: MSan workarounds for socket, time, and test_faulthandler. (GH-11375)
https://github.com/python/cpython/commit/b474e6774d60fa67d5373e361a0ed53c18b24f53
msg332784 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-31 01:59
New changeset 01b9664740307b39c2907bd84cbb0b2c35be9df4 by Gregory P. Smith (Miss Islington (bot)) in branch '3.7':
bpo-35214: MSan workarounds for socket, time, and test_faulthandler. (GH-11375) (GH-11378)
https://github.com/python/cpython/commit/01b9664740307b39c2907bd84cbb0b2c35be9df4
msg332796 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-31 04:17
New changeset e5796c42c687e1454e84dcc50e6f67db48ff69a0 by Gregory P. Smith in branch 'master':
bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (#11385)
https://github.com/python/cpython/commit/e5796c42c687e1454e84dcc50e6f67db48ff69a0
msg332797 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-31 04:34
Status on my upcoming buildbot host after today's changes:

== Tests result: FAILURE ==

375 tests OK.

11 tests failed:
    test_asyncio test_builtin test_code test_ctypes test_ioctl
    test_openpty test_os test_posix test_pty test_shutil test_uuid

32 tests skipped:
    test_bz2 test_curses test_dbm_gnu test_dbm_ndbm test_devpoll
    test_gzip test_idle test_kqueue test_lzma test_msilib
    test_ossaudiodev test_readline test_smtpnet test_socketserver
    test_sqlite test_ssl test_startfile test_tcl test_timeout test_tix
    test_tk test_ttk_guionly test_ttk_textonly test_turtle
    test_urllib2net test_urllibnet test_winconsoleio test_winreg
    test_winsound test_xmlrpc_net test_zipfile64 test_zlib


Most of those are dying due to pty use (openpty, etc) which is not properly memory sanitizer traced.  test_posix appears to have something
I can fix by annotating in the code.

after that, I'll decide how to tell my buildbot not to run those tests so we can have a green buildbot memory sanitizing everything else.
msg332799 - (view) Author: miss-islington (miss-islington) Date: 2018-12-31 04:39
New changeset 5d2e4b1ff2f01b6aeac2f2f302f363d3eed225fa by Miss Islington (bot) in branch '3.7':
bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (GH-11385)
https://github.com/python/cpython/commit/5d2e4b1ff2f01b6aeac2f2f302f363d3eed225fa
msg332801 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-31 05:13
New changeset 1d300ce1d8238136595c8fea76266a4755cd73a2 by Gregory P. Smith in branch 'master':
bpo-35214: Annotate posix calls for clang MSan. (#11389)
https://github.com/python/cpython/commit/1d300ce1d8238136595c8fea76266a4755cd73a2
msg332804 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-12-31 06:14
New changeset efcf08d8ca9084d8248715f0634c21b705f02ca2 by Gregory P. Smith in branch '3.7':
[3.7] bpo-35214: Annotate posix calls for clang MSan. (GH-11389) (GH-11391)
https://github.com/python/cpython/commit/efcf08d8ca9084d8248715f0634c21b705f02ca2
History
Date User Action Args
2018-12-31 06:14:36gregory.p.smithsetmessages: + msg332804
2018-12-31 05:53:59gregory.p.smithsetpull_requests: + pull_request10760
2018-12-31 05:53:49gregory.p.smithsetpull_requests: + pull_request10759
2018-12-31 05:53:36gregory.p.smithsetpull_requests: + pull_request10758
2018-12-31 05:13:04gregory.p.smithsetmessages: + msg332801
2018-12-31 04:52:02gregory.p.smithsetpull_requests: + pull_request10757
2018-12-31 04:51:50gregory.p.smithsetpull_requests: + pull_request10756
2018-12-31 04:51:38gregory.p.smithsetpull_requests: + pull_request10755
2018-12-31 04:39:31miss-islingtonsetmessages: + msg332799
2018-12-31 04:34:15gregory.p.smithsetmessages: + msg332797
2018-12-31 04:18:19miss-islingtonsetpull_requests: + pull_request10754
2018-12-31 04:18:17miss-islingtonsetpull_requests: + pull_request10753
2018-12-31 04:18:15miss-islingtonsetpull_requests: + pull_request10752
2018-12-31 04:17:59gregory.p.smithsetmessages: + msg332796
2018-12-31 02:58:46gregory.p.smithsetpull_requests: + pull_request10744
2018-12-31 02:58:32gregory.p.smithsetpull_requests: + pull_request10743
2018-12-31 02:58:14gregory.p.smithsetpull_requests: + pull_request10742
2018-12-31 01:59:21gregory.p.smithsetmessages: + msg332784
2018-12-31 01:06:00miss-islingtonsetpull_requests: + pull_request10725
2018-12-31 01:05:58miss-islingtonsetpull_requests: + pull_request10726
2018-12-31 01:05:56miss-islingtonsetpull_requests: + pull_request10724
2018-12-31 01:05:39gregory.p.smithsetmessages: + msg332778
2018-12-30 23:39:26gregory.p.smithsetpull_requests: + pull_request10717
2018-12-30 23:39:16gregory.p.smithsetpull_requests: + pull_request10716
2018-12-30 23:39:07gregory.p.smithsetpull_requests: + pull_request10715
2018-12-11 18:30:58gregory.p.smithsetmessages: + msg331642
versions: - Python 3.6
2018-12-11 16:15:10serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg331639
2018-11-15 14:46:34izbyshevsetnosy: + izbyshev
2018-11-14 19:55:11gregory.p.smithsetmessages: + msg329929
2018-11-14 01:27:35gregory.p.smithsetpull_requests: + pull_request9789
2018-11-14 00:39:40miss-islingtonsetmessages: + msg329875
2018-11-13 21:50:08miss-islingtonsetmessages: + msg329870
2018-11-13 21:17:16miss-islingtonsetpull_requests: + pull_request9783
2018-11-13 21:17:06miss-islingtonsetpull_requests: + pull_request9782
2018-11-13 21:17:00gregory.p.smithsetmessages: + msg329867
2018-11-13 08:42:54gregory.p.smithsetpull_requests: + pull_request9766
2018-11-13 06:27:09miss-islingtonsetmessages: + msg329810
2018-11-13 06:20:26miss-islingtonsetnosy: + miss-islington
messages: + msg329809
2018-11-13 06:01:43miss-islingtonsetpull_requests: + pull_request9765
2018-11-13 06:01:33miss-islingtonsetpull_requests: + pull_request9764
2018-11-13 06:01:26gregory.p.smithsetmessages: + msg329808
2018-11-13 05:44:39gregory.p.smithsetmessages: + msg329807
2018-11-13 05:43:52gregory.p.smithsetpull_requests: + pull_request9763
2018-11-13 05:35:04gregory.p.smithsetmessages: + msg329806
2018-11-13 03:52:45miss-islingtonsetpull_requests: + pull_request9762
2018-11-13 03:52:23miss-islingtonsetpull_requests: + pull_request9761
2018-11-13 03:39:07pablogsalsetmessages: + msg329805
2018-11-13 03:37:03alexsetmessages: + msg329804
2018-11-13 03:34:46pablogsalsetmessages: + msg329803
2018-11-13 03:08:53benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg329798
2018-11-13 03:04:54gregory.p.smithsetpull_requests: + pull_request9760
2018-11-13 03:02:52pablogsalsetnosy: + pablogsal
2018-11-13 02:59:25alexsetnosy: + alex
2018-11-13 00:10:33gregory.p.smithsetmessages: + msg329777
2018-11-12 21:58:02gregory.p.smithsetpull_requests: + pull_request9751
2018-11-12 21:48:26gregory.p.smithsetmessages: + msg329759
2018-11-12 20:29:36gregory.p.smithsetpull_requests: + pull_request9749
2018-11-12 20:07:26gregory.p.smithsetmessages: + msg329750
2018-11-12 07:29:41gregory.p.smithsetkeywords: + patch
pull_requests: + pull_request9746
2018-11-12 07:26:34gregory.p.smithcreate