classification
Title: Compiler warnings in ubsan builds
Type: compile error Stage: patch review
Components: Build Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, eric.smith, eric.snow, miss-islington, vstinner
Priority: normal Keywords: patch

Created on 2020-06-17 07:00 by christian.heimes, last changed 2020-11-18 17:45 by miss-islington.

Pull Requests
URL Status Linked Edit
PR 20929 merged christian.heimes, 2020-06-17 07:06
PR 23365 merged vstinner, 2020-11-18 13:52
PR 23369 merged christian.heimes, 2020-11-18 15:01
PR 23370 merged miss-islington, 2020-11-18 15:39
Messages (11)
msg371712 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-06-17 07:00
I'm seeing several compiler warnings in ubsan builds:

$ ./configure --with-address-sanitizer --with-undefined-behavior-sanitizer
$ make clean
$ make
Parser/string_parser.c: In function ‘decode_unicode_with_escapes’:
Parser/string_parser.c:98:17: warning: null destination pointer [-Wformat-overflow=]
   98 |                 sprintf(p, "\\U%08x", chr);
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~
Parser/string_parser.c:98:17: warning: null destination pointer [-Wformat-overflow=]
Parser/string_parser.c:98:17: warning: null destination pointer [-Wformat-overflow=]
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5697:25,
    inlined from ‘assemble’ at Python/compile.c:6036:18:
Python/compile.c:5651:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5651 |         *lnotab++ = k;
      |         ~~~~~~~~~~^~~
Objects/unicodeobject.c: In function ‘xmlcharrefreplace’:
Objects/unicodeobject.c:849:16: warning: null destination pointer [-Wformat-overflow=]
  849 |         str += sprintf(str, "&#%d;", PyUnicode_READ(kind, data, i));
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Objects/unicodeobject.c:849:16: warning: null destination pointer [-Wformat-overflow=]
Objects/unicodeobject.c:849:16: warning: null destination pointer [-Wformat-overflow=]
Python/pylifecycle.c: In function ‘Py_FinalizeEx’:
Python/pylifecycle.c:1339:25: warning: unused variable ‘interp’ [-Wunused-variable]
 1339 |     PyInterpreterState *interp = tstate->interp;
      |
msg371713 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-06-17 07:07
PR 20929 addresses three out of four warnings found by GCC 10's ubsan on Fedora 32.
msg381332 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-18 12:31
I see more warnings and a new leak detection on latest master:

gcc -pthread -c -fsanitize=undefined -fsanitize=address -fno-omit-frame-pointer  -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE -o Objects/codeobject.o Objects/codeobject.c
In function ‘emit_pair’,
    inlined from ‘emit_delta’ at Objects/codeobject.c:423:14,
    inlined from ‘code_getlnotab’ at Objects/codeobject.c:462:18:
Objects/codeobject.c:414:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
  414 |     *lnotab++ = b;
      |     ~~~~~~~~~~^~~
In function ‘emit_pair’,
    inlined from ‘emit_delta’ at Objects/codeobject.c:436:14,
    inlined from ‘code_getlnotab’ at Objects/codeobject.c:462:18:
Objects/codeobject.c:414:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
  414 |     *lnotab++ = b;
      |     ~~~~~~~~~~^~~
In function ‘emit_pair’,
    inlined from ‘emit_delta’ at Objects/codeobject.c:429:14,
    inlined from ‘code_getlnotab’ at Objects/codeobject.c:462:18:
Objects/codeobject.c:414:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
  414 |     *lnotab++ = b;
      |     ~~~~~~~~~~^~~
gcc -pthread -c -fsanitize=undefined -fsanitize=address -fno-omit-frame-pointer  -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE -o Python/compile.o Python/compile.c
In function ‘assemble_emit_linetable_pair’,
    inlined from ‘assemble_line_range’ at Python/compile.c:5614:18:
Python/compile.c:5586:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5586 |     *lnotab++ = ldelta;
      |     ~~~~~~~~~~^~~~~~~~
In function ‘assemble_emit_linetable_pair’,
    inlined from ‘assemble_line_range’ at Python/compile.c:5608:18:
Python/compile.c:5586:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5586 |     *lnotab++ = ldelta;
      |     ~~~~~~~~~~^~~~~~~~
In function ‘assemble_emit_linetable_pair’,
    inlined from ‘assemble’ at Python/compile.c:6011:10:
Python/compile.c:5586:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5586 |     *lnotab++ = ldelta;
      |     ~~~~~~~~~~^~~~~~~~


==669952==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 57 byte(s) in 1 object(s) allocated from:
    #0 0x7fa6397cc667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667)
    #1 0x777579 in PyUnicode_New Objects/unicodeobject.c:1459
    #2 0x86aa4e in unicode_decode_utf8 Objects/unicodeobject.c:5129
    #3 0x8b6050 in PyUnicode_DecodeUTF8Stateful Objects/unicodeobject.c:5259
    #4 0x8b6050 in PyUnicode_FromString Objects/unicodeobject.c:2311
    #5 0x8b6050 in PyUnicode_InternFromString Objects/unicodeobject.c:15788
    #6 0x8f1e0b in create_filter Python/_warnings.c:67
    #7 0x8f1e0b in init_filters Python/_warnings.c:95
    #8 0x8f1e0b in _PyWarnings_InitState Python/_warnings.c:123
    #9 0xa52178 in pycore_init_types Python/pylifecycle.c:704
    #10 0xa52178 in pycore_interp_init Python/pylifecycle.c:760
    #11 0xa5eab1 in pyinit_config Python/pylifecycle.c:807
    #12 0xa5eab1 in pyinit_core Python/pylifecycle.c:970
    #13 0xa60037 in Py_InitializeFromConfig Python/pylifecycle.c:1155
    #14 0x47a842 in pymain_init Modules/main.c:66
    #15 0x4802a2 in pymain_main Modules/main.c:698
    #16 0x4802a2 in Py_BytesMain Modules/main.c:731
    #17 0x7fa638a5b041 in __libc_start_main (/lib64/libc.so.6+0x27041)
msg381334 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-18 12:43
The reference leak was introduced in 86ea58149c3 / GH-13159 / bpo-36737. PR GH-20929 fixes it, too.
msg381335 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-18 12:47
Sorry, I meant to add Eric S. :)
msg381336 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-11-18 12:50
Too many Eric S's!
msg381338 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-18 13:51
> The reference leak was introduced in 86ea58149c3 / GH-13159 / bpo-36737. PR GH-20929 fixes it, too.

Usually, it's not that the subinterpreter work introduces new leak, but makes old leak suddenly visible.

The create_filter() leak was introduced way earlier:

commit 9b99747386b690007027c3be2a5d7cfe3d3634f5
Author: Nick Coghlan <ncoghlan@gmail.com>
Date:   Mon Jan 8 12:45:02 2018 +1000

    bpo-31975 (PEP 565): Show DeprecationWarning in __main__ (GH-4458)

It's a minor leak since the create_filter() function is called exactly 5 times at startup. It's a leak of 5 strong references, it's not a big deal :-)
msg381349 - (view) Author: miss-islington (miss-islington) Date: 2020-11-18 14:57
New changeset d1e38d4023aa29e7ed64d4f8eb9c1e4a3c86a2e5 by Victor Stinner in branch 'master':
bpo-40998: Fix a refleak in create_filter() (GH-23365)
https://github.com/python/cpython/commit/d1e38d4023aa29e7ed64d4f8eb9c1e4a3c86a2e5
msg381353 - (view) Author: miss-islington (miss-islington) Date: 2020-11-18 15:39
New changeset 07f2adedf0940b06d136208ec386d69b7d2d5b43 by Christian Heimes in branch 'master':
bpo-40998: Address compiler warnings found by ubsan (GH-20929)
https://github.com/python/cpython/commit/07f2adedf0940b06d136208ec386d69b7d2d5b43
msg381354 - (view) Author: miss-islington (miss-islington) Date: 2020-11-18 16:01
New changeset 994c68f586441cee755508e9357e6e03e2b7a887 by Miss Islington (bot) in branch '3.9':
bpo-40998: Address compiler warnings found by ubsan (GH-20929)
https://github.com/python/cpython/commit/994c68f586441cee755508e9357e6e03e2b7a887
msg381362 - (view) Author: miss-islington (miss-islington) Date: 2020-11-18 17:45
New changeset 35bf8ea7bef7151a420a67638e88d6a1fd81d1a0 by Christian Heimes in branch '3.9':
[3.9] bpo-40998: Fix a refleak in create_filter() (GH-23365) (GH-23369)
https://github.com/python/cpython/commit/35bf8ea7bef7151a420a67638e88d6a1fd81d1a0
History
Date User Action Args
2020-11-18 17:45:56miss-islingtonsetmessages: + msg381362
2020-11-18 16:01:57miss-islingtonsetmessages: + msg381354
2020-11-18 15:39:11miss-islingtonsetpull_requests: + pull_request22263
2020-11-18 15:39:01miss-islingtonsetmessages: + msg381353
2020-11-18 15:01:06christian.heimessetpull_requests: + pull_request22262
2020-11-18 14:57:34miss-islingtonsetnosy: + miss-islington
messages: + msg381349
2020-11-18 13:52:58vstinnersetstage: patch review
pull_requests: + pull_request22258
2020-11-18 13:51:52vstinnersetnosy: + vstinner
messages: + msg381338
2020-11-18 12:50:09eric.smithsetmessages: + msg381336
2020-11-18 12:47:32christian.heimessetmessages: + msg381335
2020-11-18 12:45:52eric.smithsetnosy: + eric.snow
2020-11-18 12:43:27christian.heimessetnosy: + eric.smith
messages: + msg381334
2020-11-18 12:31:36christian.heimessetmessages: + msg381332
2020-06-17 07:07:32christian.heimessetmessages: + msg371713
stage: patch review -> (no value)
2020-06-17 07:06:40christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request20108
2020-06-17 07:00:21christian.heimescreate