classification
Title: ./configure --with-pydebug should use -O0 rather than -Og
Type: Stage: resolved
Components: Build Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, cstratak, pablogsal, pitrou, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2019-10-02 16:35 by vstinner, last changed 2020-03-19 23:33 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 16544 closed vstinner, 2019-10-02 16:52
Messages (12)
msg353746 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-02 16:35
In Python 2.7, when using ./configure --with-pydebug, Python is built using -O0 compiler optimization level: disable all optimizations. The comment is quite explicit: "Optimization messes up debuggers, so turn it off for debug builds".

if test "$Py_DEBUG" = 'true' ; then
    # Optimization messes up debuggers, so turn it off for
    # debug builds.
    OPT="-g -O0 -Wall $STRICT_PROTO"
else
    OPT="-g $WRAP -O3 -Wall $STRICT_PROTO"
fi

In Python 3, -Og is preferred over -O0 for pydebug, if -Og is available:

if test "$Py_DEBUG" = 'true' ; then
    # Optimization messes up debuggers, so turn it off for
    # debug builds.
    if "$CC" -v --help 2>/dev/null |grep -- -Og > /dev/null; then
        OPT="-g -Og -Wall"
    else
        OPT="-g -O0 -Wall"
    fi
else
    OPT="-g $WRAP -O3 -Wall"
fi

Problem: in my experience, gdb traceback doesn't make sense sometimes, and gdb fails to inspect some function arguments and some variables which are displayed as <optimized out>.

See a very concrete example with a test_gdb failure on x86-64 when Python is compiled using gcc -Og:
https://bugzilla.redhat.com/show_bug.cgi?id=1734327#c22

My colleague who is working on gdb suggests to use -O0:
https://bugzilla.redhat.com/show_bug.cgi?id=1734327#c27

Since I started contributing to Python, I always forced gcc -O0 because any other optimization level caused me many issues in gdb. I'm using -O0 for 10 years with sucess.

The GCC documentation says "It is a better choice than -O0 for producing debuggable code because some compiler passes that collect debug information are disabled at -O0." But my experience says the opposite.

Note: instead of -g, we could use -g3 to include debug information for macros and defines.

--

I'm proposing to change the *default* compiler flags from -Og to -O0. Obviously, Linux distributions and developers are free to override the compiler optimization level. For example: ./configure --with-pydebug CFLAGS="-Og" ensures that Python is always built using -Og.

I propose to modify 3.7, 3.8 and master branches.
msg353748 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-02 16:44
I basically propose to revert bpo-23445 change:

commit 3d6c784371bccc2407048652bce50c5bccf9b1af
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Wed Feb 11 19:39:16 2015 +0100

    Issue #23445: pydebug builds now use "gcc -Og" where possible, to make the resulting executable faster.
msg353750 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-02 16:50
test_gdb is failing in different architectures and different operating systems:

* Fedora, Python 3.8, x86-64:
  https://bugzilla.redhat.com/show_bug.cgi?id=1734327

* RHEL8, Python 3.6, s390x
  https://bugzilla.redhat.com/show_bug.cgi?id=1712977

* RHEL 8, Python 3.6, ppc64le
  https://bugzilla.redhat.com/show_bug.cgi?id=1714733

I'm quite sure that most issues are caused by the compiler optimization level which is not -O0.

See also bpo-37631: the debug build of the RHEL8 package is supposed to be compiled using -O0 using EXTRA_CFLAGS variable, but -O0 in EXTRA_CFLAGS variable is overriden by -02 in CFLAGS_NODIST.
msg353753 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-02 16:56
My use case is to debug a crash using a Python compiled in debug module, like python3-debug program in Fedora. Since the debug ABI is now compatible with the release build, the idea is to attempt to reproduce a crash in gdb using python3-debug instead of python3, and then use gdb to see what's going on.

With -Og, the call stack is wrong sometimes, and some function arguments and local variables cannot be read (displayed as <optimized out>).

On Travis CI, a few months ago, Python was built in debug mode using -O3. But it was a side effect of OpenSSL flags if I recall correctly.

With my PR 16544, Travis CI now uses -O0.
msg353795 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-10-03 01:18
Do note though that if the -D_FORTIFY_SOURCE=2 hardening flag is used, the compilation will fail with an optimization level less than -Og.

Haven't tried yet with -D_FORTIFY_SOURCE=1 to see if it works with -O0.
msg353798 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-10-03 01:36
Correction, not fail, just a ton of warnings. The same is true for -D_FORTIFY_SOURCE=1
msg353802 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-10-03 01:50
If -Og is breaking debugging, isn't that a compiler bug? The GCC manpage claims " -Og enables optimizations that do not interfere with debugging."
msg353806 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-10-03 02:22
It seems that it's still being worked on from gcc's side:

https://gcc.gnu.org/bugzilla//show_bug.cgi?id=78685
msg353822 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-03 07:11
> Do note though that if the -D_FORTIFY_SOURCE=2 hardening flag is used, the compilation will fail with an optimization level less than -Og.

Does it make any sense to use a debug build in production? Does it make sense to enable hardening on a debug buil?d
msg353828 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-10-03 08:32
The purpose of using -Og is that it significantly speeds up tests. You can always pass CFLAGS="-O0" if you debug particular debugger issues, but in normal cases -Og looks more beneficial. It saves hours of time of common developers.
msg361052 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 12:15
There consensus seems to be towards the status quo: -Og is a good trade off between debug and performance for most core developers usage.

For the Fedora package, it may make sense to use -O0, but that should be discussed in Fedora ;-)

So I close the issue. Thanks for the interesting discussion :-)
msg364641 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-19 23:33
See also bpo-40019: "test_gdb  should better detect when Python is optimized".
History
Date User Action Args
2020-03-19 23:33:21vstinnersetmessages: + msg364641
2020-01-30 12:15:34vstinnersetstatus: open -> closed
resolution: rejected
messages: + msg361052

stage: patch review -> resolved
2019-10-04 02:35:35ned.deilysetversions: - Python 3.7
2019-10-03 08:32:54serhiy.storchakasetmessages: + msg353828
2019-10-03 07:11:14vstinnersetmessages: + msg353822
2019-10-03 02:22:48cstrataksetmessages: + msg353806
2019-10-03 01:50:35benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg353802
2019-10-03 01:36:33cstrataksetmessages: + msg353798
2019-10-03 01:18:28cstrataksetnosy: + cstratak
messages: + msg353795
2019-10-02 16:56:56vstinnersetnosy: + pitrou, serhiy.storchaka, pablogsal
messages: + msg353753
2019-10-02 16:52:39vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request16133
2019-10-02 16:50:40vstinnersetmessages: + msg353750
2019-10-02 16:44:00vstinnersetmessages: + msg353748
2019-10-02 16:35:07vstinnercreate