classification
Title: Improve conditional check for test_gdb
Type: enhancement Stage: patch review
Components: Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, cstratak, jdemeyer, vstinner
Priority: normal Keywords: patch

Created on 2019-06-24 08:31 by jdemeyer, last changed 2019-07-02 10:01 by jdemeyer.

Pull Requests
URL Status Linked Edit
PR 14331 open jdemeyer, 2019-06-24 08:53
PR 14395 open jdemeyer, 2019-06-26 14:26
Messages (15)
msg346358 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-24 08:31
Many tests in test_gdb are not run if python_is_optimized() is true. However, it's not entirely clear why we have this condition. For example, on my system, with the default CPython configuration, Python *is* optimized but still all gdb tests pass if I artificially enable them.

Maybe we need debug symbols, but the existence of debug symbols and optimization are orthogonal.
msg346359 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-24 08:52
The tests FAIL when CPython is compiled with "-O0 -g0" (without optimizations, without debug info) but tests PASS when compiled with "-O3 -g", showing that the check for "-g" is the right one.
msg346600 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 11:06
Depending on the compiler optimization level, gdb may or may not br able to read variables and function arguments. IHMO it is fine to skip test_gdb if Python "is optimized". I don't think it is worth it to check the exact optimizatio level.

The worst debugging experience is provided by PGO build.
msg346601 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-26 11:31
> Depending on the compiler optimization level, gdb may or may not br able to read variables and function arguments.

I haven't seen that problem. As I said, all GDB tests pass when compiled with "-O3 -g".

> IHMO it is fine to skip test_gdb if Python "is optimized".

The fact that these GDB tests are not run in any CI build is bad. For example, we had a buildbot failure after merging PEP 590 because of this. We should try to catch test breakage as early as possible, the buildbot should be the last resort.
msg346608 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 12:11
> I haven't seen that problem. As I said, all GDB tests pass when compiled with "-O3 -g".

Again, it depends on the optimization level, but IMHO it's not worth it to implement an heuristic to check if gdb has enough info to run tests.


> The fact that these GDB tests are not run in any CI build is bad.

gdb is not installed on Travis CI, but test_gdb is run on almost all buildbots.


> We should try to catch test breakage as early as possible, the buildbot should be the last resort.

Maybe open an issue to attempt to install gdb on Travis CI?

test_gdb breakage are quiet rare. If you modify code which can impact test_gdb, you can run test_gdb locally. Because breakages are rare, I never invested time on this issue: there are more blocking issues in my agenda ;-)
msg346609 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-26 12:18
> If you modify code which can impact test_gdb, you can run test_gdb locally

I do run test_gdb locally but it's not very useful since most tests are skipped by default because python_is_optimized() is True. That's the whole point of this issue: tests are skipped without a good reason.
msg346612 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 12:28
> I do run test_gdb locally but it's not very useful since most tests are skipped by default because python_is_optimized() is True.

Ah, I forgot to mention that I always compile Python using:

./configure -C --with-pydebug CFLAGS="-O0" && make

-O0 is not really needed, but sometimes -Og is too aggressive and fails to read some symbols. Or maybe it's just me who is not used to -Og yet :-)

I prefer -O0 because the compilation is way faster! But don't use this config for benchmarks ;-)
msg346619 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-26 14:03
> The worst debugging experience is provided by PGO build.

That's true but unrelated to this issue. We already skip test_gdb completely when PGO is enabled.
msg346624 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 14:41
> That's true but unrelated to this issue. We already skip test_gdb completely when PGO is enabled.

What do you mean? support.PGO is something different: it's only used while profiling Python before Python is recompiled a second time.
msg346625 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-26 14:42
I meant this part in test_gdb.py:

if ((sysconfig.get_config_var('PGO_PROF_USE_FLAG') or 'xxx') in
    (sysconfig.get_config_var('PY_CORE_CFLAGS') or '')):
    raise unittest.SkipTest("test_gdb is not reliable on PGO builds")
msg346627 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 14:54
If you want to understand why tests are currently skipped when Python is optimized, you should dig into the Git history. Not simply remove the decorator because it looks wrong to you. There is likely a story behind it.

https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence
msg346630 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-26 15:15
These python_is_optimized() tests were added in #8605 and #13628. Those issues talk about tests failing when CPython is compiled with -O2 or -O3. But, as I said here, I see no problems today with gcc -O3. So I'm guessing that either gcc or gdb improved such that these tests pass with more recent versions.
msg346633 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-06-26 15:21
There is a fleet of buildbots with a variety og versions of gcc and gdb, so if a change like that is pushed, all the fleet has to be monitored for potential failures, as there are many older OSes supported there.
msg346711 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-06-27 05:12
> Maybe open an issue to attempt to install gdb on Travis CI?

I created PR 14395 for this.
msg347125 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2019-07-02 10:01
> if a change like that is pushed, all the fleet has to be monitored for potential failures, as there are many older OSes supported there.

If this breaks some buildbots, then we can find out more precisely under which conditions they fail. For example, if they fail when GDB is older than a certain version, we can adjust the tests to check for that version.
History
Date User Action Args
2019-07-02 10:01:02jdemeyersetmessages: + msg347125
2019-06-27 05:12:25jdemeyersetmessages: + msg346711
2019-06-26 15:21:50cstrataksetnosy: + cstratak
messages: + msg346633
2019-06-26 15:15:29jdemeyersetmessages: + msg346630
2019-06-26 14:54:49vstinnersetmessages: + msg346627
2019-06-26 14:42:45jdemeyersetmessages: + msg346625
2019-06-26 14:41:25vstinnersetmessages: + msg346624
2019-06-26 14:26:04jdemeyersetpull_requests: + pull_request14209
2019-06-26 14:03:25jdemeyersetmessages: + msg346619
2019-06-26 12:28:10vstinnersetmessages: + msg346612
2019-06-26 12:18:11jdemeyersetmessages: + msg346609
2019-06-26 12:11:28vstinnersetmessages: + msg346608
2019-06-26 11:31:27jdemeyersetmessages: + msg346601
2019-06-26 11:06:32vstinnersetnosy: + vstinner
messages: + msg346600
2019-06-24 08:53:35jdemeyersetkeywords: + patch
stage: patch review
pull_requests: + pull_request14151
2019-06-24 08:52:56jdemeyersetmessages: + msg346359
2019-06-24 08:31:13jdemeyercreate