classification
Title: test_inspect fails in refleak mode
Type: Stage: resolved
Components: Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, andrei.avk, lukasz.langa, miss-islington, pablogsal
Priority: Keywords: patch

Created on 2021-08-02 09:55 by pablogsal, last changed 2021-08-03 13:08 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27544 merged pablogsal, 2021-08-02 11:13
PR 27546 merged miss-islington, 2021-08-02 12:29
PR 27571 merged andrei.avk, 2021-08-03 01:55
PR 27578 merged miss-islington, 2021-08-03 12:47
Messages (16)
msg398744 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 09:55
From https://bugs.python.org/issue44206 :

  File "/home/mark/repos/cpython/Lib/inspect.py", line 1154, in walktree
    classes.sort(key=attrgetter('__module__', '__name__'))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'str' and 'module'


I can reproduce that failure with a debug build and either of
./python -m test -R 3:3  test_inspect
./python -m test test_inspect -F
msg398745 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 09:55
I'm bisecting
msg398747 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 09:57
f468ede4a2b7ab5c72ae85ab04cb56904300cd23 is the first bad commit
commit f468ede4a2b7ab5c72ae85ab04cb56904300cd23
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Fri Jul 30 10:46:42 2021 -0700

    bpo-44648: Fix error type in inspect.getsource() in interactive session (GH-27171) (GH-27495)

    (cherry picked from commit 48a62559dfaf775e4f1cc56b19379c799e8e2587)

    Co-authored-by: andrei kulakov <andrei.avk@gmail.com>

 Lib/inspect.py                                          |  2 ++
 Lib/test/test_inspect.py                                | 17 ++++++++++++++++-
 .../Library/2021-07-15-16-51-32.bpo-44648.2o49TB.rst    |  3 +++
 3 files changed, 21 insertions(+), 1 deletion(-)
 create mode 100644 bpo-44648.2o49TB.rst">Misc/NEWS.d/next/Library/2021-07-15-16-51-32.bpo-44648.2o49TB.rst
msg398749 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 10:09
The reason buildbots have not catched this is because the re-run login seems to not detect the problem on the second run:

./python.exe -m test test_inspect -j 1 -u all -W --slowest --fail-env-changed --timeout=11700 -R 3:3 -u-cpu -w

WARNING: Disable --verbose3 because it's incompatible with --huntrleaks: see http://bugs.python.org/issue27103
0:00:00 load avg: 2.73 Run tests in parallel using 1 child processes (timeout: 3 hour 15 min, worker timeout: 3 hour 20 min)
0:00:03 load avg: 2.92 [1/1/1] test_inspect failed (1 error)
beginning 6 repetitions
123456
.test test_inspect failed -- Traceback (most recent call last):
  File "/Users/pgalindo3/github/cpython/Lib/test/test_inspect.py", line 398, in test_getclasses
    tree = inspect.getclasstree([cls[1] for cls in classes])
  File "/Users/pgalindo3/github/cpython/Lib/inspect.py", line 1185, in getclasstree
    return walktree(roots, children, None)
  File "/Users/pgalindo3/github/cpython/Lib/inspect.py", line 1158, in walktree
    results.append(walktree(children[c], children, c))
  File "/Users/pgalindo3/github/cpython/Lib/inspect.py", line 1154, in walktree
    classes.sort(key=attrgetter('__module__', '__name__'))
TypeError: '<' not supported between instances of 'str' and 'module'

== Tests result: FAILURE ==

10 slowest tests:
- test_inspect: 3.4 sec

1 test failed:
    test_inspect

1 re-run test:
    test_inspect

0:00:03 load avg: 2.92 Re-running failed tests in verbose mode
0:00:03 load avg: 2.92 Re-running test_inspect in verbose mode (matching: test_getclasses)
beginning 6 repetitions
123456
test_getclasses (test.test_inspect.TestRetrievingSourceCode) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
.test_getclasses (test.test_inspect.TestRetrievingSourceCode) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
.test_getclasses (test.test_inspect.TestRetrievingSourceCode) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
.test_getclasses (test.test_inspect.TestRetrievingSourceCode) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
.test_getclasses (test.test_inspect.TestRetrievingSourceCode) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
.test_getclasses (test.test_inspect.TestRetrievingSourceCode) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK
.

== Tests result: FAILURE then SUCCESS ==

1 test OK.

10 slowest tests:
- test_inspect: 3.4 sec

Total duration: 4.6 sec
Tests result: FAILURE then SUCCESS
msg398750 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 10:09
Check for instance the logs in https://buildbot.python.org/all/#/builders/679/builds/92/steps/5/logs/stdio
msg398752 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 11:05
From f468ede4a2b7ab5c72ae85ab04cb56904300cd23:

This code looks quite dangerous:

class TestGetsourceInteractive(unittest.TestCase):
    def tearDown(self):
        mod.ParrotDroppings.__module__ = mod
        sys.modules['__main__'] = self.main

That is modifing sys.modules globally!
msg398753 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 11:12
Ah, the problem is that:

        mod.ParrotDroppings.__module__ = mod

is wrong: `__module__` must be a string, not a module object
msg398754 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 11:14
PR 27544 just addresses the main problem, but I still have concerns about the test affecting sys.modules directly, making it dangerous to run in parallel.
msg398755 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-08-02 11:15
With PR 27544:

❯ ./python.exe -m test test_inspect -R :
0:00:00 load avg: 2.91 Run tests sequentially
0:00:00 load avg: 2.91 [1/1] test_inspect
beginning 9 repetitions
123456789
.........

== Tests result: SUCCESS ==

1 test OK.

Total duration: 13.7 sec
Tests result: SUCCESS
msg398757 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-02 11:54
New changeset 626d397cc1612ea5eef153dd910834c2ee00ddbd by Pablo Galindo Salgado in branch 'main':
bpo-44808: Fix test_inspect in refleak mode (GH-27544)
https://github.com/python/cpython/commit/626d397cc1612ea5eef153dd910834c2ee00ddbd
msg398759 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-08-02 12:02
Pablo: thanks for investigating and fixing this, I will look for examples of changing or patching sys.modules in the testsuite to see how to avoid global effects. I think we can keep this issue open for the follow-up discussion / fix?
msg398767 - (view) Author: miss-islington (miss-islington) Date: 2021-08-02 13:41
New changeset a1eaa74d9dcd973ec278cefc22aac7f514faee4b by Miss Islington (bot) in branch '3.10':
bpo-44808: Fix test_inspect in refleak mode (GH-27544)
https://github.com/python/cpython/commit/a1eaa74d9dcd973ec278cefc22aac7f514faee4b
msg398788 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-08-02 17:59
I've been looking into avoiding global modification of modules['__main__'], the options are:

 1. remove the test and leave it at that

 2. remove the test and open a new issue for re-adding a test ; and look for a way to do it safely

 3. create a temp script file, add code to it that creates a class, sets `sys.modules['__main__'].__file__ = None` and inspects the class, and run this module in a new interpreter, checks exception raised and does sys.exit(1); and check the exit code after running this script.

 4. patch inspect.sys.modules; as far as I understand, this would mean other modules accessing sys.modules in other threads won't be affected, and sys.modules['__main__'] is currently used in inspect in a very limited way, so it can probably be shown that this patching won't affect other tests that use inspect, -- but I would need to look more into this.

I'm not sure if #3 is too heavy/complex for this test.

The advantage of #1 and #2 is that it's easy to do and ensures that there won't be strange random test failures that are hard to track down and debug (which is something that I'm afraid can happen right now, caused by this test).

#4 I've added to get comments / thought.
msg398812 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-08-03 01:56
Please disregard my last comment, I think I found a good way to fix it, the PR is up here:
https://github.com/python/cpython/pull/27571/files
msg398824 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-03 12:47
New changeset 58325971de0faf330c9c38269dae8315a0746e59 by andrei kulakov in branch 'main':
bpo-44808: fixes test for interactive inspect getsource of a class (GH-27571)
https://github.com/python/cpython/commit/58325971de0faf330c9c38269dae8315a0746e59
msg398825 - (view) Author: miss-islington (miss-islington) Date: 2021-08-03 13:08
New changeset bc2841c7a9fe2b05ef77ebdf77701188dc83b2ce by Miss Islington (bot) in branch '3.10':
bpo-44808: fixes test for interactive inspect getsource of a class (GH-27571)
https://github.com/python/cpython/commit/bc2841c7a9fe2b05ef77ebdf77701188dc83b2ce
History
Date User Action Args
2021-08-03 13:08:55lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-08-03 13:08:07miss-islingtonsetmessages: + msg398825
2021-08-03 12:47:53lukasz.langasetmessages: + msg398824
2021-08-03 12:47:39miss-islingtonsetpull_requests: + pull_request26084
2021-08-03 01:56:42andrei.avksetmessages: + msg398812
2021-08-03 01:55:08andrei.avksetpull_requests: + pull_request26078
2021-08-02 19:02:33pablogsalsetpriority: release blocker ->
2021-08-02 17:59:16andrei.avksetmessages: + msg398788
2021-08-02 13:41:38miss-islingtonsetmessages: + msg398767
2021-08-02 12:29:40miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26055
2021-08-02 12:02:05andrei.avksetnosy: + andrei.avk
messages: + msg398759
2021-08-02 11:54:35lukasz.langasetmessages: + msg398757
2021-08-02 11:15:09pablogsalsetmessages: + msg398755
2021-08-02 11:14:39pablogsalsetmessages: + msg398754
2021-08-02 11:13:40pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request26053
2021-08-02 11:12:02pablogsalsetmessages: + msg398753
2021-08-02 11:05:43pablogsalsetmessages: + msg398752
2021-08-02 10:09:51pablogsalsetmessages: + msg398750
2021-08-02 10:09:41pablogsalsetmessages: + msg398749
2021-08-02 09:58:24pablogsalsetnosy: + lukasz.langa
2021-08-02 09:57:56pablogsalsetmessages: + msg398747
2021-08-02 09:56:09pablogsalsetpriority: normal -> release blocker
2021-08-02 09:55:56pablogsalsetmessages: + msg398745
2021-08-02 09:55:50pablogsalcreate