classification
Title: Out of source tree builds failing on main - test_importlib others unreliable
Type: behavior Stage: resolved
Components: Build Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.snow Nosy List: barry, eric.snow, gregory.p.smith, ned.deily, steve.dower
Priority: high Keywords: patch

Created on 2021-10-18 07:53 by gregory.p.smith, last changed 2021-11-20 00:11 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 29040 merged eric.snow, 2021-10-18 21:00
PR 29063 merged eric.snow, 2021-10-19 17:42
PR 29300 merged eric.snow, 2021-10-28 21:12
PR 29649 merged steve.dower, 2021-11-19 23:27
Messages (16)
msg404159 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-18 07:53
The behavior is odd, I don't really know how to characterize it other than something serious has gone wrong.  Memory corruption thus differing failures perhaps?  it seems like maybe a race condition?

on the main branch (i cannot reproduce this on 3.10 thankfully):

1) git checkout https://github.com/python/cpython.git upstream
2) mkdir b-u && cd b-u
3) ../upstream/configure --with-pydebug && make -j3 && ./python -m test.regrtest test_importlib

- sometimes it hangs forever.
- sometimes it crashes with a long list of error messages coming out of regrtest itself.  The crashes appear to happen in a child process so regrtest can continue and run other tests if you tell it to run more.  Sometimes it'll be an error about "import _frozenimport_lib as bootstrap" failing with sys.modules containing None. (Linux does that for me)  On macOS when it doesn't hang, I get a blowup from test_importlib/test_threaded_import.py.

I've reproduced this on Linux and macOS.  macOS alternates between a traceback blowup and hang with an occasional pass.  On Linux it is quite consistently a huge chain of stacktraces.

Linux:

```
0:00:00 load avg: 2.48 Run tests sequentially
0:00:00 load avg: 2.48 [1/1] test_importlib
Failed to import test module: test.test_importlib.builtin.test_finder
Traceback (most recent call last):
  File "/home/greg/oss/python/cpython/gpshead/Lib/importlib/__init__.py", line 16, in <module>
    import _frozen_importlib as _bootstrap
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: import of _frozen_importlib halted; None in sys.modules

During handling of the above exception, another exception occurred:

...  tons more  ...
```

Our buildbot fleet is not bloody looking.  Which is why I tried it on two different systems and OSes before reporting.  When I do a ./configure and run everything within the source tree it does not fail.  This is only for proper out-of-tree builds (which are what I always use - as should everybody).

Do our buildbots only do in-tree builds?
msg404160 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-18 08:00
for easy pasting:

git clone https://github.com/python/cpython.git upstream && \
mkdir b-u && cd b-u && \
../upstream/configure --with-pydebug && make -j3 && \
./python -m test.regrtest test_importlib

[change python to python.exe for macos, adjust -j as appropriate]

(if I can figure out git bisect command i'll start one before going to bed...)

(oh hey, i've got macOS producing the same error as Linux now somehow)
msg404161 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-18 08:03
further down in the tracebacks in that situation:

```
  File "/Users/greg/oss/python/upstream/Lib/importlib/_bootstrap.py", line 887, in _fix_up_module
    assert module.__file__ == __file__, (module.__file__, __file__)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ('/Users/greg/oss/python/upstream/Lib/zipimport.py', '/Users/greg/oss/python/b-u/../upstream/Lib/zipimport.py')
```
msg404167 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-18 09:15
git bisect on Linux yielded this culprit change:


```
79cf20e48d0b5d69d9fac2a0204b5ac2c366066a is the first bad commit
commit 79cf20e48d0b5d69d9fac2a0204b5ac2c366066a
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Thu Oct 14 15:32:18 2021 -0600

    bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656)
```

...
Running a git bisect ritual on macOS wasn't helpful, test_importlib out of tree too often simply hangs there (going much further back).  Apparently more than one possible issue at play here.  Lets let this issue be for the Linux one.
msg404188 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-18 14:26
I'm looking into it.
msg404191 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-18 15:08
> Do our buildbots only do in-tree builds?

From what I understand, we do not have any (stable?) buildbots that check out-of-tree builds.

FWIW, in the last month I started testing my changes out-of-tree before merging, after Victor once pointed out that I had broken such builds.  Clearly there must be a gap in what I'm doing.
msg404192 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-10-18 15:28
@gps, on macOS, you may need to bump the open file descriptors limit for the test process, i.e. ulimit -n 1000. I build and run out-of-tree builds all the time on macOS.
msg404222 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-18 21:04
The PR I put up should fix the problem.  If it doesn't then I'll probably drop the checks in the second half of FrozenImporter._fix_module().
msg404226 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2021-10-18 21:45
FWIW, Greg's test case does not fail for me with 6a533a4238
msg404345 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-19 20:11
New changeset f4b12440cf24d7636755aac5b2645e47713557c7 by Eric Snow in branch 'main':
bpo-45506: Go back to not running most of test_embed in out-of-tree builds. (gh-29063)
https://github.com/python/cpython/commit/f4b12440cf24d7636755aac5b2645e47713557c7
msg404838 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-22 23:20
New changeset 17c61045c51512add61a9e75e9c7343cf4e4fb82 by Eric Snow in branch 'main':
bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (#29040)
https://github.com/python/cpython/commit/17c61045c51512add61a9e75e9c7343cf4e4fb82
msg404839 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-22 23:23
i'm considering this fixed based on my testing.  we still lack buildbots using out of tree setup.  i'll ponder that.  But this issue need not track that infrastructure.
msg404842 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-22 23:27
Thanks for bringing this up, Greg!
msg405632 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-11-03 17:47
New changeset 7b438282d39e742b2628a32e7c6699a140ff4cfb by Eric Snow in branch 'main':
bpo-45506: Stop skipping test_embed. (gh-29300)
https://github.com/python/cpython/commit/7b438282d39e742b2628a32e7c6699a140ff4cfb
msg406629 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-11-19 23:26
Hijacking this issue number for a related test failure I found: if you "make install" Python and then run the test_embed tests from the build directory, some will successfully find the stdlib under PREFIX and fail because they expect to never find it.

PR incoming.
msg406632 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-11-20 00:11
New changeset 4c616911b69ce07fb35da1721506bfaba0998c30 by Steve Dower in branch 'main':
bpo-45506: Fix test_embed expecting to not find stdlib in source tree build when stdlib has been installed. (GH-29649)
https://github.com/python/cpython/commit/4c616911b69ce07fb35da1721506bfaba0998c30
History
Date User Action Args
2021-11-20 00:11:43steve.dowersetmessages: + msg406632
2021-11-19 23:27:50steve.dowersetpull_requests: + pull_request27880
2021-11-19 23:26:04steve.dowersetnosy: + steve.dower
messages: + msg406629
2021-11-03 17:47:25eric.snowsetmessages: + msg405632
2021-10-28 21:12:31eric.snowsetpull_requests: + pull_request27564
2021-10-22 23:27:39eric.snowsetmessages: + msg404842
2021-10-22 23:23:14gregory.p.smithsetstatus: open -> closed
messages: + msg404839

assignee: eric.snow
resolution: fixed
stage: patch review -> resolved
2021-10-22 23:20:11eric.snowsetmessages: + msg404838
2021-10-19 20:11:20eric.snowsetmessages: + msg404345
2021-10-19 17:42:51eric.snowsetpull_requests: + pull_request27330
2021-10-18 21:45:12barrysetnosy: + barry
messages: + msg404226
2021-10-18 21:04:21eric.snowsetmessages: + msg404222
2021-10-18 21:00:16eric.snowsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request27311
2021-10-18 15:28:21ned.deilysetnosy: + ned.deily
messages: + msg404192
2021-10-18 15:08:36eric.snowsetmessages: + msg404191
2021-10-18 14:26:29eric.snowsetmessages: + msg404188
2021-10-18 09:15:50gregory.p.smithsetmessages: + msg404167
2021-10-18 08:05:09gregory.p.smithsetnosy: + eric.snow
2021-10-18 08:03:33gregory.p.smithsetmessages: + msg404161
2021-10-18 08:00:05gregory.p.smithsetmessages: + msg404160
2021-10-18 07:53:48gregory.p.smithcreate