x86-64 macOS 3.x:
https://buildbot.python.org/all/#/builders/366/builds/322

The build 322 is the first error. Build 320 was fine, 321 failed with "retry lost connection test (retry)".

(...)
gcc -c  -Wno-unused-result -Wsign-compare -g -O0 -Wall -Qunused-arguments  -Qunused-arguments   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE -o Modules/_math.o Modules/_math.c
 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup    ' OPT='-g -O0 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	./python.exe -E ./setup.py  build
running build
running build_ext
error: [Errno 2] No such file or directory: '/Users/buildbot/buildarea/3.x.billenstein-macos/build/target/include/python3.11d/pyconfig.h'
make: *** [sharedmods] Error 1
program finished with exit code 2
elapsedTime=29.576823

I can't reproduce that failure with that checkout and I'm not even entirely sure where that error is coming from. My guess it that something went wrong during the previous build that resulted in the connection lost (a system crash perhaps?) that left the build area or the file system in an unusual state.

@mattbillenstein, could you check the buildbot and perhaps delete the build directory?

I rebuilt it a while ago which could have caused this - can you re-run the
affected build?

thx

m

On Wed, Jun 02, 2021 at 06:08:16PM +0000, Ned Deily wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
> I can't reproduce that failure with that checkout and I'm not even entirely sure where that error is coming from. My guess it that something went wrong during the previous build that resulted in the connection lost (a system crash perhaps?) that left the build area or the file system in an unusual state.
> 
> @mattbillenstein, could you check the buildbot and perhaps delete the build directory?
> 
> ----------
> nosy: +mattbillenstein
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue44290>
> _______________________________________

Hmm, I just tried to restart it from the builbot page but the build request is just sitting there ATM.

https://buildbot.python.org/all/#/builders/366

Hmm, digging, afaict the buildbot-worker is up and running - something on the
master? I just updated my setup to use python3 instead of python2...

On Wed, Jun 02, 2021 at 06:24:23PM +0000, Ned Deily wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
> Hmm, I just tried to restart it from the builbot page but the build request is just sitting there ATM.
> 
> https://buildbot.python.org/all/#/builders/366
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue44290>
> _______________________________________

Can you say at what point you did the upgrade, i.e. between which two builds here?

https://buildbot.python.org/all/#/builders/366

Probably at 321 where it lost connection - I would have shut it down and wiped
the buildarea.

I'm not sure what's going on now, I'm walking back versions of buildbot-worker
and they seem to never connect to the master...

m

On Wed, Jun 02, 2021 at 07:25:49PM +0000, Ned Deily wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
> Can you say at what point you did the upgrade, i.e. between which two builds here?
> 
> https://buildbot.python.org/all/#/builders/366
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue44290>
> _______________________________________

Perhaps one of the buildbot experts can help?  Zach? Pablo? Victor?

I don't see anything relevant in the logs. These are the last messages regarding this worker:

2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS 3.7
2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS custom
2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS 3.8
2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS 3.10
2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS PR
2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS 3.9
2021-06-02 18:38:26+0000 [-] Worker billenstein-macos detached from x86-64 macOS 3.x

> Hmm, I just tried to restart it from the builbot page but the build request is just sitting there ATM.
> https://buildbot.python.org/all/#/builders/366

Not sure what happened. I restarted the buildbot server manually. I cancelled the old manual build and scheduled a new manual build which is now running.

Hmm, yeah, it did seem to me like the server was wedged or something...  glad to see it going.

M

--
Matt Billenstein
matt@vazor.com

> On Jun 2, 2021, at 6:13 PM, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@python.org> added the comment:
> 
>> Hmm, I just tried to restart it from the builbot page but the build request is just sitting there ATM.
>> https://buildbot.python.org/all/#/builders/366
> 
> Not sure what happened. I restarted the buildbot server manually. I cancelled the old manual build and scheduled a new manual build which is now running.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue44290>
> _______________________________________

The 2 latest builds are green (success), so I close the issue.
https://buildbot.python.org/all/#/builders/366

Thanks for the help!

Alas, it looks like build 325 failed in the same way as build 322:

https://buildbot.python.org/all/#/builders/366/builds/325

I wonder if there might be a Makefile race condition.

So, I'd been trying various things before the master restart and I was using
python3 supplied by homebrew - I decided I didn't like that dependency and
switched it back to /usr/bin/python3 as supplied by the system [1].

I did a rebuild on 325 a couple more times and both of those failed in the
same way - now I've switched back to homebrew python3 and compile succeeded
this time...

I'll let this bake, seems pretty strange changing the interpreter running the
buildbot would matter - everything else should be the same however.

m

1:
mattb@macbook-pro:~ $ /usr/bin/python3
Python 3.8.2 (default, Apr  8 2021, 23:19:18) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

mattb@macbook-pro:~ $ /usr/local/bin/python3
Python 3.9.5 (default, May  4 2021, 03:36:27) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

On Thu, Jun 03, 2021 at 04:20:40AM +0000, Ned Deily wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
> Alas, it looks like build 325 failed in the same way as build 322:
> 
> https://buildbot.python.org/all/#/builders/366/builds/325
> 
> I wonder if there might be a Makefile race condition.
> 
> ----------
> resolution: out of date -> 
> stage: resolved -> 
> status: closed -> open
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue44290>
> _______________________________________

"So, I'd been trying various things before the master restart (...) I'll let this bake, seems pretty strange changing the interpreter running the
buildbot would matter - everything else should be the same however."

If you can, try to stop your buildbot client when you upgrade your system. System upgrades always caused hiccups on buildbots, it's ok. It's just suprising when I got an error and don't know that the system was upgraded.

I close again the issue ;-)

I have been stopping it since the initial problem - there’s something wonky with using the system python on  macos afaict...

M

--
Matt Billenstein
matt@vazor.com

> On Jun 3, 2021, at 4:39 AM, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@python.org> added the comment:
> 
> "So, I'd been trying various things before the master restart (...) I'll let this bake, seems pretty strange changing the interpreter running the
> buildbot would matter - everything else should be the same however."
> 
> If you can, try to stop your buildbot client when you upgrade your system. System upgrades always caused hiccups on buildbots, it's ok. It's just suprising when I got an error and don't know that the system was upgraded.
> 
> I close again the issue ;-)
> 
> ----------
> resolution:  -> fixed
> stage:  -> resolved
> status: open -> closed
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue44290>
> _______________________________________

Well, as soon as the buildbot worker is green, it works for me :-D