Since the buildbot server migrated to a new machine, the web page losts is connection and the whole web page is reloaded every minute. Try for example:

https://buildbot.python.org/all/#/release_status

The new buildbot.python.org machine is now behind a load balancer.

TCP connections closed after 1 minute already affected clients: see bpo-41642. It seems like HTTPS connections (tcp/443) are also affected.

I was wondering if there was any update on whether or not this new behavior can be corrected?

I was attempting to review a buildbot failure today and it's actually pretty tough to "race the refresh" when trying to review the build steps and logs.

We need someone of the PSF Infra team to increase the load balancer delay. Or fix the load balancer.

I am away from my computer at the moment, but there is a direct access hostname for the buildbot host that was announced to the build server owners. Configuring that bypasses the load balancer.

Apologies, wrong issue. I’ll have to take a closer look at this.

Is there anything I can help with to move this forward?
Investigating buildbot failures continues to be very annoying.

OK, I've confirmed that HAProxy seems to be the issue. WebSockets opened to the nginx proxy on the server or directly to the twisted server successfully remain indefinitely.

If anyone familiar with HAProxy would be interested in helping debug, the current relevant information is:

- Ubuntu 18.04.5 LTS
- HA-Proxy version 1.8.8-1ubuntu0.11 2020/06/22

Here is the current HAProxy configuration: https://gist.github.com/ewdurbin/d8a42c30a04d6cb5763431200acaecde which is generated from this salt state: https://github.com/python/psf-salt/tree/master/salt/haproxy and this pillar data: https://github.com/python/psf-salt/blob/master/pillar/base/haproxy.sls

I can provide any additional information that would be helpful to someone trying to sort out what's going on.

I looked at these files.

HAProxy configuration, "defaults" section contains:

    timeout connect 5000
    timeout client  50000
    timeout server  50000

These timeouts are not overriden in "frontend main" nor in "backend buildbot-master" sections.

pillar/base/haproxy.sls seems to try to override these variables:

https://github.com/python/psf-salt/blob/d89e5ef2e86f45c1766c8b93d6e9621b0ab1bb09/pillar/base/haproxy.sls#L7-L11

I don't see these in the rendered HAProxy config, but I see "timeout tunnel 3600s" is rendered as "timeout tunnel 1d" in "backend buildbot-master".

Apologies, I gisted a version that was from attempts to debug the timeouts. It's been updated.

Ee:
> Apologies, I gisted a version that was from attempts to debug the timeouts. It's been updated.

Aaaah :-D No problem. Can you please try:

* replace "timeout client 30s" with "timeout client 1d"
* replace "timeout server 30s" with "timeout server 1d"

in buildbot-master server of the pillar config:

https://github.com/python/psf-salt/blob/d89e5ef2e86f45c1766c8b93d6e9621b0ab1bb09/pillar/base/haproxy.sls#L7-L11

> "timeout server 1d"

Hum. I'm not sure if "1d" syntax is accepted. Maybe use "3600s".

Ee, apparently this issue in the buildbot repo may be related: https://github.com/buildbot/buildbot/issues/4078

Could you investigate if we can use this on our PSF server?

I'm going to try it our on the server but apparently there are some problems: https://github.com/buildbot/buildbot/issues/5991

Can someone confirm if they still have this problem on the buildbot server?

> Can someone confirm if they still have this problem on the buildbot server?

Oh wow, I don't have the issue anymore! Thank you very much :-) Previously, the website was barely usable :-(