Implement Server.serve_forever and corresponding APIs #76843

1st1 · 2018-01-25T06:44:25Z

BPO	32662
Nosy	@vstinner, @asvetlov, @1st1
PRs	bpo-32662: Implement Server.start_serving() and Server.serve_forever() #5312 bpo-32662: Try making test_asyncio.test_server more reliable #5338

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-29.17:03:12.778>
created_at = <Date 2018-01-25.06:44:24.572>
labels = ['type-feature', '3.7', 'expert-asyncio']
title = 'Implement Server.serve_forever and corresponding APIs'
updated_at = <Date 2018-05-29.17:03:12.777>
user = 'https://github.com/1st1'

bugs.python.org fields:

activity = <Date 2018-05-29.17:03:12.777>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.17:03:12.778>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2018-01-25.06:44:24.572>
creator = 'yselivanov'
dependencies = []
files = []
hgrepos = []
issue_num = 32662
keywords = ['patch']
message_count = 7.0
messages = ['310657', '310717', '310740', '310748', '310749', '310750', '310786']
nosy_count = 3.0
nosy_names = ['vstinner', 'asvetlov', 'yselivanov']
pr_nums = ['5312', '5338']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32662'
versions = ['Python 3.7']

1st1 · 2018-01-25T06:44:24Z

As discussed, we want to make Server objects more usable in async/await code and more compatible with asyncio.run.

This is also needed to handle a use case when two or more servers are created and need to start listening at the same time.

We propose to:

Add a new bool flag defaulting to True to loop.create_server and loop.create_unix_server: start_serving. By default, loop will return a server that is already accepting connections. When start_serving is set to False, create_server and create_unix_server will return a server that will not listen on its sockets.
A new idempotent Server.start_serving() method can be used to make server listen on its sockets (useful when a server object was created with start_serving=False).
A new Server.serve_forever() method that calls start_serving() and blocks forever, until cancelled. When cancelled, it closes its server object.
A new Server.is_serving() method. This is useful to introspect a server object in unittests.
Server objects should be async context managers. Server.__aexit__ should close the server and await on Server.wait_closed().

With these new APIs, the following pattern becomes possible:

    async def main():
        srv = await asyncio.start_server(...)
        async with srv:
            await srv.serve_forever()

    asyncio.run(main())

1st1 · 2018-01-25T23:08:12Z

New changeset c9070d0 by Yury Selivanov in branch 'master':
bpo-32662: Implement Server.start_serving() and Server.serve_forever() (bpo-5312)
c9070d0

1st1 · 2018-01-26T06:31:05Z

New changeset 4112c5b by Yury Selivanov in branch 'master':
bpo-32662: Try making test_asyncio.test_server more reliable (bpo-5338)
4112c5b

vstinner · 2018-01-26T09:56:43Z

test_asyncio.test_start_server_1() hangs on "s390x Debian 3.x" buildbot:
http://buildbot.python.org/all/#builders/13/builds/579

test_set_nodelay (test.test_asyncio.test_selector_events.TestSelectorUtils) ... ok
test_start_server_1 (test.test_asyncio.test_server.ProactorStartServerTests) ... skipped 'Windows only'
Exception in thread test-client:
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 193, in run
    self._prog(TestSocketWrapper(self._sock))
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
    with self.tcp_client(lambda sock: client(sock, addr)):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 19, in client
    sock.connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 195, in run
    self._test._abort_socket_test(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
    self.fail(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: [Errno 111] Connection refused

Timeout (0:15:00)!
Thread 0x000003fffcdcd710 (most recent call first):
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/selectors.py", line 468 in select
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 1555 in _run_once
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 450 in run_forever
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 482 in run_until_complete
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 45 in test_start_server_1
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 615 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 663 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/init.py", line 1861 in _run_suite
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/init.py", line 1951 in run_unittest
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 176 in runtest_inner
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 291 in rerun_failed_tests
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 540 in _main
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 510 in main
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 585 in main
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/main.py", line 2 in <module>
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 193 in _run_module_as_main
test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Makefile:1090: recipe for target 'buildbottest' failed
make: *** [buildbottest] Error 1
program finished with exit code 2
elapsedTime=2050.270461

vstinner · 2018-01-26T09:58:39Z

The test fails differently on s390x SLES 3.x:
http://buildbot.python.org/all/#/builders/16/builds/576

...
test_force_close (test.test_asyncio.test_selector_events.SelectorTransportTests) ... ok
test_set_nodelay (test.test_asyncio.test_selector_events.TestSelectorUtils) ... ok
test_start_server_1 (test.test_asyncio.test_server.ProactorStartServerTests) ... skipped 'Windows only'
test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Exception in thread test-client:
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 193, in run
    self._prog(TestSocketWrapper(self._sock))
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
    with self.tcp_client(lambda sock: client(sock, addr)):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 19, in client
    sock.connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 195, in run
    self._test._abort_socket_test(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
    self.fail(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: [Errno 111] Connection refused

/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py:45: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 56340)>
  self.loop.run_until_complete(main_task)
ERROR
test_start_unix_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... ok
test_start_tls_client_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
test_start_tls_wrong_args (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
test_start_tls_client_1 (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/collections/__init__.py:404: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 42302)>
  result = _self._make(map(kwds.pop, field_names, _self))
Unhandled error in exception handler
context: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<BaseStartServer.test_start_server_1.<locals>.main() done, defined at /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py:31> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x3fff979bbf8>()]>>}
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 1459, in call_exception_handler
    self._exception_handler(self, context)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 22, in loop_exception_handler
    self.loop.default_exception_handler(context)
AttributeError: 'NoneType' object has no attribute 'default_exception_handler'
ok
test_start_tls_server_1 (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... ok
test_start_tls_wrong_args (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... ok
test_cancel_handshake (test.test_asyncio.test_sslproto.SslProtoHandshakeTests) ... ok
test_close_during_handshake (test.test_asyncio.test_sslproto.SslProtoHandshakeTests) .
...

======================================================================
ERROR: test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 45, in test_start_server_1
    self.loop.run_until_complete(main_task)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 493, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

vstinner · 2018-01-26T10:00:41Z

Just to be complete, the test also fails on x86 Gentoo Non-Debug with X 3.x:
http://buildbot.python.org/all/#/builders/99/builds/564

...
test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Exception in thread test-client:
Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 193, in run
    self._prog(TestSocketWrapper(self._sock))
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
    with self.tcp_client(lambda sock: client(sock, addr)):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 19, in client
    sock.connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 195, in run
    self._test._abort_socket_test(ex)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
    self.fail(ex)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: [Errno 111] Connection refused

Timeout (0:15:00)!
Thread 0xb7d69700 (most recent call first):
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/selectors.py", line 468 in select
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 1555 in _run_once
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 450 in run_forever
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 482 in run_until_complete
File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 45 in test_start_server_1
...

1st1 · 2018-01-26T20:07:07Z

I think these failures were on the previous version of the test; i've fixed it last night and it shouldn't timeout this way anymore.

1st1 added 3.7 (EOL) end of life topic-asyncio type-feature A feature request or enhancement labels Jan 25, 2018

1st1 closed this as completed Jan 25, 2018

vstinner reopened this Jan 26, 2018

1st1 closed this as completed May 29, 2018

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Server.serve_forever and corresponding APIs #76843

Implement Server.serve_forever and corresponding APIs #76843

1st1 commented Jan 25, 2018

1st1 commented Jan 25, 2018

1st1 commented Jan 25, 2018

1st1 commented Jan 26, 2018

vstinner commented Jan 26, 2018

vstinner commented Jan 26, 2018

vstinner commented Jan 26, 2018

1st1 commented Jan 26, 2018

Implement Server.serve_forever and corresponding APIs #76843

Implement Server.serve_forever and corresponding APIs #76843

Comments

1st1 commented Jan 25, 2018

1st1 commented Jan 25, 2018

1st1 commented Jan 25, 2018

1st1 commented Jan 26, 2018

vstinner commented Jan 26, 2018

vstinner commented Jan 26, 2018

vstinner commented Jan 26, 2018

1st1 commented Jan 26, 2018