msg405530 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-02 18:32 |
To create a new server with `loop.create_server` that listens on all interfaces and a random port, I'd expect passing in `host=""`, `port=0` to work (per the documentation). However, as written this results in 2 different ports being used - one for ipv4 and one for ipv6. Instead I'd expect a single random port be determined once, and reused for all other interfaces.
Running the example test code (attached) results in:
```
$ python test.py
listening on 0.0.0.0:38023
listening on :::40899
Traceback (most recent call last):
File "/home/jcristharif/Code/distributed/test.py", line 36, in <module>
asyncio.run(main())
File "/home/jcristharif/miniconda3/envs/dask/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/jcristharif/miniconda3/envs/dask/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/home/jcristharif/Code/distributed/test.py", line 30, in main
assert len(ports) == 1, "Only 1 port expected!"
AssertionError: Only 1 port expected!
```
This behavior can be worked around by manually handling `port=0` outside of asyncio, but as it stands naive use can result in accidentally listening on multiple ports.
|
msg405540 - (view) |
Author: Eric V. Smith (eric.smith) * |
Date: 2021-11-02 19:44 |
Is there an OS interface to ensure the same port on both stacks? I don't know of one (although of course one might exist), in which case I don't know how you'd ensure they're both the same.
|
msg405544 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-02 20:24 |
> Is there an OS interface to ensure the same port on both stacks?
I don't think this is needed? Right now the code processes as:
- Expand host + port + family + flags into a list of one or more tuples of socket options (https://github.com/python/cpython/blob/401272e6e660445d6556d5cd4db88ed4267a50b3/Lib/asyncio/base_events.py#L1432-L1436)
- Iterate through this list, creating a new socket for each option tuple, and bind to the corresponding host + port (https://github.com/python/cpython/blob/401272e6e660445d6556d5cd4db88ed4267a50b3/Lib/asyncio/base_events.py#L1441-L1464)
In this case, each call to `socket.bind` gets a 0 port, thus binding to a new random open port number each time.
What I'm asking for is that if the user passes in `port=0`, then the port is extracted in the first call to `socket.bind` when looping and used for all subsequent `socket.bind` calls in the loop. This way we only ever choose a single random open port rather than 1 for each interface. FWIW, this is also what tornado does when `port=0` is provided.
|
msg405545 - (view) |
Author: Eric V. Smith (eric.smith) * |
Date: 2021-11-02 20:30 |
What do you do if a port is bound for IPv4, but is in use for IPv6?
|
msg405547 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-02 20:40 |
Hmmm, I'd find that situation a bit surprising, but I suppose it could happen. Looks like tornado just errors, and that seems to work fine for them in practice (https://github.com/tornadoweb/tornado/blob/790715ae0f0a30b9ee830bfee75bb7fa4c4ec2f6/tornado/netutil.py#L153-L182). Binding IPv4 first might help reduce the chance of a collision, since I suspect there are more IPv4-only applications than IPv6-only.
|
msg405591 - (view) |
Author: Andrew Svetlov (asvetlov) * |
Date: 2021-11-03 09:14 |
Tornado solution sounds weak; it can fail the server start if free ports are available.
I concur with Eric, I'm not aware of an OS API call that binds both IPv4 and IPv6 to the same random port.
|
msg405612 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-03 11:57 |
> I'm not aware of an OS API call that binds both IPv4 and IPv6 to the same random port.
Sure, but `loop.create_server` is already higher-level than a single OS API call.
By default `create_server` will already bind multiple sockets if `host=""`, `host=None`, or if `host` is a list. I'm arguing that the current behavior with `port=0` in these situations is unexpected. Other libraries (like tornado) have come to the same conclusion, and have implemented logic to handle this that seems to work well in practice (though can fail, as you've pointed out).
Is there a use case where the current behavior (binding to multiple random ports) is desired?
|
msg405613 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-03 12:00 |
If you decline that a change is needed here, at the very least the current behavior of `port=0` should be documented. I'd be happy to push up a fix if so.
|
msg405647 - (view) |
Author: Andrew Svetlov (asvetlov) * |
Date: 2021-11-03 20:41 |
PR for documentation fix is appreciated.
Random fails to bind the same port if free ports are available is kind of regression.
Is tornado the only example or you are aware of other libraries with such behavior?
|
msg405651 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-03 21:42 |
> Is tornado the only example or you are aware of other libraries with such behavior?
A quick survey of other language network stacks didn't turn anything up, *But* I also didn't find any implementations (other than asyncio & tornado) that bind multiple sockets with a single api call (as `create_server` does).
I think part of the issue here is that dual IPV6 & IPV4 support is intentionally disabled in asyncio (and tornado), so two sockets are needed (one to support each interface). Other TCP implementations (e.g. both go and rust) don't disable this, so one listener == one socket. This makes comparing API designs across stacks harder - with e.g. Go it's straightforward to listen on a random port on IPV4 & IPV6 with a single TCPListener, since both can be handled by a single socket. Since this is disabled (by default) in asyncio we end up using 2 sockets and run into the issue described above.
Also note that this issue will trigger for any address that resolves to multiple interfaces (not just `host=""`). For example, on osx `localhost` will resolve to `::1` and `127.0.0.1` by default, meaning that the following fairly straightforward asyncio code has a bug in it:
```python
# Start a server on localhost with a random port
server = await loop.create_server(
EchoServerProtocol,
host="localhost",
port=0
)
# Retrieve and log the port
port = server.sockets[0].getsockname()[1]
print(f"listening at tcp://localhost:{port}")
```
As written, this looks correct enough, but on systems where localhost resolves to multiple interfaces this will accidentally listen on multiple ports (instead of one). This can be fixed with some additional logic external to asyncio, but it makes for a much less straightforward asyncio example.
|
msg406951 - (view) |
Author: Jim Crist-Harif (jcristharif) * |
Date: 2021-11-24 19:04 |
Apologies for the delay here. I've pushed a documentation patch at https://github.com/python/cpython/pull/29760.
|
msg406954 - (view) |
Author: miss-islington (miss-islington) |
Date: 2021-11-24 19:44 |
New changeset d71c7bc7339eb82de493c66ebbbfa1cad250ac78 by Jim Crist-Harif in branch 'main':
bpo-45693: Document `port` parameter to `loop.create_server` (GH-29760)
https://github.com/python/cpython/commit/d71c7bc7339eb82de493c66ebbbfa1cad250ac78
|
msg406958 - (view) |
Author: Eric V. Smith (eric.smith) * |
Date: 2021-11-24 20:39 |
New changeset 8cabcde8d66bfd8abc98b862c93c66946f8514a1 by Miss Islington (bot) in branch '3.10':
bpo-45693: Document `port` parameter to `loop.create_server` (GH-29760) (GH-29762)
https://github.com/python/cpython/commit/8cabcde8d66bfd8abc98b862c93c66946f8514a1
|
msg406959 - (view) |
Author: Eric V. Smith (eric.smith) * |
Date: 2021-11-24 20:40 |
New changeset 151c9bf649a049f52df388a8f2390988949abf59 by Miss Islington (bot) in branch '3.9':
bpo-45693: Document `port` parameter to `loop.create_server` (GH-29760) (GH-29763)
https://github.com/python/cpython/commit/151c9bf649a049f52df388a8f2390988949abf59
|
msg406960 - (view) |
Author: Eric V. Smith (eric.smith) * |
Date: 2021-11-24 20:40 |
Thanks for the PR, @jcristharif.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:59:52 | admin | set | github: 89856 |
2021-11-24 20:41:12 | eric.smith | set | assignee: docs@python
nosy:
+ docs@python components:
+ Documentation versions:
+ Python 3.11, - Python 3.8 |
2021-11-24 20:40:55 | eric.smith | set | status: open -> closed resolution: fixed messages:
+ msg406960
stage: patch review -> resolved |
2021-11-24 20:40:13 | eric.smith | set | messages:
+ msg406959 |
2021-11-24 20:39:55 | eric.smith | set | messages:
+ msg406958 |
2021-11-24 19:44:23 | miss-islington | set | pull_requests:
+ pull_request28002 |
2021-11-24 19:44:19 | miss-islington | set | pull_requests:
+ pull_request28001 |
2021-11-24 19:44:15 | miss-islington | set | pull_requests:
+ pull_request28000 |
2021-11-24 19:44:08 | miss-islington | set | nosy:
+ miss-islington messages:
+ msg406954
|
2021-11-24 19:04:41 | jcristharif | set | messages:
+ msg406951 |
2021-11-24 19:03:15 | jcristharif | set | keywords:
+ patch stage: patch review pull_requests:
+ pull_request27998 |
2021-11-03 21:42:00 | jcristharif | set | messages:
+ msg405651 |
2021-11-03 20:41:21 | asvetlov | set | messages:
+ msg405647 |
2021-11-03 12:00:09 | jcristharif | set | messages:
+ msg405613 |
2021-11-03 11:57:09 | jcristharif | set | messages:
+ msg405612 |
2021-11-03 09:14:09 | asvetlov | set | messages:
+ msg405591 |
2021-11-02 20:40:04 | jcristharif | set | messages:
+ msg405547 |
2021-11-02 20:30:09 | eric.smith | set | messages:
+ msg405545 |
2021-11-02 20:24:07 | jcristharif | set | messages:
+ msg405544 |
2021-11-02 19:44:21 | eric.smith | set | nosy:
+ eric.smith messages:
+ msg405540
|
2021-11-02 18:48:27 | jcristharif | set | versions:
+ Python 3.8, Python 3.10 |
2021-11-02 18:32:35 | jcristharif | create | |