➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: urllib does not send http/1.1 ALPN extension
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, orsenthil, pmenzel
Priority: normal Keywords: patch

Created on 2020-06-13 11:10 by pmenzel, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 20959 merged christian.heimes, 2020-06-18 09:59
Messages (5)
msg371450 - (view) Author: Paul Menzel (pmenzel) Date: 2020-06-13 11:10 
Having the TURN server Coturn [1] set up in a Jitsi Meet installation, Python’s urllib requests fail, while it works with cURL and browsers.

```
$ curl -I https://jitsi.molgen.mpg.de
HTTP/2 200 
server: nginx/1.14.2
date: Sat, 13 Jun 2020 11:09:19 GMT
content-type: text/html
vary: Accept-Encoding
strict-transport-security: max-age=63072000

```

```
>>> import urllib.request
>>> response = urllib.request.urlopen('https://jitsi.molgen.mpg.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
```

[1]: https://github.com/coturn/coturn/
msg371619 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-06-16 07:36 
It looks like the server is refusing requests that don't have an ALPN extension set in TLS layer. At first I though that the server may only support HTTP/2. The curl requests in your examples uses HTTP/2. urllib only supports HTTP/1.1 and HTTP/1.0.

Then I tried "curl --http1.1 https://jitsi.molgen.mpg.de". The request also succeeds because curl is sending "ALPN, offering http/1.1".

This request works for me:

>>> import ssl
>>> import urllib.request
>>> ctx = ssl.create_default_context()
>>> ctx.set_alpn_protocols(['http/1.1'])
>>> urllib.request.urlopen('https://jitsi.molgen.mpg.de', context=ctx)
<http.client.HTTPResponse object at 0x6040009b7960>

urllib could set the ALPN header by default when the user does not supply a custom context. It looks like curl always adds an ALPN extension. I don't see code in Python requests that sets ALPN extension. On the other hand Tom Christie's excellent httpx library does set ALPN extension just like curl.
msg371620 - (view) Author: Paul Menzel (pmenzel) Date: 2020-06-16 07:57 
Wow, great job in figuring this out. Passing the context, it works here too.

I just wanted to add, that I looked at this with Wireshark, and to not complicate things, first tried plain HTTP.

```
$ curl http://jitsi.molgen.mpg.de
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
```

With Python, there is also the connection lost error. But Wireshark shows that the 301 HTTP packet is send back by the server.

```
$ python3
Python 3.8.3 (default, May 14 2020, 11:03:12) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> response = urllib.request.urlopen('http://jitsi.molgen.mpg.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
```
msg371621 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-06-16 08:06 
Look closer at the traceback. You'll see that urllib follows redirects (http_response -> http_error_302 -> open -> HTTPSConnection). curl doesn't follow redirects by default.
msg380899 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-13 15:37 
New changeset f97406be4c0a02c1501c7ab8bc8ef3850eddb962 by Christian Heimes in branch 'master':
bpo-40968: Send http/1.1 ALPN extension (#20959)
https://github.com/python/cpython/commit/f97406be4c0a02c1501c7ab8bc8ef3850eddb962
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85140
2020-11-13 15:38:00christian.heimessetmessages: + msg380899
2020-06-18 09:59:33christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request20138
2020-06-18 09:34:11christian.heimessettitle: urllib is unable to deal with TURN server infront -> urllib does not send http/1.1 ALPN extension
versions: + Python 3.7, Python 3.9, Python 3.10
2020-06-16 08:06:26christian.heimessetmessages: + msg371621
2020-06-16 07:57:15pmenzelsetmessages: + msg371620
2020-06-16 07:36:59christian.heimessetnosy: + christian.heimes
messages: + msg371619
2020-06-14 07:47:10SilentGhostsetnosy: + orsenthil
2020-06-13 11:10:14pmenzelcreate