Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

urllib does not send http/1.1 ALPN extension #85140

Closed
pmenzel mannequin opened this issue Jun 13, 2020 · 6 comments
Closed

urllib does not send http/1.1 ALPN extension #85140

pmenzel mannequin opened this issue Jun 13, 2020 · 6 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@pmenzel
Copy link
Mannequin

pmenzel mannequin commented Jun 13, 2020

BPO 40968
Nosy @orsenthil, @tiran
PRs
  • bpo-40968: Send http/1.1 ALPN extension #20959
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-06-13.11:10:14.395>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library']
title = 'urllib does not send http/1.1 ALPN extension'
updated_at = <Date 2020-11-13.15:38:00.535>
user = 'https://bugs.python.org/pmenzel'

    bugs.python.org fields:

    activity = <Date 2020-11-13.15:38:00.535>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-13.11:10:14.395>
creator = 'pmenzel'
dependencies = []
files = []
hgrepos = []
issue_num = 40968
keywords = ['patch']
message_count = 5.0
messages = ['371450', '371619', '371620', '371621', '380899']
nosy_count = 3.0
nosy_names = ['orsenthil', 'christian.heimes', 'pmenzel']
pr_nums = ['20959']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40968'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']
    @pmenzel
    Copy link
    Mannequin Author

    pmenzel mannequin commented Jun 13, 2020

    Having the TURN server Coturn 1 set up in a Jitsi Meet installation, Python’s urllib requests fail, while it works with cURL and browsers.

    $ curl -I https://jitsi.molgen.mpg.de
HTTP/2 200 
server: nginx/1.14.2
date: Sat, 13 Jun 2020 11:09:19 GMT
content-type: text/html
vary: Accept-Encoding
strict-transport-security: max-age=63072000


    >>> import urllib.request
>>> response = urllib.request.urlopen('https://jitsi.molgen.mpg.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

    @pmenzel pmenzel mannequin added 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jun 13, 2020
    @tiran
    Copy link
    Member

    tiran commented Jun 16, 2020

    It looks like the server is refusing requests that don't have an ALPN extension set in TLS layer. At first I though that the server may only support HTTP/2. The curl requests in your examples uses HTTP/2. urllib only supports HTTP/1.1 and HTTP/1.0.

    Then I tried "curl --http1.1 https://jitsi.molgen.mpg.de". The request also succeeds because curl is sending "ALPN, offering http/1.1".

    This request works for me:

    >>> import ssl
>>> import urllib.request
>>> ctx = ssl.create_default_context()
>>> ctx.set_alpn_protocols(['http/1.1'])
>>> urllib.request.urlopen('https://jitsi.molgen.mpg.de', context=ctx)
<http.client.HTTPResponse object at 0x6040009b7960>

    urllib could set the ALPN header by default when the user does not supply a custom context. It looks like curl always adds an ALPN extension. I don't see code in Python requests that sets ALPN extension. On the other hand Tom Christie's excellent httpx library does set ALPN extension just like curl.

    @pmenzel
    Copy link
    Mannequin Author

    pmenzel mannequin commented Jun 16, 2020

    Wow, great job in figuring this out. Passing the context, it works here too.

    I just wanted to add, that I looked at this with Wireshark, and to not complicate things, first tried plain HTTP.

    $ curl http://jitsi.molgen.mpg.de
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

    With Python, there is also the connection lost error. But Wireshark shows that the 301 HTTP packet is send back by the server.

    $ python3
Python 3.8.3 (default, May 14 2020, 11:03:12) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> response = urllib.request.urlopen('http://jitsi.molgen.mpg.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

    @tiran
    Copy link
    Member

    tiran commented Jun 16, 2020

    Look closer at the traceback. You'll see that urllib follows redirects (http_response -> http_error_302 -> open -> HTTPSConnection). curl doesn't follow redirects by default.

    @tiran tiran added 3.7 (EOL) end of life 3.9 only security fixes 3.10 only security fixes labels Jun 18, 2020
    @tiran tiran changed the title urllib is unable to deal with TURN server infront urllib does not send http/1.1 ALPN extension Jun 18, 2020
    @tiran tiran added 3.7 (EOL) end of life 3.9 only security fixes 3.10 only security fixes labels Jun 18, 2020
    @tiran tiran changed the title urllib is unable to deal with TURN server infront urllib does not send http/1.1 ALPN extension Jun 18, 2020
    @tiran
    Copy link
    Member

    tiran commented Nov 13, 2020

    New changeset f97406b by Christian Heimes in branch 'master':
    bpo-40968: Send http/1.1 ALPN extension (bpo-20959)
    f97406b

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @furkanonder
    Copy link
    Sponsor Contributor

    @tiran Issue seems to be closed. We can close the issue.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @orsenthil @tiran @furkanonder