This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Http client, Bad Status Line triggered for no reason
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, eric.smith, sicarius
Priority: normal Keywords:

Created on 2020-11-22 09:59 by sicarius, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (9)
msg381606 - (view) Author: sicarius noidea (sicarius) Date: 2020-11-22 09:59
Hey,
BadStatusLine triggered when protocol version is in lowercase. 
I've encountered a server that answers "Http/1.0  404 Not Found\r\n" instead of "HTTP/1.0  404 Not Found\r\n"

## Expected Result

Requests understanding the status line.

## Actual Result

Requests is closing the connection.

## Reproduction Steps
### Setup a server that answers the line above
bash: ```while 1;do echo "Http/1.0  404 Not Found\r\n" | sudo nc -lnvp 80; done```
### get the server
```python
import requests
req = req = requests.get("http://127.0.0.1/", verify=False, allow_redir=False )
```

## problem location
Look at line 287 of http/client.py
the word "HTTP" should be matched in lowercase too.
```python
if not version.startswith("HTTP/"):```

Regards.
msg381638 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-11-23 00:33
requests is a third-party library, and this isn't the right place to report issues with it. It looks like the requests issue tracker is at https://github.com/psf/requests/issues

If you can duplicate this problem with only using the python standard library, then please let us know and we'll take a look. I'd have to do some research through the standards to determine if the problem is really with your server, though.
msg381645 - (view) Author: sicarius noidea (sicarius) Date: 2020-11-23 06:54
Hi,

Here I'm using requests to show the behavior because it relies on python's http lib, and it is faster/simplier to use. The Exception "BadStatusLine" is a part or the http/client.py library.

As per the RFC2616 section 6.1 https://tools.ietf.org/html/rfc2616#section-6.1, there's nothing specifying that the HTTP verb must be uppercase, I think it's more a matter of "common sense". I might have missed something though!
msg381646 - (view) Author: sicarius noidea (sicarius) Date: 2020-11-23 07:05
Here is the poc for this error with http.client only (for the server: use the same nc command as my first message): ```python
import http.client
>>> h1 = http.client.HTTPConnection("127.0.0.1:80")
>>> h1.request("GET", "/")
>>> r1 = h1.getresponse()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 289, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: Http/1.0  404 Not Found
```
msg381648 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-11-23 07:46
Thanks for the reproducer and the research!

https://tools.ietf.org/html/rfc2616#section-3.1 says the result header is "HTTP", and doesn't say anything else is acceptable. I'd be interested in what other frameworks (probably in other languages) support. I'll do some research.
msg381649 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-23 07:50
https://tools.ietf.org/html/rfc2616#section-3.1 defines HTTP version indicator as

   HTTP-Version   = "HTTP" "/" 1*DIGIT "." 1*DIGIT

so the check

   version.startswith("HTTP/")

is correct.
msg381651 - (view) Author: sicarius noidea (sicarius) Date: 2020-11-23 08:43
Hi @christian.heimes,
Thank you for your research too. We've also discovered that this check is correct, but this check is very strict and blocking (error raised, stopping the connection), we should maybe be more "laxist" and allow the  lowercase version ? As they do in the others libs ? I've nerver encountered this error with urllib for instance. 

The server that answered this HTTP response line is a clone of the "spip" framework used in many websites. This is clearly a human behavior, but this http.client error could be a bit more "intelligent" I guess.
msg381658 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-23 09:22
urllib is a high level API on top of http.client. It wraps and uses http.client internally. urllib does not accept invalid protocol identifiers either:

>>> urllib.request.urlopen('http://localhost:8080')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib64/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 1379, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib64/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 289, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: Http/1.1 200 OK


curl is more forgiving but does not recognize "Http/1.1" as a valid HTTP/1.1 header either. Instead it assumes that any non-valid header means HTTP/1.0.

$ curl -v http://localhost:8080
*   Trying ::1:8080...
* connect to ::1 port 8080 failed: Connection refused
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.69.1
> Accept: */*
> 
* HTTP 1.0, assume close after body
< Http/1.1 200 OK
< Server: BaseHTTP/0.6 Python/3.8.6
< Date: Mon, 23 Nov 2020 09:10:38 GMT
< Content-Type: text/plain
< 
* Closing connection 0


I'm against changing the behavior of http.client.
msg381659 - (view) Author: sicarius noidea (sicarius) Date: 2020-11-23 09:29
Alright, that was a bad idea.
History
Date User Action Args
2022-04-11 14:59:38adminsetgithub: 86598
2020-11-23 09:29:12sicariussetstatus: open -> closed
resolution: rejected
messages: + msg381659

stage: resolved
2020-11-23 09:22:54christian.heimessetmessages: + msg381658
2020-11-23 08:44:00sicariussetmessages: + msg381651
2020-11-23 07:50:05christian.heimessetnosy: + christian.heimes
messages: + msg381649
2020-11-23 07:46:10eric.smithsetmessages: + msg381648
2020-11-23 07:05:31sicariussetmessages: + msg381646
2020-11-23 06:54:35sicariussetmessages: + msg381645
2020-11-23 00:33:33eric.smithsetnosy: + eric.smith
messages: + msg381638
2020-11-22 09:59:54sicariuscreate