classification
Title: Empty body {} in POST requests leads to 405 Method not allowed error
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: andrei.avk, bhushan.shelke, christian.heimes, paul.moore, ronaldoussoren, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2020-11-20 09:20 by bhushan.shelke, last changed 2020-12-27 03:32 by andrei.avk.

Files
File name Uploaded Description Edit
sample_flask.py bhushan.shelke, 2020-11-20 09:20
flask_client.py bhushan.shelke, 2020-11-25 16:25
flask_client_long_headers.py bhushan.shelke, 2020-12-14 07:50
packet-1.PNG bhushan.shelke, 2020-12-14 08:01
packet-2.PNG bhushan.shelke, 2020-12-14 08:02
Messages (15)
msg381470 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-11-20 09:20
I have a Tomcat+Java based server exposing REST APIs. I am writing a client in python to consume those APIs. Everything is fine until I send empty body in POST request. It is a valid use case for us. If I send empty body I get 400 bad request error - Invalid character found in method name [{}POST]. HTTP method names must be tokens. If I send empty request from POSTMAN or Java or CURL it works fine, problem is only when I used python as a client. 

If I use python based server (attached a simple server program for same) then I get 405 Method not allowed instead of 400 that I get using java based server.

Server code is Following is python snippet -

json_object={}
header = {'alias': 'A', 'Content-Type' : 'application/json', 'Content-Length' : '0'} 
resp = requests.post(url, auth=(username, password), headers=header, json=json_object) 
I tried using data as well instead of json param to send payload with not much of success.

I captured the wireshark dumps to undertand it further and found that, the request tomcat received is not as per RFC2616 (https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html). Especially the part - Request-Line = Method SP Request-URI SP HTTP-Version CRLF Because I could see in from wireshark dumps it looked like - {}POST MY-APP-URI HTTP/1.1

As we can see the empty body is getting prefixed with http-method, hence tomcat reports that as an error. I then looked at python http library code -client.py. Following are relevant details -

File - client.py

Method - _send_output (starting at line # 1001) - It first sends the header at line #1010 and then the body somewhere down in the code. I thought(I could be wrong here) perhaps in this case header is way longer 310 bytes than body 2 bytes, so by the time complete header is sent on wire body is pushed and hence TCP frames are order in such a way that body appears first. To corroborate this I added a delay of 1 second just after sending header line#1011 and bingo, the error disappeared and it started working fine. Not sure if this is completely correct analysis, but can someone in the know can confirm or let me know how to fix this.
msg381471 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-20 09:27
Are you using the requests library (https://requests.readthedocs.io/en/master/)?
msg381472 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-11-20 09:33
Yes Ronald I am using requests library however I tried HTTPSConnection class from http.client package as well getting same error, may be because cause is in the core http library itself
msg381473 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-20 09:43
Please check with the requests project as well, I'm not sure how much of the stdlib HTTP client is used by requests.

This issue can stay open because the issue is reproducible using the stdlib.
msg381474 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-11-20 10:06
I have looked at requests lib code. As far as I could understand following is the flow of packages used in this case -

request "uses->" urllib3 "uses->" http
msg381835 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-11-25 15:03
Any tentative Date for fix to be generally available?
msg381836 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-25 15:16
So far we don't even know for sure if there is a bug in Python's http module. It would be helpful if you could provide a reproducer that does not use any 3rd party code like requests.
msg381840 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-25 15:29
I also noticed that you are passing an explicit content length. {'Content-Length' : '0'} is wrong for a POST payload of "{}" with standard transfer encoding. It should be len("{}") == 2.

A zero content length only valid for chunked transfer encoding, which either requires header {"transfer-encoding": "chunked"} or encode_chunked=True.
msg381846 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-11-25 16:25
I had uploaded one file(simple_flask.py) to be used as server to the ticket when I created the ticket. Since ticket allows only one file upload, I'd pasted relevant code in ticket itself. I've now uploaded the client python file(flask_client.py) to ticket which should help you reproduce this. This is not using any third party library.

Also regarding content-length - It was part triaging I was doing, I continue to get the issue even if I remove that header completely. You can notice in sample client program that I have attached.
msg382798 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-12-09 18:38
Any update on this? Did the files I submitted help in reproducing/identifying the issue?
msg382808 - (view) Author: Andrei Kulakov (andrei.avk) * Date: 2020-12-09 22:15
I cannot reproduce with 3.9.0, using attached files -- I do not get either 400 or 405; I get "None received" message:

```python3 flask_client.py                                                                                                                                                                                                                    --INS--
b'"None received"\n'
<http.client.HTTPResponse object at 0x10d288f10>
```
msg382959 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-12-14 07:50
Hmm... intriguing... I just downloaded the files, an executed again, following is response

Server
```127.0.0.1 - - [14/Dec/2020 13:07:34] "{}POST http://localhost:6000/getResponse HTTP/1.1" 405 -```

Client
```py flask_client.py
b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>405 Method Not Allowed</title>\n<h1>Method Not Allowed</h1>\n<p>The method is not allowed for the requested URL.</p>\n'
<http.client.HTTPResponse object at 0x0000020BE4532CD0>
```

Since my theory is based on - body being appended sent fist before header in TCP packet. I just uploaded a file(flask_client_long_headers.py) having long header values. Could you please use this and check if you can reproduce it now. Also please suggest me any other alternative through which I could send more relevant information.
msg382960 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-12-14 08:01
Also attaching screenshot of TCP dump captured via wireshark
Packet-1
msg382961 - (view) Author: Bhushan Shelke (bhushan.shelke) Date: 2020-12-14 08:02
Attaching screenshot of TCP dump captured via wireshark
Packet-2
msg383835 - (view) Author: Andrei Kulakov (andrei.avk) * Date: 2020-12-27 03:32
I've tried it with flask_client_long_headers.py , but I get the same output as I've gotten before with flask_client.py .

Can you try it on a few different systems?
History
Date User Action Args
2020-12-27 03:32:41andrei.avksetmessages: + msg383835
2020-12-14 08:02:43bhushan.shelkesetfiles: + packet-2.PNG

messages: + msg382961
2020-12-14 08:01:46bhushan.shelkesetfiles: + packet-1.PNG

messages: + msg382960
2020-12-14 07:50:07bhushan.shelkesetfiles: + flask_client_long_headers.py

messages: + msg382959
2020-12-09 22:15:06andrei.avksetnosy: + andrei.avk
messages: + msg382808
2020-12-09 18:38:40bhushan.shelkesetmessages: + msg382798
2020-11-25 16:25:54bhushan.shelkesetfiles: + flask_client.py

messages: + msg381846
2020-11-25 15:29:56christian.heimessetmessages: + msg381840
2020-11-25 15:16:24christian.heimessetnosy: + christian.heimes
messages: + msg381836
2020-11-25 15:03:25bhushan.shelkesetmessages: + msg381835
2020-11-20 10:06:17bhushan.shelkesetmessages: + msg381474
2020-11-20 09:43:19ronaldoussorensetmessages: + msg381473
2020-11-20 09:33:15bhushan.shelkesetmessages: + msg381472
2020-11-20 09:27:20ronaldoussorensetnosy: + ronaldoussoren
messages: + msg381471
2020-11-20 09:26:35bhushan.shelkesetcomponents: + Library (Lib)
2020-11-20 09:20:23bhushan.shelkecreate