Issue36064
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-02-21 10:23 by sylye, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Pull Requests | |||
---|---|---|---|
URL | Status | Linked | Edit |
PR 11976 | closed | matrixise, 2019-02-21 12:44 | |
PR 11990 | merged | mdk, 2019-02-22 15:22 |
Messages (17) | |||
---|---|---|---|
msg336198 - (view) | Author: Lye (sylye) | Date: 2019-02-21 10:23 | |
I found out in the docs 3.6, in the class urllib.request.Request, for the input of 'data' data types, it says : "The supported object types include bytes, file-like objects, and iterables." But after testing it with data type dict for the 'data' input, I got error of: "can't concat str to bytes" It seems the docs should't say the 'data' data types support iterables. There more detail discussion is at : https://stackoverflow.com/questions/54802272/some-fundamental-concept-used-in-python-docs Hope this helps, thanks ! |
|||
msg336199 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 10:27 | |
Thank you for your issue but could you provide an example file? Have a nice day. For the new contributors, this issue is marked as easy. |
|||
msg336200 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 10:44 | |
Hi Lye, I didn't read your link, sorry you don't need to provide an example. We can work with this example. |
|||
msg336202 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 10:53 | |
Hi Lye, I am not sure it's a bug because I don't have any problem with your script. Here is your script. import urllib import urllib.request my_url = "https://api.foo.com" my_headers = { "Content-Type" : "application/x-www-form-urlencoded" } my_data = { "client_id" : "ppp", "client_secret" : "000", "grant_type" : "client_credentials" } req = urllib.request.Request(url=my_url, data=my_data, headers=my_headers) response = urllib.request.urlopen(req) html = response.read() print(html) and here is the result with the version 3.6 of Python: > docker run --rm -v $PWD:/src -it python:3.6 python /src/demo.py Traceback (most recent call last): File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) ... File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/src/demo.py", line 12, in <module> response = urllib.request.urlopen(req) ... File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 111] Connection refused> I don't get your error. and the version of this docker image is 3.6.8 > docker run --rm -v $PWD:/src -it python:3.6 python --version Python 3.6.8 |
|||
msg336203 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 10:56 | |
and I get the same result with the python:3.6.7 image docker > docker run --rm -v $PWD:/src -it python:3.6.7 python /src/demo.py Traceback (most recent call last): File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) ... File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/src/demo.py", line 12, in <module> response = urllib.request.urlopen(req) ... File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 111] Connection refused> I am really sorry but I can't reproduce your issue. What is your Linux distribution or OS? Thank you |
|||
msg336208 - (view) | Author: Lye (sylye) | Date: 2019-02-21 11:32 | |
Hi @matrixise, Sorry that the example in that stackoverflow site is a dummy API server. I tested my code in a real API server so I can produce the error. I have re-tested my code with that dummy site and yes you won't able to get that error. To reproduce that error, you may try a valid API server, like 'https://httpbin.org/post' . I just tested again by replacing : my_url = "https://api.foo.com" with my_url = "https://httpbin.org/post" and able to get the same error again. Apology for not using a real API server in my example :) |
|||
msg336209 - (view) | Author: Lye (sylye) | Date: 2019-02-21 11:34 | |
And I tested in both : Centos 6.10 4.14.77-70.59.amzn1.x86_64 |
|||
msg336210 - (view) | Author: Karthikeyan Singaravelan (xtreak) * | Date: 2019-02-21 12:00 | |
I guess key should be bytes instead of a dict with string key. Maybe the docs could be improved about the case where key in case of dict should be bytes. >>> from urllib.request import urlopen, Request >>> r = Request("http://httpbin.org/post", data={b'a': 'a'}) >>> resp = urlopen(r) >>> print(resp.read()) b'{\n "args": {}, \n "data": "", \n "files": {}, \n "form": {\n "a": ""\n }, \n "headers": {\n "Accept-Encoding": "identity", \n "Content-Length": "1", \n "Content-Type": "application/x-www-form-urlencoded", \n "Host": "httpbin.org", \n "User-Agent": "Python-urllib/3.8"\n }, \n "json": null, \n "origin": "123.201.136.26, 123.201.136.26", \n "url": "https://httpbin.org/post"\n}\n' See https://bugs.python.org/issue25439 for an open patch to ensure this is type checked. >>> from urllib.request import urlopen, Request >>> r = Request("http://httpbin.org/post", data={'a': 'a'}) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/urllib/request.py", line 332, in __init__ self.data = data File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/urllib/request.py", line 370, in data raise TypeError("Key should be type of bytes in POST data") TypeError: Key should be type of bytes in POST data Current code is not checking the type makes it pass during construction and then throw an error later like below : >>> from urllib.request import urlopen, Request >>> r = Request("http://httpbin.org/post", data={'a': 'a'}) >>> urlopen(r) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open response = self._open(req, data) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open '_open', req) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1345, in http_open return self.do_open(http.client.HTTPConnection, req) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request self._send_request(method, url, body, headers, encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1054, in _send_output + b'\r\n' TypeError: can't concat str to bytes |
|||
msg336213 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 12:28 | |
ok, I confirm this issue, now, with httpbin.org I get the same issue. and it's just when we try to send the stream to the server. |
|||
msg336214 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 12:38 | |
@xtreak ok because it's not related to the documentation, I have a fix for this issue but I am not sure about the title for this PR but what do you suggest? "Fix the encoding in _send_output when sending an iterable object via urllib.request.Request" ? |
|||
msg336215 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 12:46 | |
@xtreak I have just pushed a PR, if you think it's an issue related to the documentation, you can ignore my PR and in this case I will close it. now, it's not related to the doc, could you review it and give me your feedback? Thank you, |
|||
msg336216 - (view) | Author: Karthikeyan Singaravelan (xtreak) * | Date: 2019-02-21 14:02 | |
Stéphane, I don't know much about urllib to review this. The reason I said it's better document this is that there was an extended discussion on issue25439 along with a PR about type checking breaking some behavior and ambiguity in what is allowed currently. This PR seems to encode str to bytes implicitly and it needs to be documented as such that this str is encoded to bytes under the hood. I am not sure of backporting it as a fix to stable releases. I think it's better to raise an error instead of doing something implicitly. I am adding Martin and Senthil to the issue who might have a better context. |
|||
msg336217 - (view) | Author: Stéphane Wirtel (matrixise) * | Date: 2019-02-21 14:15 | |
Thank you @xtreak for adding Martin and Senthil. |
|||
msg336290 - (view) | Author: Martin Panter (martin.panter) * | Date: 2019-02-22 10:16 | |
I agree the documentation is insufficient. It should have said if “data” is iterated, it must yield bytes-like objects. I agree it is unwise to yet more special cases for the uploaded data types. When Lye passed the dictionary of three keys and values, I suspect they weren’t expecting it to upload just the key names using one HTTP chunk for each key. See Issue 23740 about the mess of data types currently supported. |
|||
msg339092 - (view) | Author: Carol Willing (willingc) * | Date: 2019-03-29 02:15 | |
New changeset 9e30fbac019d9c0a31d444a711e845c7e883caf5 by Carol Willing (Julien Palard) in branch 'master': bpo-36064: Clarify allowed data types for urllib.request.Request. (GH-11990) https://github.com/python/cpython/commit/9e30fbac019d9c0a31d444a711e845c7e883caf5 |
|||
msg339093 - (view) | Author: Carol Willing (willingc) * | Date: 2019-03-29 02:18 | |
I've merged Julien Palard's doc PR. Is this sufficient to close this issue? |
|||
msg339096 - (view) | Author: Karthikeyan Singaravelan (xtreak) * | Date: 2019-03-29 04:53 | |
I would propose closing it since docs were made improved and if type checking needs to be enforced then it can be discussed in the linked open issues that have patches. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:11 | admin | set | github: 80245 |
2019-03-29 09:13:50 | mdk | set | status: open -> closed resolution: fixed stage: patch review -> resolved |
2019-03-29 04:53:31 | xtreak | set | messages: + msg339096 |
2019-03-29 02:18:42 | willingc | set | messages: + msg339093 |
2019-03-29 02:15:36 | willingc | set | nosy:
+ willingc messages: + msg339092 |
2019-02-22 15:22:31 | mdk | set | pull_requests: + pull_request12013 |
2019-02-22 10:16:03 | martin.panter | set | messages: + msg336290 |
2019-02-21 14:15:04 | matrixise | set | messages: + msg336217 |
2019-02-21 14:02:53 | xtreak | set | nosy:
+ orsenthil, martin.panter messages: + msg336216 |
2019-02-21 12:46:23 | matrixise | set | messages: + msg336215 |
2019-02-21 12:44:44 | matrixise | set | keywords:
+ patch stage: patch review pull_requests: + pull_request12000 |
2019-02-21 12:38:27 | matrixise | set | messages: + msg336214 |
2019-02-21 12:28:01 | matrixise | set | messages: + msg336213 |
2019-02-21 12:00:29 | xtreak | set | nosy:
+ xtreak messages: + msg336210 |
2019-02-21 11:34:00 | sylye | set | messages: + msg336209 |
2019-02-21 11:32:58 | sylye | set | messages: + msg336208 |
2019-02-21 10:56:15 | matrixise | set | messages: + msg336203 |
2019-02-21 10:53:34 | matrixise | set | messages: + msg336202 |
2019-02-21 10:44:36 | matrixise | set | messages: + msg336200 |
2019-02-21 10:27:45 | matrixise | set | keywords:
+ easy nosy: + matrixise messages: + msg336199 |
2019-02-21 10:23:41 | sylye | create |