Issue22450
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2014-09-20 23:50 by rhettinger, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
accept.diff | rhettinger, 2014-09-20 23:50 |
Messages (19) | |||
---|---|---|---|
msg227194 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2014-09-20 23:50 | |
The use of urllib for REST APIs is impaired in the absence of a "Accept: */*" header such as that added automatically by the requests package or by the CURL command-line tool. # Example that gets an incorrect result due to the missing header import urllib print urllib.urlopen('http://graph.facebook.com/raymondh').headers['Content-Type'] # Equivalent call using CURL $ curl -v http://graph.facebook.com/raymondh ... * Connected to graph.facebook.com (31.13.75.1) port 80 (#0) > GET /raymondh HTTP/1.1 > User-Agent: curl/7.30.0 > Host: graph.facebook.com > Accept: */* > |
|||
msg227195 - (view) | Author: Senthil Kumaran (orsenthil) * | Date: 2014-09-21 00:00 | |
Patch looks good. Will need similar addition in urllib2 and inclusion of tests. |
|||
msg227196 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2014-09-21 00:00 | |
Can you explain how the result is incorrect? >>> f = urllib.request.urlopen('http://graph.facebook.com/raymondh') >>> json.loads(f.read().decode()) {'link': 'https://www.facebook.com/raymondh', 'id': '562805507', 'last_name': 'Hettinger', 'gender': 'male', 'first_name': 'Raymond', 'name': 'Raymond Hettinger', 'locale': 'en_US', 'username': 'raymondh'} |
|||
msg227197 - (view) | Author: Senthil Kumaran (orsenthil) * | Date: 2014-09-21 00:03 | |
Well, the result with loading using json will be same. but without sending Accept */*. The content-type returned is text/javascript; charset=UTF-8 and with sending of Accept */* the content-type is set to application/json; charset=UTF-8 (which is more desirable). |
|||
msg227198 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2014-09-21 00:07 | |
> The content-type returned is text/javascript; charset=UTF-8 and with > sending of Accept */* the content-type is set to application/json; > charset=UTF-8 (which is more desirable). Is that a bug in urllib, or in Facebook's HTTP implementation? Frankly, we shouldn't jump to conclusions just because one specific use case is made better by this. Forcing an accept header may totally change the output of other servers and break existing uses. (and besides, the content-type header is unimportant when you know what to expect, which is normally the case when calling an API) |
|||
msg240468 - (view) | Author: Martin Panter (martin.panter) * | Date: 2015-04-11 10:56 | |
The RFC <https://tools.ietf.org/html/rfc7231#page-39> says “A request without any Accept header field implies that the user agent will accept any media type in response”, which sounds the same as “Accept: */*”. I don’t understand why adding it should make a real difference. If you really desire only application/json, you should probably include “Accept: application/json” in the request. Otherwise, it would probably be more robust to make your program accept both types. I have come across the same deal with application/atom+xml vs text/xml vs application/xml. |
|||
msg253813 - (view) | Author: Martin Panter (martin.panter) * | Date: 2015-10-31 22:33 | |
I propose rejecting this one, in favour of the caller adding their own “Accept: */*” (or more preferably, “Accept: application/json”) header. What do you think, Raymond or Senthil? |
|||
msg253828 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2015-11-01 07:18 | |
> What do you think, Raymond Before dismissing this, we should get a better understanding of why "Accept: */*" is so widely used in practice. Here's what we know so far: * The header made a difference to the Facebook Graph API. * Curl (a minimalist) includes "Accept: */*", Host, and User-Agent. * Firefox includes "*/*" at the end of its list of acceptable types. * Kenneth Reitz's requests module uses "Accept: */*" by default. * The poolmanager in urllib3 uses "Accept: */*" by default and has a comment that that and the "Host" header are both needed by proxies. * I'm also seeing "Accept: */*" in book examples as well. See https://books.google.com/books?id=fVuWayXLdYIC&pg=PA22 and http://doc.bonfire-project.eu/R1/api/example-session.html |
|||
msg253834 - (view) | Author: Martin Panter (martin.panter) * | Date: 2015-11-01 09:30 | |
According to all the HTTP 1.1 RFCs, having */* at the end means you accept any other content type if none of the higher priority ones are available (otherwise you risk a 406 Not Acceptable error). So that explains why Firefox has */* tacked on. Requests copied from Curl: <https://github.com/kennethreitz/requests/commit/6140fac>. Similarly, it is in urllib3 “because that’s what cURL had by default”. Brief discussion at <https://github.com/shazow/urllib3/pull/93#issuecomment-8209904>, where they decided to leave things as they already were. So all roads seem to lead to Curl. Curl’s “initial revision” (Dec 1999) had “Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*”, which was changed to “Accept: */*” in <https://github.com/bagder/curl/commit/93e32e6> in 2004. I don’t see any reasons given. I just left a question on Git Hub about this, so maybe we might get some sort of answer. Wget also includes “Accept: */*”. But it gives no explanations either, and it was present right from the “initial revision” also in Dec 1999 (presumably Source Forge started about then). |
|||
msg253906 - (view) | Author: Martin Panter (martin.panter) * | Date: 2015-11-02 11:27 | |
The Curl programmer replied basically saying there was no scientific reason, but since Curl was previously sending a custom Accept header, it was safer to leave a bare-bones Accept header in than completely remove it. Plus he thought it might be slighly more compatible with websites. |
|||
msg273968 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2016-08-30 21:17 | |
Update: After more research, I learned that while 'Accept: */*' should not have an effect on the origin webserver, it can and does have an effect on proxy servers. Origin servers are allowed to vary the content-type of responses when given different Accept headers. When they do so, they should also send "Vary: Accept". Proxy servers such as NGinx and Varnish respond to the "Vary: Accept" by caching the different responses using a combination of url and the accept header as the cache key. If the request has 'Accept: */*', then the cache lookup returns the same result as if the 'Accept: */*' had been passed directly to the server. However, if the Accept header is omitted, the proxy cache can return any of the cached responses (typically the most recent, regardless of content-type). Accordingly, it is a good practice to include 'Accept: */*' in the request so that you get a consistent result (what the server would have returned) rather than the inconsistent and unpredictable content-types you would receive in the absence of the Accept header. I believe that is why the other tools and book examples use 'Accept: */*' even though the origin wouldn't care. |
|||
msg273970 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2016-08-30 21:21 | |
Putting it another way: To an origin server, 'Accept: */*' means it can return anything it wants. To a proxy server, the absence of an accept header means in can return anything it has cached (possibly different from what the origin server would have returned). In contract, to a proxy server, 'Accept: */*' means return exactly what the origin server would have returned with the same headers. |
|||
msg273982 - (view) | Author: Martin Panter (martin.panter) * | Date: 2016-08-31 00:33 | |
“Proxy servers such as NGinx and Varnish: . . . if the Accept header is omitted, the proxy cache can return any of the cached responses.” This is not really my area of expertise, but this behaviour is inconsistent with my understanding of how Accept and Vary are supposed to work in general. I would expect a cache to treat a missing Accept field as a separate “value” that does not match any specific Accept value. See <https://www.w3.org/mid/20040223204041.GA32719@mail.shareable.org>. Also, what about a server that sets “Vary: Cookie”, to send a response that depends on whether the user has already seen the page. Do these NGinx and Varnish caches respond with a random response if Cookie is missing? I still think if you care about the media type, it is better practice to specify what types you want with a more explicit Accept value. And if you don’t care about the media type, the NGinx/Varnish behaviour may not be a problem anyway. |
|||
msg275474 - (view) | Author: Cory Benfield (Lukasa) * | Date: 2016-09-09 22:24 | |
So, leaping in on the Requests side of things for a moment, two notes. Firstly: copying curl is rarely a bad thing to do, especially for a behaviour curl has had for a long time. However, in this case the stronger argument is that just because the RFCs say that Accept: */* is implied doesn't mean it can safely be omitted. In practice, origin servers behave unexpectedly when the header is omitted, and in general behave more predictable when it is emitted. For that reason, it should be added by Python's standard library. HTTP/1.1 is a protocol where "as deployed" means much more than "as specified", sadly. |
|||
msg275487 - (view) | Author: Martin Panter (martin.panter) * | Date: 2016-09-09 23:09 | |
I’m still not convinced. But my argument about the user specifying Accept if they care about the media type works both ways, so I am not that fussed if others want to make the change. Are there any examples of servers that behave worse than the application/json vs text/json example? E.g. returning XML vs JSON or something? |
|||
msg275489 - (view) | Author: Roundup Robot (python-dev) | Date: 2016-09-09 23:24 | |
New changeset e84105b48436 by Raymond Hettinger in branch '2.7': Issue #22450: Use "Accept: */*" in the default headers for urllib https://hg.python.org/cpython/rev/e84105b48436 |
|||
msg275498 - (view) | Author: Kenneth Reitz (kennethreitz) | Date: 2016-09-09 23:43 | |
I fully second Corey's comment. |
|||
msg275499 - (view) | Author: Roundup Robot (python-dev) | Date: 2016-09-09 23:45 | |
New changeset 00da8bfa2a60 by Raymond Hettinger in branch '3.5': Issue #22450: Use "Accept: */*" in the default headers for urllib.request https://hg.python.org/cpython/rev/00da8bfa2a60 |
|||
msg275501 - (view) | Author: Senthil Kumaran (orsenthil) * | Date: 2016-09-09 23:48 | |
@Martin, I weight in 'curl's behavior for de-facto things that differ slightly from standards. It's simply what folks have gotten used to, and sometimes expect. @Raymond, unit-tests will be a good addition too. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:08 | admin | set | github: 66640 |
2016-09-09 23:48:48 | orsenthil | set | messages: + msg275501 |
2016-09-09 23:46:16 | rhettinger | set | status: open -> closed resolution: fixed |
2016-09-09 23:45:26 | python-dev | set | messages: + msg275499 |
2016-09-09 23:43:29 | kennethreitz | set | nosy:
+ kennethreitz messages: + msg275498 |
2016-09-09 23:24:36 | python-dev | set | nosy:
+ python-dev messages: + msg275489 |
2016-09-09 23:09:30 | martin.panter | set | messages: + msg275487 |
2016-09-09 22:24:14 | Lukasa | set | nosy:
+ Lukasa messages: + msg275474 |
2016-08-31 00:33:28 | martin.panter | set | messages: + msg273982 |
2016-08-30 21:21:59 | rhettinger | set | messages: + msg273970 |
2016-08-30 21:17:55 | rhettinger | set | messages: + msg273968 |
2015-11-02 11:27:45 | martin.panter | set | messages: + msg253906 |
2015-11-01 09:30:43 | martin.panter | set | messages: + msg253834 |
2015-11-01 07:18:04 | rhettinger | set | messages: + msg253828 |
2015-10-31 22:33:22 | martin.panter | set | messages: + msg253813 |
2015-04-11 10:56:55 | martin.panter | set | nosy:
+ martin.panter messages: + msg240468 |
2014-09-21 09:47:11 | Arfrever | set | nosy:
+ Arfrever |
2014-09-21 00:07:18 | pitrou | set | messages: + msg227198 |
2014-09-21 00:03:52 | orsenthil | set | messages: + msg227197 |
2014-09-21 00:00:21 | pitrou | set | nosy:
+ pitrou messages: + msg227196 |
2014-09-21 00:00:01 | orsenthil | set | nosy:
+ orsenthil messages: + msg227195 versions: + Python 3.4, Python 3.5 |
2014-09-20 23:50:32 | rhettinger | create |