This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Crawler doesn't follow redirection
Type: behavior Stage: resolved
Components: Distutils2 Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: tarek Nosy List: alexis, eric.araujo, gruszczy, raduv, tarek
Priority: normal Keywords:

Created on 2011-06-17 20:13 by gruszczy, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (5)
msg138547 - (view) Author: Filip Gruszczyński (gruszczy) Date: 2011-06-17 20:13
When looking at issue #12348 I have noticed following behaviour:

from packaging.pypi.simple import Crawler
c = Crawler()
c.get_releases('webob')

Traceback (most recent call last):
  File "test_crawl.py", line 3, in <module>
    c.get_releases('webob')
  File "/home/gruszczy/Projects/python/Lib/packaging/pypi/simple.py", line 192, in get_releases
    raise ProjectNotFound
packaging.pypi.errors.ProjectNotFound

This is because when request for /simple/webob/ is made 301 is returned to /simple/WebOb/ and HttpError raised, which is silenced and returned:

        except urllib.error.HTTPError as v:
            return v

Wouldn't it be good to do something about? How about crawler following redirects?
msg138549 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-17 20:36
Sure, let’s follow redirections (and log them).
msg138550 - (view) Author: Filip Gruszczyński (gruszczy) Date: 2011-06-17 20:37
OK, I'll try to come up with some patch.
msg161235 - (view) Author: Radu Voicilas (raduv) Date: 2012-05-20 22:25
I can no longer reproduce the issue on tip. Here's the output I get trying to do c.get_releases('webob'):

$ ./python issue12355.py 
version has not been parsed: WebOb-0.8.3-py2.5
version has not been parsed: WebOb-0.9.1-py2.4
version has not been parsed: WebOb-0.8-py2.4
version has not been parsed: WebOb-0.8.4-py2.5
version has not been parsed: WebOb-0.8.4-py2.4
version has not been parsed: WebOb-0.8.5-py2.4
version has not been parsed: WebOb-0.9-py2.4
version has not been parsed: WebOb-0.8.5-py2.5
version has not been parsed: WebOb-0.9.1-py2.5
version has not been parsed: WebOb-0.8.3-py2.4
version has not been parsed: WebOb-0.8.1-py2.4
version has not been parsed: WebOb-0.8.2-py2.4
version has not been parsed: WebOb-0.9-py2.5
version has not been parsed: WebOb-0.8.2-py2.5
[103083 refs]
msg161869 - (view) Author: Alexis Metaireau (alexis) * (Python triager) Date: 2012-05-29 11:02
Did someone changed anything in the codebase regarding this (or did it solved itself magically?)
History
Date User Action Args
2022-04-11 14:57:18adminsetgithub: 56564
2014-03-13 20:31:19eric.araujosetstatus: open -> closed
resolution: out of date
stage: needs patch -> resolved
2012-05-29 11:02:13alexissetmessages: + msg161869
2012-05-20 22:25:00raduvsetnosy: + raduv
messages: + msg161235
2011-06-17 20:37:52gruszczysetmessages: + msg138550
2011-06-17 20:36:24eric.araujosettype: crash -> behavior
messages: + msg138549
stage: needs patch
2011-06-17 20:13:31gruszczycreate