➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author karl.pradene
Recipients karl.pradene
Date 2021-05-21.19:20:59
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1621624859.22.0.835080944412.issue44209@roundup.psfhosted.org>
In-reply-to
Content 
In robotparser.py
On line 222
path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
tranform the entry Disallow: /?
in the google.com/robots.txt
in : Disallow: /
making every can_fetch request return False
History
Date User Action Args
2021-05-21 19:20:59karl.pradenesetrecipients: + karl.pradene
2021-05-21 19:20:59karl.pradenesetmessageid: <1621624859.22.0.835080944412.issue44209@roundup.psfhosted.org>
2021-05-21 19:20:59karl.pradenelinkissue44209 messages
2021-05-21 19:20:59karl.pradenecreate