➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: urllib.robotparser fail on Disallow: /? from google.com
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: karl.pradene
Priority: normal Keywords:

Created on 2021-05-21 19:20 by karl.pradene, last changed 2022-04-11 14:59 by admin.

Messages (1)
msg394144 - (view) Author: Karl Y. Pradene (karl.pradene) Date: 2021-05-21 19:20 
In robotparser.py
On line 222
path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
tranform the entry Disallow: /?
in the google.com/robots.txt
in : Disallow: /
making every can_fetch request return False
History
Date User Action Args
2022-04-11 14:59:46adminsetgithub: 88375
2021-05-21 19:20:59karl.pradenecreate