Title: Robots.txt
Type: Stage: resolved
Components: Versions:
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: admin2, eric.smith
Priority: normal Keywords:

Created on 2020-09-17 11:50 by admin2, last changed 2020-10-31 00:06 by iritkatriel. This issue is now closed.

Messages (2)
msg377055 - (view) Author: Skyreka Agence Web (admin2) Date: 2020-09-17 11:50
I can't  documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site:, I believe that this is how it's implemented now in most parsers ?
msg377057 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-09-17 12:07
It's not clear what bug you're describing, or how it relates to Python. Can you provide more information, and show what behavior your seeing, and what behavior you're expecting?
Date User Action Args
2020-10-31 00:06:14iritkatrielsetstatus: pending -> closed
resolution: rejected
stage: resolved
2020-09-18 00:41:06eric.smithsetstatus: open -> pending
2020-09-17 12:07:25eric.smithsetnosy: + eric.smith
messages: + msg377057
2020-09-17 11:50:11admin2create