➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author admin2
Recipients admin2
Date 2020-09-17.11:50:11
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1600343411.93.0.266726612445.issue41803@roundup.psfhosted.org>
In-reply-to
Content 
I can't  documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.skyreka.com/robots.txt, I believe that this is how it's implemented now in most parsers ?
History
Date User Action Args
2020-09-17 11:50:11admin2setrecipients: + admin2
2020-09-17 11:50:11admin2setmessageid: <1600343411.93.0.266726612445.issue41803@roundup.psfhosted.org>
2020-09-17 11:50:11admin2linkissue41803 messages
2020-09-17 11:50:11admin2create