➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author matelesecretaire67
Recipients gallicrooster, matelesecretaire67
Date 2022-04-06.14:23:27
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1649255007.06.0.508408003025.issue39187@roundup.psfhosted.org>
In-reply-to
Content 
I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: https://www.st-info.fr/robots.txt, I believe that this is how it's implemented now in most parsers ?
History
Date User Action Args
2022-04-06 14:23:27matelesecretaire67setrecipients: + matelesecretaire67, gallicrooster
2022-04-06 14:23:27matelesecretaire67setmessageid: <1649255007.06.0.508408003025.issue39187@roundup.psfhosted.org>
2022-04-06 14:23:27matelesecretaire67linkissue39187 messages
2022-04-06 14:23:27matelesecretaire67create