➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author eamanu
Recipients eamanu
Date 2019-09-09.12:28:17
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1568032097.51.0.453299797445.issue38067@roundup.psfhosted.org>
In-reply-to
Content 
Hello I open an issue from a PR that I think need discussion:

I write here the @fuwaraido's PR description:

> I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.
History
Date User Action Args
2019-09-09 12:28:17eamanusetrecipients: + eamanu
2019-09-09 12:28:17eamanusetmessageid: <1568032097.51.0.453299797445.issue38067@roundup.psfhosted.org>
2019-09-09 12:28:17eamanulinkissue38067 messages
2019-09-09 12:28:17eamanucreate