➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add headers parameter on RobotFileParser
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eamanu
Priority: normal Keywords:

Created on 2019-09-09 12:28 by eamanu, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 15726 eamanu, 2019-09-09 12:28
Messages (1)
msg351440 - (view) Author: Emmanuel Arias (eamanu) * Date: 2019-09-09 12:28 
Hello I open an issue from a PR that I think need discussion:

I write here the @fuwaraido's PR description:

> I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.
History
Date User Action Args
2022-04-11 14:59:20adminsetgithub: 82248
2019-09-09 12:28:17eamanucreate