This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: robotsparser deny all with some rules
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: contact, terry.reedy
Priority: normal Keywords:

Created on 2020-11-23 17:53 by contact, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (3)
msg381683 - (view) Author: Net Offensive (contact) Date: 2020-11-23 17:53

Notre développeur a un soucis avec l'utilisation de cette librairie. Dans le cadre d'un projet SEO, nous souhaiterions scrapper les pages de notre réseau de site.

Nous avons essayé de tester avec l'un de nos site dont les pages se présentent comme ce guide sur le référencement :

Elle ne sont pas détectées comme des pages à cause de leur forme en repertoire. A ton besoin forcément de créer des pages du type : page.ext ?

C'est pourtant un format utilisé sur Wordpress et d'autres CMS.




Our developer has a problem with the use of this library. As part of an SEO project, we would like to scramble the pages of our site network.

We tried to test with one of our site whose pages look like this SEO guide:

They are not detected as pages because of their directory shape. Do you necessarily need to create pages of the type: page.ext?

It is however a format used on Wordpress and other CMS.


Translated with (free version)
msg381685 - (view) Author: Net Offensive (contact) Date: 2020-11-23 17:55
Sorry if my message is not clear enough. I will ask my developer to come and complete the information if needed.
msg381977 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-11-28 05:56
This issue tracker is for proposing changes to the github CPython repository, used to make python releases.  Your post does not propose a change and does not demonstrate that there is a bug in current Python, which is 3.9.  So my current opinion is that this issue should be closed as 'not a bug'.  Questions about using Python should be directed to a user help forum, such as python-list or or some equivalent in French.
Date User Action Args
2022-04-11 14:59:38adminsetgithub: 86613
2021-01-21 21:59:10terry.reedysetstatus: open -> closed
resolution: not a bug
stage: resolved
2020-11-28 05:56:41terry.reedysetnosy: + terry.reedy
messages: + msg381977
2020-11-23 17:55:51contactsetmessages: + msg381685
2020-11-23 17:53:43contactcreate