This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author larsfuse
Recipients larsfuse, terry.reedy
Date 2018-12-17.10:02:06
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1545040926.74.0.788709270274.issue35457@psf.upfronthosting.co.za>
In-reply-to
Content
> (...) refers users, for file structure, to http://www.robotstxt.org/orig.html. This says nothing about the effect of an empty file, so I don't see this as a bug.

That is incorrect. From that url you can find:
> The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

So this is definitely a bug.
History
Date User Action Args
2018-12-17 10:02:06larsfusesetrecipients: + larsfuse, terry.reedy
2018-12-17 10:02:06larsfusesetmessageid: <1545040926.74.0.788709270274.issue35457@psf.upfronthosting.co.za>
2018-12-17 10:02:06larsfuselinkissue35457 messages
2018-12-17 10:02:06larsfusecreate