Author xtreak
Recipients berker.peksag, gallicrooster, larsfuse, terry.reedy, xtreak
Date 2020-01-02.08:36:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
There is a behavior change. parse() sets the modified time and unless the modified time is set the can_fetch method returns false. In Python 2 the parse method was called only when the file is non-empty [0] but in Python 3 it's always called though the file is empty [1] . The change was done with 1afc1696167547a5fa101c53e5a3ab4717f8852c to always read parse and then in 122541beceeccce4ef8a9bf739c727ccdcbf2f28 modified function was always called during parse thus setting the modified_time to return True from can_fetch in the end.

I think the behavior of robotparser for empty file was undefined allowing these changes and it will be good to have a test for this behavior.

Date User Action Args
2020-01-02 08:36:56xtreaksetrecipients: + xtreak, terry.reedy, berker.peksag, larsfuse, gallicrooster
2020-01-02 08:36:56xtreaksetmessageid: <>
2020-01-02 08:36:56xtreaklinkissue35457 messages
2020-01-02 08:36:55xtreakcreate