I tried the patch by doing this:

import robotparser   
def prompt_user_passwd(self, host, realm):
    return None, None
robotparser.URLopener.prompt_user_passwd = prompt_user_passwd    # temp patch


This dealt with the problem effectively; robots.txt files are being processed normally, and if reading one causes an authentication request, it's handled as if no password was input, without any interaction.  

So this could probably go in.