Author Tchinmai7
Recipients Tchinmai7
Date 2021-03-22.23:25:16
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1616455516.19.0.272349627897.issue43597@roundup.psfhosted.org>
In-reply-to
Content
IMO this could be enhanced by adding a sslcontext parameter to read method

a sample change would it could look like
```
def read(self, sslcontext=None):
    """Reads the robots.txt URL and feeds it to the parser."""
    try:
        if sslcontext:
           f = urllib.request.urlopen(self.url, context=sslcontext)
        else:
           f = urllib.request.urlopen(self.url)
    except urllib.error.HTTPError as err:
        if err.code in (401, 403):
            self.disallow_all = True
        elif err.code >= 400 and err.code < 500:
            self.allow_all = True
    else:
        raw = f.read()
        self.parse(raw.decode("utf-8").splitlines())

```

Happy to send a PR if this proposal makes sense.
History
Date User Action Args
2021-03-22 23:25:16Tchinmai7setrecipients: + Tchinmai7
2021-03-22 23:25:16Tchinmai7setmessageid: <1616455516.19.0.272349627897.issue43597@roundup.psfhosted.org>
2021-03-22 23:25:16Tchinmai7linkissue43597 messages
2021-03-22 23:25:16Tchinmai7create