Message389352
IMO this could be enhanced by adding a sslcontext parameter to read method
a sample change would it could look like
```
def read(self, sslcontext=None):
"""Reads the robots.txt URL and feeds it to the parser."""
try:
if sslcontext:
f = urllib.request.urlopen(self.url, context=sslcontext)
else:
f = urllib.request.urlopen(self.url)
except urllib.error.HTTPError as err:
if err.code in (401, 403):
self.disallow_all = True
elif err.code >= 400 and err.code < 500:
self.allow_all = True
else:
raw = f.read()
self.parse(raw.decode("utf-8").splitlines())
```
Happy to send a PR if this proposal makes sense. |
|
Date |
User |
Action |
Args |
2021-03-22 23:25:16 | Tchinmai7 | set | recipients:
+ Tchinmai7 |
2021-03-22 23:25:16 | Tchinmai7 | set | messageid: <1616455516.19.0.272349627897.issue43597@roundup.psfhosted.org> |
2021-03-22 23:25:16 | Tchinmai7 | link | issue43597 messages |
2021-03-22 23:25:16 | Tchinmai7 | create | |
|