Why do downloaded files start with <!DOCTYPE?

AMD64 Debian root 3.x:
https://buildbot.python.org/all/#/builders/345/builds/132

5 tests failed:
    test_codecmaps_cn test_codecmaps_jp test_codecmaps_kr
    test_codecmaps_tw test_hashlib

I'm not sure if all failures are download issues.


Examples:

ERROR: test_mapping_file (test.test_codecmaps_kr.TestCP949Map)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/multibytecodec_support.py", line 305, in test_mapping_file
    self._test_mapping_file_plain()
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/multibytecodec_support.py", line 319, in _test_mapping_file_plain
    csetval = eval(data[0])
  File "<string>", line 1
    <!DOCTYPE
    ^
SyntaxError: invalid syntax


ERROR: test_named_sequences_full (test.test_ucn.UnicodeNamesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_ucn.py", line 181, in test_named_sequences_full
    testdata = support.open_urlresource(url, encoding="utf-8",
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 540, in open_urlresource
    raise TestFailed('invalid resource %r' % fn)
test.support.TestFailed: invalid resource '/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/data/NamedSequences.txt'


ERROR: test_blake2b_vectors (test.test_hashlib.HashLibTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_hashlib.py", line 733, in test_blake2b_vectors
    for msg, key, md in read_vectors('blake2b'):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_hashlib.py", line 85, in read_vectors
    parts[0] = bytes.fromhex(parts[0])
ValueError: non-hexadecimal number found in fromhex() arg at position 0



AMD64 Debian root 3.9 has similar issues.

It is also not safe to pass data downloaded from untrusted source to eval().

> It is also not safe to pass data downloaded from untrusted source to eval().

To make matters worse, it's downloaded via HTTP (rather than HTTPS) - so anyone who can mess with the network of a machine running the Python testsuite can run arbitrary code on that machine.

(I contacted security@python.org about this a couple of hours ago, but I guess this is effectively public now anyways :D)

> It is also not safe to pass data downloaded from untrusted source to eval().

Would you mind to open a separated issue for that?

Does someone want to propose a change to avoid eval() in the tests?

Has this been ongoing, or is it something that started in the past day and a half? I've had intermittent internet issues (and expect them to continue for another half day or thereabouts), so it's possible that this is actually a symptom of that.

If that's the case, then the issue would be that my gateway returned a failure page of some sort, which would be another good reason to download the content via HTTPS (since that response happens only on non-encrypted downloads).

Hi,

seems that this test doesn't occur ? or I'm very wrong? 

https://buildbot.python.org/all/#/builders/345

Yeah, it seems like Chris's Internet issues are gone ;-)

That ducks the immediate issue, but I think the underlying issue would still be worth solving. How hard would it be to fetch this via an encrypted connection? It's still not 100% secure, but it would be resilient against some of these sorts of issues.

If someone wants to use HTTPS on pythontest.net, please open a new issue. I don't recall who operates this service. Ask Ernest or Benjamin maybe? See https://github.com/python/pythontestdotnet/ project. https://pythontest.net/ has an invalid certificate. Ernest might be able to set up a valid certificate.

For buildbot failures, I prefer to be able to close issues as soon as possible, since I manage tons of buildbot failures. Otherwise, I'm quickly overwhelmed.

The initial issue "AMD64 Debian root 3.x fails" is fixed ;-)