classification
Title: AMD64 Debian root 3.x: tests fail because downloaded files start with:
Type: Stage: resolved
Components: Tests Versions: Python 3.10
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Rosuav, The Compiler, eamanu, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2020-10-05 08:48 by vstinner, last changed 2020-10-11 12:51 by vstinner. This issue is now closed.

Messages (9)
msg378003 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-05 08:48
Why do downloaded files start with <!DOCTYPE?

AMD64 Debian root 3.x:
https://buildbot.python.org/all/#/builders/345/builds/132

5 tests failed:
    test_codecmaps_cn test_codecmaps_jp test_codecmaps_kr
    test_codecmaps_tw test_hashlib

I'm not sure if all failures are download issues.


Examples:

ERROR: test_mapping_file (test.test_codecmaps_kr.TestCP949Map)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/multibytecodec_support.py", line 305, in test_mapping_file
    self._test_mapping_file_plain()
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/multibytecodec_support.py", line 319, in _test_mapping_file_plain
    csetval = eval(data[0])
  File "<string>", line 1
    <!DOCTYPE
    ^
SyntaxError: invalid syntax


ERROR: test_named_sequences_full (test.test_ucn.UnicodeNamesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_ucn.py", line 181, in test_named_sequences_full
    testdata = support.open_urlresource(url, encoding="utf-8",
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 540, in open_urlresource
    raise TestFailed('invalid resource %r' % fn)
test.support.TestFailed: invalid resource '/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/data/NamedSequences.txt'


ERROR: test_blake2b_vectors (test.test_hashlib.HashLibTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_hashlib.py", line 733, in test_blake2b_vectors
    for msg, key, md in read_vectors('blake2b'):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_hashlib.py", line 85, in read_vectors
    parts[0] = bytes.fromhex(parts[0])
ValueError: non-hexadecimal number found in fromhex() arg at position 0



AMD64 Debian root 3.9 has similar issues.
msg378009 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-10-05 09:31
It is also not safe to pass data downloaded from untrusted source to eval().
msg378017 - (view) Author: Florian Bruhin (The Compiler) * Date: 2020-10-05 10:51
> It is also not safe to pass data downloaded from untrusted source to eval().

To make matters worse, it's downloaded via HTTP (rather than HTTPS) - so anyone who can mess with the network of a machine running the Python testsuite can run arbitrary code on that machine.

(I contacted security@python.org about this a couple of hours ago, but I guess this is effectively public now anyways :D)
msg378018 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-05 10:56
> It is also not safe to pass data downloaded from untrusted source to eval().

Would you mind to open a separated issue for that?

Does someone want to propose a change to avoid eval() in the tests?
msg378032 - (view) Author: Chris Angelico (Rosuav) * Date: 2020-10-05 14:29
Has this been ongoing, or is it something that started in the past day and a half? I've had intermittent internet issues (and expect them to continue for another half day or thereabouts), so it's possible that this is actually a symptom of that.

If that's the case, then the issue would be that my gateway returned a failure page of some sort, which would be another good reason to download the content via HTTPS (since that response happens only on non-encrypted downloads).
msg378347 - (view) Author: Emmanuel Arias (eamanu) * Date: 2020-10-09 20:14
Hi,

seems that this test doesn't occur ? or I'm very wrong? 

https://buildbot.python.org/all/#/builders/345
msg378428 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-11 12:27
Yeah, it seems like Chris's Internet issues are gone ;-)
msg378431 - (view) Author: Chris Angelico (Rosuav) * Date: 2020-10-11 12:35
That ducks the immediate issue, but I think the underlying issue would still be worth solving. How hard would it be to fetch this via an encrypted connection? It's still not 100% secure, but it would be resilient against some of these sorts of issues.
msg378433 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-11 12:51
If someone wants to use HTTPS on pythontest.net, please open a new issue. I don't recall who operates this service. Ask Ernest or Benjamin maybe? See https://github.com/python/pythontestdotnet/ project. https://pythontest.net/ has an invalid certificate. Ernest might be able to set up a valid certificate.

For buildbot failures, I prefer to be able to close issues as soon as possible, since I manage tons of buildbot failures. Otherwise, I'm quickly overwhelmed.

The initial issue "AMD64 Debian root 3.x fails" is fixed ;-)
History
Date User Action Args
2020-10-11 12:51:31vstinnersetmessages: + msg378433
2020-10-11 12:35:56Rosuavsetmessages: + msg378431
2020-10-11 12:27:08vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg378428

stage: resolved
2020-10-09 20:14:34eamanusetmessages: + msg378347
2020-10-09 20:06:50eamanusetnosy: + eamanu
2020-10-05 14:29:42Rosuavsetnosy: + Rosuav
messages: + msg378032
2020-10-05 10:56:38vstinnersetmessages: + msg378018
2020-10-05 10:51:01The Compilersetnosy: + The Compiler
messages: + msg378017
2020-10-05 09:31:19serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg378009
2020-10-05 08:48:07vstinnercreate