test_nntplib fails on CI #86960

serhiy-storchaka · 2020-12-31T09:36:39Z

BPO	42794
Nosy	@malemburg, @pitrou, @vstinner, @tiran, @ned-deily, @serhiy-storchaka, @corona10, @miss-islington, @rrhodes
PRs	bpo-42794: Update test_nntplib to use offical group name for testing #24037 [3.9] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24039 [3.8] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24040 [3.7] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24041 [3.6] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24042 bpo-41994: Fix refcount issues in Python/import.c #22632
Files	openssl.cnf

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-01.19:25:55.606>
created_at = <Date 2020-12-31.09:36:38.503>
labels = ['3.8', 'tests', '3.9', '3.10']
title = 'test_nntplib fails on CI'
updated_at = <Date 2021-01-05.15:35:44.649>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-01-05.15:35:44.649>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-01.19:25:55.606>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-12-31.09:36:38.503>
creator = 'serhiy.storchaka'
dependencies = []
files = ['49714']
hgrepos = []
issue_num = 42794
keywords = ['patch']
message_count = 21.0
messages = ['384106', '384111', '384112', '384114', '384122', '384155', '384156', '384157', '384159', '384161', '384163', '384164', '384169', '384171', '384176', '384177', '384184', '384185', '384197', '384198', '384202']
nosy_count = 9.0
nosy_names = ['lemburg', 'pitrou', 'vstinner', 'christian.heimes', 'ned.deily', 'serhiy.storchaka', 'corona10', 'miss-islington', 'trrhodes']
pr_nums = ['24037', '24039', '24040', '24041', '24042', '22632']
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42794'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

serhiy-storchaka · 2020-12-31T09:36:38Z

Since yesterday ALL PRs are blocked by failing test_nntplib.

For example https://github.com/python/cpython/runs/1629664606?check_suite_focus=true:

======================================================================
ERROR: test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 99, in test_descriptions
    desc = descs[self.GROUP_NAME]
KeyError: 'comp.lang.python'

======================================================================
FAIL: test_description (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 85, in test_description
    self.assertIn("Python", desc)
AssertionError: 'Python' not found in ''

rrhodes · 2020-12-31T12:30:06Z

Server is returning '215 Newsgroup descriptions in form "group description"', but an empty list of lines, so it's reaching the 'nothing' case in _getdescriptions: https://github.com/python/cpython/blob/master/Lib/nntplib.py#L660

vstinner · 2020-12-31T12:33:41Z

Right now, it works for me:

$ ./python -m test -u all test_nntplib  -v -m test_descriptions
...
test_descriptions (test.test_nntplib.NNTPv1Tests) ... ok
test_descriptions (test.test_nntplib.NNTPv2Tests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ok
skipped "<class 'test.test_nntplib.NetworkedNNTP_SSLTests'> got [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1122) connecting to 'nntp.aioe.org'"
...
Tests result: SUCCESS

serhiy-storchaka · 2020-12-31T12:57:50Z

On my computer tests are passed. They are only failed on CI.

malemburg · 2020-12-31T14:45:50Z

FWIW, I'm getting the same errors in PR #23140

Checking on the server that's being used, the newsgroup description is empty indeed: https://news.aioe.org/index.php?id=statistics-about-groups&group=comp.lang.python

corona10 · 2021-01-01T08:46:06Z

== CPython 3.10.0a3+ (heads/master:c8a7b8fa1b, Jan 1 2021, 08:28:41) [GCC 7.5.0]
== Linux-5.4.0-1031-azure-x86_64-with-glibc2.27 little-endian
== cwd: /home/corona10/cpython/build/test_python_66899æ
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 0.00 Run tests sequentially
0:00:00 load avg: 0.00 [1/1] test_nntplib
test_descriptions (test.test_nntplib.NNTPv1Tests) ... ok
test_descriptions (test.test_nntplib.NNTPv2Tests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests) ... ERROR

======================================================================
ERROR: test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/corona10/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/corona10/cpython/Lib/test/test_nntplib.py", line 99, in test_descriptions
    desc = descs[self.GROUP_NAME]
KeyError: 'comp.lang.python'

Ran 4 tests in 4.734s

FAILED (errors=1)
test test_nntplib failed
test_nntplib failed

== Tests result: FAILURE ==

1 test failed:
test_nntplib

Total duration: 4.8 sec
Tests result: FAILURE

My test machine success to reproduce this issue.

(Pdb) descs
{}

There might be 4 ways to solve this issue.

Re-create group and skip test while descs is empty
Update test to use other NNTP server which supports SSL

corona10 · 2021-01-01T08:46:21Z

4 ways / 2 ways

corona10 · 2021-01-01T08:56:43Z

I submit the patch to use group_name which is mentioned from https://news.aioe.org/manual/aioe-hierarchy/

IMHO, it will be not changed frequently than what we used before.

serhiy-storchaka · 2021-01-01T10:31:02Z

The test was passed on my computer because test.test_nntplib.NetworkedNNTP_SSLTests was skipped because of the configuration of SSL on Ubuntu 20.04 (see bpo-41561).

After using custom openssl.cnf I get the same errors.

$ OPENSSL_CONF=~/py/openssl.cnf ./python -m test -vuall test_nntplib -m 'test_description*'

The NNTP server nntp.aioe.org is used instead of snews.gmane.org since 2010 (45ca987). Sadly there is no reference to any discussion, maybe Antoine remember some details. What other alternatives were considered? How to handle similar breakage of external servers?

corona10 · 2021-01-01T11:31:46Z

How to handle similar breakage of external servers?

For tests with the external environment,
How about skipping tests if servers are not available?
It is assumed that if there is a change related to NNTP, it is well checked in the CI and review.
This is because, at this point, reviewers and patch submitters check whether tests with external servers are passed.

The reason for testing with the outside is to confirm that the whole operation works well.
If external servers are not available, we can emit warning messages so that we can recognize when an issue occurs.

Other than that, I can think of local server testing or mocking testing, but I am not sure if it is easy way for the SSL environment.

corona10 · 2021-01-01T12:07:48Z

For tests with the external environment,
How about skipping tests if servers are not available?

Hmm, I change my mind this is not a good idea :(

tiran · 2021-01-01T12:57:44Z

Does this issue mean that I should include nntplib in PEP-594 again?

malemburg · 2021-01-01T13:58:19Z

On 01.01.2021 13:57, Christian Heimes wrote:

Does this issue mean that I should include nntplib in PEP-594 again?

The test fails because it was relying on an external news server's
configuratoin. This doesn't mean the code itself is broken, just the
test, so clearly: no.

corona10 · 2021-01-01T14:20:45Z

New changeset ec31653 by Dong-hee Na in branch 'master':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037)
ec31653

miss-islington · 2021-01-01T14:40:22Z

New changeset 381f3e4 by Miss Islington (bot) in branch '3.8':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037)
381f3e4

miss-islington · 2021-01-01T14:42:46Z

New changeset b20d5e5 by Miss Islington (bot) in branch '3.9':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037)
b20d5e5

tiran · 2021-01-01T15:20:50Z

The test fails because it was relying on an external news server's
configuratoin. This doesn't mean the code itself is broken, just the
test, so clearly: no.

Broken and unstable tests are one of two reasons why I listed nntplib in my PEP, https://www.python.org/dev/peps/pep-0594/#nntplib .

Q.E.D.

malemburg · 2021-01-01T16:43:05Z

Thanks for the patch, Dong-hee Na.

Christian: Tests relying on external resources will always have such issues. This doesn't mean that the code which is being tested is outdated or broken. It's just an issue with the tests. Perhaps we ought to disable external tests for NNTP on CI or ask the PSF to setup a news server for testing purposes (just like we have for SSL tests), e.g. news.pythontest.net mirroring comp.lang.python.

ned-deily · 2021-01-01T18:37:42Z

New changeset 8200ee6 by Miss Islington (bot) in branch '3.7':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) (GH-24041)
8200ee6

ned-deily · 2021-01-01T18:42:19Z

New changeset 546baba by Miss Islington (bot) in branch '3.6':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) (GH-24042)
546baba

vstinner · 2021-01-01T19:25:56Z

Thanks Dong-hee Na for the fix!

We have many tests using external services, like test_ssl that Christian may know ;-)
https://pythondev.readthedocs.io/infra.html#services-used-by-unit-tests

serhiy-storchaka added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes tests Tests in the Lib/test dir labels Dec 31, 2020

vstinner closed this as completed Jan 1, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_nntplib fails on CI #86960

test_nntplib fails on CI #86960

serhiy-storchaka commented Dec 31, 2020

serhiy-storchaka commented Dec 31, 2020

rrhodes mannequin commented Dec 31, 2020

vstinner commented Dec 31, 2020

serhiy-storchaka commented Dec 31, 2020

malemburg commented Dec 31, 2020

corona10 commented Jan 1, 2021

corona10 commented Jan 1, 2021

corona10 commented Jan 1, 2021

serhiy-storchaka commented Jan 1, 2021

corona10 commented Jan 1, 2021

corona10 commented Jan 1, 2021

tiran commented Jan 1, 2021

malemburg commented Jan 1, 2021

corona10 commented Jan 1, 2021

miss-islington commented Jan 1, 2021

miss-islington commented Jan 1, 2021

tiran commented Jan 1, 2021

malemburg commented Jan 1, 2021

ned-deily commented Jan 1, 2021

ned-deily commented Jan 1, 2021

vstinner commented Jan 1, 2021

test_nntplib fails on CI #86960

test_nntplib fails on CI #86960

Comments

serhiy-storchaka commented Dec 31, 2020

serhiy-storchaka commented Dec 31, 2020

rrhodes mannequin commented Dec 31, 2020

vstinner commented Dec 31, 2020

serhiy-storchaka commented Dec 31, 2020

malemburg commented Dec 31, 2020

corona10 commented Jan 1, 2021

corona10 commented Jan 1, 2021

corona10 commented Jan 1, 2021

serhiy-storchaka commented Jan 1, 2021

corona10 commented Jan 1, 2021

corona10 commented Jan 1, 2021

tiran commented Jan 1, 2021

malemburg commented Jan 1, 2021

corona10 commented Jan 1, 2021

miss-islington commented Jan 1, 2021

miss-islington commented Jan 1, 2021

tiran commented Jan 1, 2021

malemburg commented Jan 1, 2021

ned-deily commented Jan 1, 2021

ned-deily commented Jan 1, 2021

vstinner commented Jan 1, 2021