This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_nntplib fails on CI
Type: Stage: resolved
Components: Tests Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, corona10, lemburg, miss-islington, ned.deily, pitrou, serhiy.storchaka, trrhodes, vstinner
Priority: critical Keywords: patch

Created on 2020-12-31 09:36 by serhiy.storchaka, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
openssl.cnf serhiy.storchaka, 2021-01-01 10:31
Pull Requests
URL Status Linked Edit
PR 24037 merged corona10, 2021-01-01 08:54
PR 24039 merged miss-islington, 2021-01-01 14:20
PR 24040 merged miss-islington, 2021-01-01 14:20
PR 24041 merged miss-islington, 2021-01-01 14:21
PR 24042 merged miss-islington, 2021-01-01 14:21
PR 22632 serhiy.storchaka, 2021-01-05 15:35
Messages (21)
msg384106 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-12-31 09:36
Since yesterday ALL PRs are blocked by failing test_nntplib.

For example https://github.com/python/cpython/runs/1629664606?check_suite_focus=true:

======================================================================
ERROR: test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 99, in test_descriptions
    desc = descs[self.GROUP_NAME]
KeyError: 'comp.lang.python'

======================================================================
FAIL: test_description (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 85, in test_description
    self.assertIn("Python", desc)
AssertionError: 'Python' not found in ''

----------------------------------------------------------------------
msg384111 - (view) Author: Ross Rhodes (trrhodes) * Date: 2020-12-31 12:30
Server is returning '215 Newsgroup descriptions in form "group description"', but an empty list of lines, so it's reaching the 'nothing' case in _getdescriptions: https://github.com/python/cpython/blob/master/Lib/nntplib.py#L660
msg384112 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-31 12:33
Right now, it works for me:

$ ./python -m test -u all test_nntplib  -v -m test_descriptions
...
test_descriptions (test.test_nntplib.NNTPv1Tests) ... ok
test_descriptions (test.test_nntplib.NNTPv2Tests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ok
skipped "<class 'test.test_nntplib.NetworkedNNTP_SSLTests'> got [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1122) connecting to 'nntp.aioe.org'"
...
Tests result: SUCCESS
msg384114 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-12-31 12:57
On my computer tests are passed. They are only failed on CI.
msg384122 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2020-12-31 14:45
FWIW, I'm getting the same errors in PR https://github.com/python/cpython/pull/23140

Checking on the server that's being used, the newsgroup description is empty indeed: https://news.aioe.org/index.php?id=statistics-about-groups&group=comp.lang.python
msg384155 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-01-01 08:46
== CPython 3.10.0a3+ (heads/master:c8a7b8fa1b, Jan 1 2021, 08:28:41) [GCC 7.5.0]
== Linux-5.4.0-1031-azure-x86_64-with-glibc2.27 little-endian
== cwd: /home/corona10/cpython/build/test_python_66899æ
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 0.00 Run tests sequentially
0:00:00 load avg: 0.00 [1/1] test_nntplib
test_descriptions (test.test_nntplib.NNTPv1Tests) ... ok
test_descriptions (test.test_nntplib.NNTPv2Tests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests) ... ERROR

======================================================================
ERROR: test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/corona10/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/corona10/cpython/Lib/test/test_nntplib.py", line 99, in test_descriptions
    desc = descs[self.GROUP_NAME]
KeyError: 'comp.lang.python'

----------------------------------------------------------------------

Ran 4 tests in 4.734s

FAILED (errors=1)
test test_nntplib failed
test_nntplib failed

== Tests result: FAILURE ==

1 test failed:
    test_nntplib

Total duration: 4.8 sec
Tests result: FAILURE


My test machine success to reproduce this issue.

(Pdb) descs
{}

There might be 4 ways to solve this issue.

1. Re-create group and skip test while descs is empty
2. Update test to use other NNTP server which supports SSL
msg384156 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-01-01 08:46
4 ways / 2 ways
msg384157 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-01-01 08:56
I submit the patch to use group_name which is mentioned from https://news.aioe.org/manual/aioe-hierarchy/

IMHO, it will be not changed frequently than what we used before.
msg384159 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-01-01 10:31
The test was passed on my computer because test.test_nntplib.NetworkedNNTP_SSLTests was skipped because of the configuration of SSL on Ubuntu 20.04 (see issue41561).

After using custom openssl.cnf I get the same errors.

$ OPENSSL_CONF=~/py/openssl.cnf ./python -m test -vuall test_nntplib -m 'test_description*'

The NNTP server nntp.aioe.org is used instead of snews.gmane.org since 2010 (45ca9874955b1eaf3d941b7a278b594fad50d4ee). Sadly there is no reference to any discussion, maybe Antoine remember some details. What other alternatives were considered? How to handle similar breakage of external servers?
msg384161 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-01-01 11:31
> How to handle similar breakage of external servers?

For tests with the external environment,
How about skipping tests if servers are not available?
It is assumed that if there is a change related to NNTP, it is well checked in the CI and review.
This is because, at this point, reviewers and patch submitters check whether tests with external servers are passed.

The reason for testing with the outside is to confirm that the whole operation works well.
If external servers are not available, we can emit warning messages so that we can recognize when an issue occurs.

Other than that, I can think of local server testing or mocking testing, but I am not sure if it is easy way for the SSL environment.
msg384163 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-01-01 12:07
> For tests with the external environment,
How about skipping tests if servers are not available?

Hmm, I change my mind this is not a good idea :(
msg384164 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-01 12:57
Does this issue mean that I should include nntplib in PEP 594 again?
msg384169 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-01-01 13:58
On 01.01.2021 13:57, Christian Heimes wrote:
> 
> Does this issue mean that I should include nntplib in PEP 594 again?

The test fails because it was relying on an external news server's
configuratoin. This doesn't mean the code itself is broken, just the
test, so clearly: no.
msg384171 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-01-01 14:20
New changeset ec3165320e81ac87edcb85c86c452528ddbaec1c by Dong-hee Na in branch 'master':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037)
https://github.com/python/cpython/commit/ec3165320e81ac87edcb85c86c452528ddbaec1c
msg384176 - (view) Author: miss-islington (miss-islington) Date: 2021-01-01 14:40
New changeset 381f3e4bfd4b1c440f7cb3025972fe0acd0406fc by Miss Islington (bot) in branch '3.8':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037)
https://github.com/python/cpython/commit/381f3e4bfd4b1c440f7cb3025972fe0acd0406fc
msg384177 - (view) Author: miss-islington (miss-islington) Date: 2021-01-01 14:42
New changeset b20d5e5ce95248e0fa77c5d7bf8f6f5b1231fa53 by Miss Islington (bot) in branch '3.9':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037)
https://github.com/python/cpython/commit/b20d5e5ce95248e0fa77c5d7bf8f6f5b1231fa53
msg384184 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-01 15:20
> The test fails because it was relying on an external news server's
> configuratoin. This doesn't mean the code itself is broken, just the
> test, so clearly: no.

Broken and unstable tests are one of two reasons why I listed nntplib in my PEP, https://www.python.org/dev/peps/pep-0594/#nntplib .

Q.E.D.
msg384185 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-01-01 16:43
Thanks for the patch, Dong-hee Na.

Christian: Tests relying on external resources will always have such issues. This doesn't mean that the code which is being tested is outdated or broken. It's just an issue with the tests. Perhaps we ought to disable external tests for NNTP on CI or ask the PSF to setup a news server for testing purposes (just like we have for SSL tests), e.g. news.pythontest.net mirroring comp.lang.python.
msg384197 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-01-01 18:37
New changeset 8200ee66697601a8766f234d6eb8e4c8735216fd by Miss Islington (bot) in branch '3.7':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) (GH-24041)
https://github.com/python/cpython/commit/8200ee66697601a8766f234d6eb8e4c8735216fd
msg384198 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-01-01 18:42
New changeset 546baba63a446e261d0248338f9034e56eccfc46 by Miss Islington (bot) in branch '3.6':
bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) (GH-24042)
https://github.com/python/cpython/commit/546baba63a446e261d0248338f9034e56eccfc46
msg384202 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-01 19:25
Thanks Dong-hee Na for the fix!

We have many tests using external services, like test_ssl that Christian may know ;-)
https://pythondev.readthedocs.io/infra.html#services-used-by-unit-tests
History
Date User Action Args
2022-04-11 14:59:39adminsetgithub: 86960
2021-01-05 15:35:44serhiy.storchakasetpull_requests: + pull_request22950
2021-01-01 19:25:55vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg384202

stage: resolved
2021-01-01 18:42:19ned.deilysetmessages: + msg384198
2021-01-01 18:37:42ned.deilysetnosy: + ned.deily
messages: + msg384197
2021-01-01 16:43:04lemburgsetmessages: + msg384185
stage: patch review -> (no value)
2021-01-01 15:20:49christian.heimessetmessages: + msg384184
2021-01-01 14:42:46miss-islingtonsetmessages: + msg384177
2021-01-01 14:40:21miss-islingtonsetmessages: + msg384176
2021-01-01 14:21:09miss-islingtonsetpull_requests: + pull_request22883
2021-01-01 14:21:01miss-islingtonsetpull_requests: + pull_request22882
2021-01-01 14:20:55miss-islingtonsetpull_requests: + pull_request22881
2021-01-01 14:20:47miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request22880
stage: patch review
2021-01-01 14:20:45corona10setmessages: + msg384171
2021-01-01 13:58:19lemburgsetmessages: + msg384169
2021-01-01 12:57:44christian.heimessetnosy: + christian.heimes
messages: + msg384164
2021-01-01 12:07:48corona10setmessages: + msg384163
2021-01-01 11:31:46corona10setmessages: + msg384161
2021-01-01 10:31:02serhiy.storchakasetfiles: + openssl.cnf
nosy: + pitrou
messages: + msg384159

2021-01-01 08:56:43corona10setmessages: + msg384157
stage: patch review -> (no value)
2021-01-01 08:54:30corona10setkeywords: + patch
stage: patch review
pull_requests: + pull_request22877
2021-01-01 08:46:20corona10setmessages: + msg384156
2021-01-01 08:46:06corona10setmessages: + msg384155
2021-01-01 08:36:12corona10setnosy: + corona10
2020-12-31 14:45:49lemburgsetnosy: + lemburg
messages: + msg384122
2020-12-31 12:57:49serhiy.storchakasetmessages: + msg384114
2020-12-31 12:33:40vstinnersetnosy: + vstinner
messages: + msg384112
2020-12-31 12:30:06trrhodessetnosy: + trrhodes
messages: + msg384111
2020-12-31 09:36:38serhiy.storchakacreate