classification
Title: test_nntplib: sporadic failures, network isses? server down?
Type: behavior Stage: patch review
Components: Tests Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, martin.panter, miss-islington, pitrou, vstinner
Priority: normal Keywords: buildbot, patch

Created on 2013-11-24 18:07 by vstinner, last changed 2018-09-20 20:52 by berker.peksag.

Files
File name Uploaded Description Edit
test_nntplib_logs.txt berker.peksag, 2016-06-17 21:20
nntp-server.patch martin.panter, 2016-06-19 05:31 Custom local server review
original-order.diff martin.panter, 2016-06-20 03:15 Smaller diff representing above changes review
multi-connect.patch martin.panter, 2016-12-24 02:57 Each test creates new connection review
Pull Requests
URL Status Linked Edit
PR 9446 merged berker.peksag, 2018-09-20 14:53
PR 9447 merged miss-islington, 2018-09-20 15:09
PR 9448 merged miss-islington, 2018-09-20 15:09
PR 9450 merged miss-islington, 2018-09-20 15:26
PR 9461 open berker.peksag, 2018-09-20 20:44
Messages (24)
msg204248 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-24 18:07
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/692/steps/test/logs/stdio

======================================================================
ERROR: test_capabilities (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_nntplib.py", line 251, in wrapped
    meth(self)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_nntplib.py", line 201, in test_capabilities
    resp, caps = self.server.capabilities()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 558, in capabilities
    resp, lines = self._longcmdstring("CAPABILITIES")
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 525, in _longcmdstring
    resp, list = self._getlongresp(file)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 476, in _getlongresp
    resp = self._getresp()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 449, in _getresp
    resp = self._getline()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 432, in _getline
    line = self.file.readline(_MAXLINE +1)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/socket.py", line 368, in readinto
    raise OSError("cannot read from timed out object")
OSError: cannot read from timed out object



http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7147/steps/test/logs/stdio


======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_nntplib.py", line 298, in setUpClass
    cls.server = cls.NNTP_CLASS(cls.NNTP_HOST, timeout=TIMEOUT, usenetrc=False)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 1071, in __init__
    readermode=readermode, timeout=timeout)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 334, in __init__
    self.getcapabilities()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 390, in getcapabilities
    resp, caps = self.capabilities()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 558, in capabilities
    resp, lines = self._longcmdstring("CAPABILITIES")
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 525, in _longcmdstring
    resp, list = self._getlongresp(file)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 476, in _getlongresp
    resp = self._getresp()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 449, in _getresp
    resp = self._getline()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 437, in _getline
    if not line: raise EOFError
EOFError
msg204262 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-24 20:01
The third-party server isn't extremely reliable indeed. Ideally, we would run our own NNTP server, but that's a lot more work.
msg223333 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-17 13:58
Are these sporadic failures still happening?  I've glanced over the logs and only found failures due to recent code changes.
msg245587 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-21 10:28
See Issue 1186900 about the NNTP client raising EOFError.

As for the “cannot read from timed out object” errors (also reported in Issue 19613), I think the test classes are bad. The test logs don’t seem to be available, but I reckon that the previous test to the failing test would have been skipped due to the transient Internet handler. The test classes are essentially coded like this:

class NetworkedNNTP_SSLTests(TestCase):
    @classmethod
    def setUpClass(cls):
        with support.transient_internet(cls.NNTP_HOST):
            cls.server = NNTP_SSL(cls.NNTP_HOST, timeout=TIMEOUT)
    
    def test_that_triggered_the_failure(self):
        with support.transient_internet(self.NNTP_HOST):
            ...  # Suppose this timed out and the test was skipped
    
    def test_capabilities(self):
        # Now this is reusing the existing connection, left in a broken state by
        # the previous test
        self.server.capabilities()

Either the server connection should be freshly made for each individual test case (not once per class), or all the test cases grouped by setUpClass() should be skipped if the transient Internet handler kicks in. I’m not sure if it is practical to do the second option with the standard unittest module. Perhaps the method wrapping stage could add some extra checks to see if the test should be skipped.
msg268742 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-06-17 21:20
I just saw the same test failures at http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.5/builds/998/steps/test/logs/stdio

I'm attaching the test logs for future reference.
msg268745 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-17 23:29
Thanks Berker. In this case the previous tests using the same NNTP connection object were skipped. First run:

test_unknown_command (test.test_nntplib.NetworkedNNTPTests) ... ok
test_welcome (test.test_nntplib.NetworkedNNTPTests) ... ok
test_with_statement (test.test_nntplib.NetworkedNNTPTests) ... skipped "Resource 'news.trigofacile.com' is not available"
test_xhdr (test.test_nntplib.NetworkedNNTPTests) ... skipped "Resource 'news.trigofacile.com' is not available"
test_xover (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zlogin (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zzquit (test.test_nntplib.NetworkedNNTPTests) ... ERROR

Subsequent retry:

test_xover (test.test_nntplib.NNTPv2Tests) ... ok
test_article_head_body (test.test_nntplib.NetworkedNNTPTests) ... skipped "Resource 'news.trigofacile.com' is not available"
test_capabilities (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_date (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_description (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_group (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_help (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_list (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_list_active (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_newgroups (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_over (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_unknown_command (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_welcome (test.test_nntplib.NetworkedNNTPTests) ... ok
test_with_statement (test.test_nntplib.NetworkedNNTPTests) ... ok
test_xhdr (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_xover (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zlogin (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zzquit (test.test_nntplib.NetworkedNNTPTests) ... ERROR

Test_with_statement() is a special case because it starts a fresh NNTP connection, rather than reusing self.server. Test_welcome() is also different because it does not send any new commands to the server. But I presume the other tests all try to reuse the old timed-out self.server object. The skipped messages are evidence of this.

The easy solution which I mentioned above would be to change setUpClass() to setUp(), although this might slow the tests down a bit, reconnecting to the remote server for each test method.

The solution that I prefer would be as Antoine suggested, to run our own server. I think expanding the server I created in Issue 25859 with more commands would be good enough.
msg268841 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-19 05:31
Here is a patch that changes all affected tests to use a custom server running on localhost:

* Local server is based on the existing NNTPv2Handler class, and the LocalServerTests class
* Reintegrate test_starttls() back into NetworkedNNTPTestsMixin
* Only test_with_statement() talks to the remote NNTP_HOST now (not affected by this bug)
* Make tests in NetworkedNNTPTestsMixin stricter, because the server responses are predetermined now
* Add mock comp.lang.python support to NNTPv1Handler.handle_GROUP()
* Add support for rejecting passwords in NNTPv1Handler.handle_AUTHINFO()
* New message number with non-UTF-8 message body for ARTICLE, HEAD, BODY, and non-UTF-8 subject for OVER, XOVER
msg268882 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-20 03:15
Here is a diff that shows my changes better. I temporarily moved some classes back to their original positions to reduce the diff size, though it won’t work in practice because Python needs the classes defined before using them.
msg283915 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-12-24 02:57
Multi-connect.patch is a smaller patch that changes setUpClass() → setUp(), so that each test method creates a new NNTP connection. The downside is all the reconnecting slows the test execution from 42 s down to 115 s, which is why I would like to move the testing to a local server.
msg292451 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-04-27 15:10
> ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)

I wrote an obvious fix for this one: issue #30188.
msg297123 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 01:35
Any progress on this issue?

Martin: Can you please convert your latest patch into a Pull Request?
msg325821 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-19 23:16
I didn't see test_nntplib failing recently, so I close the issue as outdated.
msg325857 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 11:40
======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTPTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/test_nntplib.py", line 292, in setUpClass
    usenetrc=False)
  File "/home/travis/build/python/cpython/Lib/nntplib.py", line 1043, in __init__
    self.sock = socket.create_connection((host, port), timeout)
  File "/home/travis/build/python/cpython/Lib/socket.py", line 724, in create_connection
    raise err
  File "/home/travis/build/python/cpython/Lib/socket.py", line 713, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address

https://travis-ci.org/python/cpython/jobs/430974933
msg325861 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 11:54
I think it would be nice if tweaked support.transient_internet() to catch EADDRNOTAVAIL. See https://github.com/python/cpython/commit/e4dcbbd7f4ac18d01c0ec85f64ae98b8281ed403#diff-7fa374913c0ecbdc621db8ea79a1212bR78 for another example.
msg325873 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-20 13:38
> I think it would be nice if tweaked support.transient_internet() to catch EADDRNOTAVAIL.

Sure, that sounds like a reasonable change? Do you want to work on a PR? If not, maybe we should open a new issue, since this issue is tidly linked to test_nntplib, whereas modifying support.transient_internet() is a more generic change.
msg325875 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-20 13:44
> Sure, that sounds like a reasonable change? 

Oh, typo: you should read "Sure, that sounds like a reasonable change." :-)
msg325878 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 14:10
Yeah, I will submit a pull request today. The reason I reopened this issue is that I didn't want to file another issue for a similar problem and wanted to use this as a meta issue for issues with test_nntplib. Also, I'd like to play Martin's patches a bit and see if they still needed.
msg325885 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 15:09
New changeset 8213eaddf3ce8e87564d2949454903a1484748b5 by Berker Peksag in branch 'master':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/8213eaddf3ce8e87564d2949454903a1484748b5
msg325887 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 15:23
New changeset 476177005e8c8d4ece3070c8c378f8b8c068e76f by Berker Peksag (Miss Islington (bot)) in branch '3.7':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/476177005e8c8d4ece3070c8c378f8b8c068e76f
msg325889 - (view) Author: miss-islington (miss-islington) Date: 2018-09-20 15:36
New changeset 1eabe19c57938dd70b66b97239be337294e15cba by Miss Islington (bot) in branch '3.6':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/1eabe19c57938dd70b66b97239be337294e15cba
msg325893 - (view) Author: miss-islington (miss-islington) Date: 2018-09-20 16:06
New changeset 170ea8ccd4235d28538ab713041502d07ad1cacd by Miss Islington (bot) in branch '2.7':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/170ea8ccd4235d28538ab713041502d07ad1cacd
msg325896 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-20 16:11
Thanks Berker! Do you want to do something with Martin's patches? If not, can you please close the issue?
msg325917 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 17:56
Thanks for the reviews, Victor. I'm planning to work on them tonight. I will close this issue and open a new one if I decide to do further work on them.
msg325931 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-09-20 20:52
PR 9461 converts Martin's nntp-server.patch. I've made some tweaks and fixed a "threading._dangling was modified by test_nntplib" warning. I will try to convert multi-connect.patch later.

I will attribute Martin as the patch author before merging the PR.
History
Date User Action Args
2018-09-20 20:52:48berker.peksagsetmessages: + msg325931
2018-09-20 20:44:54berker.peksagsetpull_requests: + pull_request8875
2018-09-20 17:56:03berker.peksagsetmessages: + msg325917
2018-09-20 16:11:56vstinnersetmessages: + msg325896
2018-09-20 16:06:08miss-islingtonsetmessages: + msg325893
2018-09-20 15:36:26miss-islingtonsetnosy: + miss-islington
messages: + msg325889
2018-09-20 15:26:29miss-islingtonsetpull_requests: + pull_request8865
2018-09-20 15:23:52berker.peksagsetmessages: + msg325887
2018-09-20 15:09:32miss-islingtonsetpull_requests: + pull_request8863
2018-09-20 15:09:21miss-islingtonsetpull_requests: + pull_request8862
2018-09-20 15:09:05berker.peksagsetmessages: + msg325885
2018-09-20 14:53:18berker.peksagsetstage: needs patch -> patch review
pull_requests: + pull_request8861
2018-09-20 14:10:06berker.peksagsetmessages: + msg325878
2018-09-20 13:44:06vstinnersetmessages: + msg325875
2018-09-20 13:38:23vstinnersetmessages: + msg325873
2018-09-20 11:54:54berker.peksagsetmessages: + msg325861
2018-09-20 11:41:13berker.peksagsetmessages: - msg325858
2018-09-20 11:40:47berker.peksagsetstatus: open

messages: + msg325858
2018-09-20 11:40:33berker.peksagsetstatus: closed -> (no value)
versions: + Python 3.8, - Python 3.5
messages: + msg325857

resolution: out of date ->
stage: resolved -> needs patch
2018-09-19 23:16:34vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg325821

stage: patch review -> resolved
2017-06-28 01:35:06vstinnersetmessages: + msg297123
2017-04-27 15:10:36vstinnersetmessages: + msg292451
2016-12-24 02:57:27martin.pantersetfiles: + multi-connect.patch

messages: + msg283915
versions: + Python 3.7
2016-06-20 03:15:18martin.pantersetfiles: + original-order.diff

messages: + msg268882
2016-06-20 00:13:00berker.peksagsetstage: needs patch -> patch review
2016-06-19 05:31:25martin.pantersetfiles: + nntp-server.patch
keywords: + patch
messages: + msg268841
2016-06-18 06:29:51BreamoreBoysetnosy: - BreamoreBoy
2016-06-17 23:29:20martin.pantersetmessages: + msg268745
2016-06-17 21:20:35berker.peksagsetfiles: + test_nntplib_logs.txt

type: behavior
versions: + Python 3.5, Python 3.6, - Python 3.3, Python 3.4
nosy: + berker.peksag

messages: + msg268742
stage: needs patch
2015-06-21 10:28:37martin.pantersetnosy: + martin.panter
messages: + msg245587
2014-07-17 13:58:08BreamoreBoysetnosy: + BreamoreBoy
messages: + msg223333
2013-11-24 20:01:10pitrousetnosy: + pitrou

messages: + msg204262
versions: + Python 3.3
2013-11-24 18:07:31vstinnercreate