http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/692/steps/test/logs/stdio

======================================================================
ERROR: test_capabilities (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_nntplib.py", line 251, in wrapped
    meth(self)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_nntplib.py", line 201, in test_capabilities
    resp, caps = self.server.capabilities()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 558, in capabilities
    resp, lines = self._longcmdstring("CAPABILITIES")
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 525, in _longcmdstring
    resp, list = self._getlongresp(file)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 476, in _getlongresp
    resp = self._getresp()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 449, in _getresp
    resp = self._getline()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/nntplib.py", line 432, in _getline
    line = self.file.readline(_MAXLINE +1)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/socket.py", line 368, in readinto
    raise OSError("cannot read from timed out object")
OSError: cannot read from timed out object



http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7147/steps/test/logs/stdio


======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_nntplib.py", line 298, in setUpClass
    cls.server = cls.NNTP_CLASS(cls.NNTP_HOST, timeout=TIMEOUT, usenetrc=False)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 1071, in __init__
    readermode=readermode, timeout=timeout)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 334, in __init__
    self.getcapabilities()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 390, in getcapabilities
    resp, caps = self.capabilities()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 558, in capabilities
    resp, lines = self._longcmdstring("CAPABILITIES")
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 525, in _longcmdstring
    resp, list = self._getlongresp(file)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 476, in _getlongresp
    resp = self._getresp()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 449, in _getresp
    resp = self._getline()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/nntplib.py", line 437, in _getline
    if not line: raise EOFError
EOFError

The third-party server isn't extremely reliable indeed. Ideally, we would run our own NNTP server, but that's a lot more work.

Are these sporadic failures still happening?  I've glanced over the logs and only found failures due to recent code changes.

See Issue 1186900 about the NNTP client raising EOFError.

As for the “cannot read from timed out object” errors (also reported in Issue 19613), I think the test classes are bad. The test logs don’t seem to be available, but I reckon that the previous test to the failing test would have been skipped due to the transient Internet handler. The test classes are essentially coded like this:

class NetworkedNNTP_SSLTests(TestCase):
    @classmethod
    def setUpClass(cls):
        with support.transient_internet(cls.NNTP_HOST):
            cls.server = NNTP_SSL(cls.NNTP_HOST, timeout=TIMEOUT)
    
    def test_that_triggered_the_failure(self):
        with support.transient_internet(self.NNTP_HOST):
            ...  # Suppose this timed out and the test was skipped
    
    def test_capabilities(self):
        # Now this is reusing the existing connection, left in a broken state by
        # the previous test
        self.server.capabilities()

Either the server connection should be freshly made for each individual test case (not once per class), or all the test cases grouped by setUpClass() should be skipped if the transient Internet handler kicks in. I’m not sure if it is practical to do the second option with the standard unittest module. Perhaps the method wrapping stage could add some extra checks to see if the test should be skipped.

I just saw the same test failures at http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.5/builds/998/steps/test/logs/stdio

I'm attaching the test logs for future reference.

Thanks Berker. In this case the previous tests using the same NNTP connection object were skipped. First run:

test_unknown_command (test.test_nntplib.NetworkedNNTPTests) ... ok
test_welcome (test.test_nntplib.NetworkedNNTPTests) ... ok
test_with_statement (test.test_nntplib.NetworkedNNTPTests) ... skipped "Resource 'news.trigofacile.com' is not available"
test_xhdr (test.test_nntplib.NetworkedNNTPTests) ... skipped "Resource 'news.trigofacile.com' is not available"
test_xover (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zlogin (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zzquit (test.test_nntplib.NetworkedNNTPTests) ... ERROR

Subsequent retry:

test_xover (test.test_nntplib.NNTPv2Tests) ... ok
test_article_head_body (test.test_nntplib.NetworkedNNTPTests) ... skipped "Resource 'news.trigofacile.com' is not available"
test_capabilities (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_date (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_description (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_group (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_help (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_list (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_list_active (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_newgroups (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_over (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_unknown_command (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_welcome (test.test_nntplib.NetworkedNNTPTests) ... ok
test_with_statement (test.test_nntplib.NetworkedNNTPTests) ... ok
test_xhdr (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_xover (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zlogin (test.test_nntplib.NetworkedNNTPTests) ... ERROR
test_zzquit (test.test_nntplib.NetworkedNNTPTests) ... ERROR

Test_with_statement() is a special case because it starts a fresh NNTP connection, rather than reusing self.server. Test_welcome() is also different because it does not send any new commands to the server. But I presume the other tests all try to reuse the old timed-out self.server object. The skipped messages are evidence of this.

The easy solution which I mentioned above would be to change setUpClass() to setUp(), although this might slow the tests down a bit, reconnecting to the remote server for each test method.

The solution that I prefer would be as Antoine suggested, to run our own server. I think expanding the server I created in Issue 25859 with more commands would be good enough.

Here is a patch that changes all affected tests to use a custom server running on localhost:

* Local server is based on the existing NNTPv2Handler class, and the LocalServerTests class
* Reintegrate test_starttls() back into NetworkedNNTPTestsMixin
* Only test_with_statement() talks to the remote NNTP_HOST now (not affected by this bug)
* Make tests in NetworkedNNTPTestsMixin stricter, because the server responses are predetermined now
* Add mock comp.lang.python support to NNTPv1Handler.handle_GROUP()
* Add support for rejecting passwords in NNTPv1Handler.handle_AUTHINFO()
* New message number with non-UTF-8 message body for ARTICLE, HEAD, BODY, and non-UTF-8 subject for OVER, XOVER

Here is a diff that shows my changes better. I temporarily moved some classes back to their original positions to reduce the diff size, though it won’t work in practice because Python needs the classes defined before using them.

Multi-connect.patch is a smaller patch that changes setUpClass() → setUp(), so that each test method creates a new NNTP connection. The downside is all the reconnecting slows the test execution from 42 s down to 115 s, which is why I would like to move the testing to a local server.

> ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)

I wrote an obvious fix for this one: issue #30188.

Any progress on this issue?

Martin: Can you please convert your latest patch into a Pull Request?

I didn't see test_nntplib failing recently, so I close the issue as outdated.

======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTPTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/test_nntplib.py", line 292, in setUpClass
    usenetrc=False)
  File "/home/travis/build/python/cpython/Lib/nntplib.py", line 1043, in __init__
    self.sock = socket.create_connection((host, port), timeout)
  File "/home/travis/build/python/cpython/Lib/socket.py", line 724, in create_connection
    raise err
  File "/home/travis/build/python/cpython/Lib/socket.py", line 713, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address

https://travis-ci.org/python/cpython/jobs/430974933

I think it would be nice if tweaked support.transient_internet() to catch EADDRNOTAVAIL. See https://github.com/python/cpython/commit/e4dcbbd7f4ac18d01c0ec85f64ae98b8281ed403#diff-7fa374913c0ecbdc621db8ea79a1212bR78 for another example.

> I think it would be nice if tweaked support.transient_internet() to catch EADDRNOTAVAIL.

Sure, that sounds like a reasonable change? Do you want to work on a PR? If not, maybe we should open a new issue, since this issue is tidly linked to test_nntplib, whereas modifying support.transient_internet() is a more generic change.

> Sure, that sounds like a reasonable change? 

Oh, typo: you should read "Sure, that sounds like a reasonable change." :-)

Yeah, I will submit a pull request today. The reason I reopened this issue is that I didn't want to file another issue for a similar problem and wanted to use this as a meta issue for issues with test_nntplib. Also, I'd like to play Martin's patches a bit and see if they still needed.

New changeset 8213eaddf3ce8e87564d2949454903a1484748b5 by Berker Peksag in branch 'master':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/8213eaddf3ce8e87564d2949454903a1484748b5

New changeset 476177005e8c8d4ece3070c8c378f8b8c068e76f by Berker Peksag (Miss Islington (bot)) in branch '3.7':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/476177005e8c8d4ece3070c8c378f8b8c068e76f

New changeset 1eabe19c57938dd70b66b97239be337294e15cba by Miss Islington (bot) in branch '3.6':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/1eabe19c57938dd70b66b97239be337294e15cba

New changeset 170ea8ccd4235d28538ab713041502d07ad1cacd by Miss Islington (bot) in branch '2.7':
bpo-19756: Prevent test failures due to EADDRNOTAVAIL (GH-9446)
https://github.com/python/cpython/commit/170ea8ccd4235d28538ab713041502d07ad1cacd

Thanks Berker! Do you want to do something with Martin's patches? If not, can you please close the issue?

Thanks for the reviews, Victor. I'm planning to work on them tonight. I will close this issue and open a new one if I decide to do further work on them.

PR 9461 converts Martin's nntp-server.patch. I've made some tweaks and fixed a "threading._dangling was modified by test_nntplib" warning. I will try to convert multi-connect.patch later.

I will attribute Martin as the patch author before merging the PR.

See also: https://bugs.python.org/issue40735 (test_with_statement)