classification
Title: CI should not depend on gmane response
Type: behavior Stage: needs patch
Components: Documentation, Tests Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, nsiregar, pablogsal, terry.reedy, vstinner, zach.ware
Priority: normal Keywords:

Created on 2019-07-16 15:49 by terry.reedy, last changed 2019-07-24 14:34 by zach.ware.

Messages (6)
msg348023 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-16 15:49
On Travis, 'Documentation tests' passing is apparently required.  Line 23 of nntplib.rst is ">>> s = NNTP('news.gmane.org')"  I use gmane and occasional failures to respond are routine.  Being able to merge should not depend on this external site.

In addition to a real failure, the following happen on PR 14675 earlier today.

https://travis-ci.org/python/cpython/jobs/559289634

Warning, treated as error:
**********************************************************************
File "library/nntplib.rst", line ?, in default
Failed example:
    s = NNTP('news.gmane.org')
Exception raised:
    Traceback (most recent call last):
      File "/home/travis/build/python/cpython/Lib/doctest.py", line 1328, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest default[0]>", line 1, in <module>
        s = NNTP('news.gmane.org')
      File "/home/travis/build/python/cpython/Lib/nntplib.py", line 1050, in __init__
        _NNTPBase.__init__(self, file, host,
      File "/home/travis/build/python/cpython/Lib/nntplib.py", line 331, in __init__
        self.welcome = self._getresp()
      File "/home/travis/build/python/cpython/Lib/nntplib.py", line 456, in _getresp
        raise NNTPTemporaryError(resp)
    nntplib.NNTPTemporaryError: 400 load at 16.59, try later

Try later is what I do.  This exception should somehow be skipped.  I don't know anything about Travis and doctest tests and how flakey doctests are supposed to be handled. If nothing else, put the whole example is a try-except block?

>>> try:  # News sites can fail to respond.
...     s = NNTP('news.gmane.org')
...     ...
... except: # If site failure, try later.
...     pass

Since output varies, I presume output checking is suppressed.
msg348024 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-16 15:53
> On Travis, 'Documentation tests' passing is apparently required.  Line 23 of nntplib.rst is ">>> s = NNTP('news.gmane.org')"  I use gmane and occasional failures to respond are routine.  Being able to merge should not depend on this external site.

Python test suite uses many external services:
https://pythondev.readthedocs.io/infra.html#services-used-by-unit-tests

You should all be skipped if the "network" resource is not used, which is the default.

Many tests were modified to catch network failures and ignore the test in this case. test_nntplib is known to be very fragile: it doesn't handle network issues.

I would suggest to remove functional tests which depend on external servers, since these tests fail often and nobody seems to care of test_nntplib.
msg348025 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2019-07-16 15:57
Victor, note that this issue is about the nntplib doctests in Doc/library/nntplib.rst.  I agree that any examples in that file that actually hit an external server should be marked as not doctests.
msg348249 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-21 15:07
gmane was down awhile yesterday, so another merge (temporarily) blocked because of this.  If either of you knows how to fix this, please do so.
msg348383 - (view) Author: Ngalim Siregar (nsiregar) * Date: 2019-07-24 12:18
is it allowed to mock news.gmane.org response?

if it is okay i would like to help
msg348387 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2019-07-24 14:34
For doctests, I would prefer to just disable some or all of the doctests in Doc/library/nntplib.rst rather than go to the effort of trying to mock things.
History
Date User Action Args
2019-07-24 14:34:35zach.waresetmessages: + msg348387
2019-07-24 12:18:58nsiregarsetnosy: + nsiregar
messages: + msg348383
2019-07-21 15:07:32terry.reedysetmessages: + msg348249
2019-07-16 15:57:51zach.waresetversions: + Python 2.7, Python 3.7, Python 3.8
nosy: + docs@python, zach.ware

messages: + msg348025

assignee: docs@python
components: + Documentation
2019-07-16 15:53:36vstinnersetmessages: + msg348024
2019-07-16 15:49:35terry.reedycreate