classification
Title: Regression test test_httpservers fails, hangs on Windows
Type: behavior Stage: needs patch
Components: Library (Lib), Tests, Windows Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Claudiu.Popa, ezio.melotti, fwierzbicki, jeff.allen, orsenthil, python-dev, r.david.murray, terry.reedy
Priority: normal Keywords: patch

Created on 2014-01-07 00:19 by jeff.allen, last changed 2014-06-29 20:06 by jeff.allen.

Files
File name Uploaded Description Edit
issue20155_py.patch jeff.allen, 2014-02-23 20:53 Patch for 2.7 review
Messages (20)
msg207497 - (view) Author: Jeff Allen (jeff.allen) * Date: 2014-01-07 00:19
When I run:
    start python -m test.test_httpservers

test_request_line_trimming reports ERROR, and the test hangs at test_version_none. If I run a copy of the test in which the latter test is skipped with @unittest.skipIf(sys.platform == "win32", "..."), the error report is:

======================================================================
ERROR: test_request_line_trimming (__main__.BaseHTTPServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_httpservers.py", line 122, in test_request_line_trimming
    res = self.con.getresponse()
  File "C:\Python33\lib\http\client.py", line 1131, in getresponse
    response.begin()
  File "C:\Python33\lib\http\client.py", line 354, in begin
    version, status, reason = self._read_status()
  File "C:\Python33\lib\http\client.py", line 316, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Python33\lib\socket.py", line 297, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

This is essentially the same for Python 2.7.6 and for the current development tip of Jython 2.7b1+, which is actually where the problem first manifested.

My machine is running 64-bit Windows 7 SP1, recently re-installed to a new, empty disk.

Careful testing, elaborating the failing tests, shows that what is sent in PUT and GET operations is not quite what is received. Something tampers with the connection between the client and the server. (Identical traffic where the verbs are not PUT and GET arrives as sent.) Something fiddles with the forward message, for example "correcting" the spurious \n in test_request_line_trimming to a full \r\n, and holding back the payload of a PUT even when it was in the first packet. On the reverse path, it appears to act on the error response itself by closing the connection, without passing it to the client.

Disabling the firewall (Windows Firewall and a commercial one), with the network cable unplugged, makes no difference. Nor does stopping anti-virus, anti-phishing, parental controls, etc.. However, stopping the Windows Basic Filtering Engine (BFE), makes the regression test run without error. Stopping the BFE takes out several dependent services, including Windows Firewall, but it seems likely the BFE itself is the culprit.

Although the cause lies in the platform, not in Python, it seems to me still an "issue" for Python that the tests fail on a common platform practically out of the box. I'll work on this in the context of the Jython test and report back here.
msg210926 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-02-11 11:49
Installed 3.4.0c1 on Windows:
test test_httpservers failed -- Traceback (most recent call last):
  File "C:\Programs\Python34.32\lib\test\test_httpservers.py", line 309, in test_invalid_requests
    self.check_status_and_reason(response, 501)
  File "C:\Programs\Python34.32\lib\test\test_httpservers.py", line 264, in check_status_and_reason
    self.assertEqual(response.status, status)
AssertionError: 200 != 501

I know very little about http server stuff.
msg211990 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-02-23 11:28
Terry, I had the same problem with that failing test_httpservers.test_invalid_request using the latest build on Windows. 
After debugging, I found out that the problem was caused by my antivirus solution. Its http scanning engine caught malformed http requests, like the one in the test ("gEt / hTTP/1.0\r\nhost: 127.0.0.1:50340\r\nAccept-Encoding: identity\r\n\r\n"; for instance) and it modified the first line by uppercasing it, thus making the perfect condition for a failing test.
msg211999 - (view) Author: Jeff Allen (jeff.allen) * Date: 2014-02-23 15:25
Thanks for adding to the evidence here. As discussed above, disabling the security product (which is Bitdefender) on my PC didn't stop the problem for me, and I'm reluctant to uninstall. I narrowed it to the Windows Base Filtering Engine, but perhaps the behaviour of the BFE is extended by installing BD.

If so, you could say this is not a Python problem, it is caused by BD "normalising" the HTTP. Or BD could say it is caused by expecting a defined result from abnormal HTTP.

I took the view it were best fixed at our end. I found I could test the same thing (AFAICT), but modify the tests so they don't get interfered with.
http://bugs.jython.org/issue2109
http://hg.python.org/jython/rev/6441fcfd940b

Would a patch made from this be applicable to CPython?
msg212004 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-02-23 16:13
Except where specifically indicated otherwise, the Python test suite should be the same, and correct, for all implementations. If that change is correct for Jython, it should be correct for CPython.

David, Ezio, or Senthil: does the change in the Jython patch look correct to any of you, enough that we should apply it?
msg212026 - (view) Author: Jeff Allen (jeff.allen) * Date: 2014-02-23 20:53
Actual patch for your convenience. I'm not set up to build CPython from source, so I've tested this with my installed CPython 2.7.6, and it's clean.

[As for keeping the tests in sync, yes that's our aim. Jython's Lib contains only the customised versions, and everything else comes from a copy of CPython's in lib-python/2.7. I'm always looking for a chance to delete one (i.e. use the common file).]
msg212652 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-03 18:49
The second test looks correct to me.  I don't understand what tset_request_line_trimming is testing (I haven't investigated), so I don't know if that one is correct, but it is certainly plausible.
msg212754 - (view) Author: Jeff Allen (jeff.allen) * Date: 2014-03-04 23:51
I worked out that the essence of the test is to insert an extra \n at the end of a GET request line. The request is syntactically invalid for HTTP. The \n\r\n appears like two blank lines, implying no headers, but the headers then follow where no data should be. The server is supposed to respond with status 501, because it does not, in fact, define a GET operation.

To find the replacement test plausible you have to accept that, with a server that doesn't define GET, the verb may as well be XYZBOGUS. Since the security filter doesn't understand that verb either (unlike GET), it doesn't interfere in the test.
msg212755 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-05 01:23
Ah, OK.  I was thinking that BaseHTTPRequestHandler defined GET, but I see that it doesn't.

So yes, this patch looks fine to me.
msg221458 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-06-24 13:38
It would be nice if this could be committed. It's cumbersome to always have to deactivate the AV solution when running the tests on Windows, in order to avoid a failure of test_httpservers.
msg221498 - (view) Author: Roundup Robot (python-dev) Date: 2014-06-24 20:51
New changeset ffdd2d0b0049 by R David Murray in branch '3.4':
#20155: use fake HTTP method names so windows doesn't hang the tests.
http://hg.python.org/cpython/rev/ffdd2d0b0049

New changeset e67ad57eed26 by R David Murray in branch 'default':
merge: #20155: use fake HTTP method names so windows doesn't hang the tests.
http://hg.python.org/cpython/rev/e67ad57eed26

New changeset b0526da56c54 by R David Murray in branch '2.7':
#20155: use fake HTTP method names so windows doesn't hang the tests.
http://hg.python.org/cpython/rev/b0526da56c54
msg221499 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-06-24 20:52
Done.  Thanks Jeff.
msg221803 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-06-28 16:45
After updating and rebuilding (32 bit VC express, on 64 bit Win 7), I get the same error as in msg210926, with or without -uall, even after turning my antivirus off.

  File "F:\Python\dev\4\py34\lib\test\test_httpservers.py", line 310, in test_invalid_requests
    self.check_status_and_reason(response, 501)
  File "F:\Python\dev\4\py34\lib\test\test_httpservers.py", line 265, in check_status_and_reason
    self.assertEqual(response.status, status)
AssertionError: 200 != 501

Is this a failure of the patch or a different issue?
msg221858 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-06-29 16:12
If deactivating the AV/firewall doesn't change the behavior, it is presumably a different problem.  But let's see what Claudiu has to say, since he was able to reproduce and then circumvent the original problem.
msg221859 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-06-29 16:28
Terry is right, this patch doesn't completely work. But with this fix the problem is solved:

diff -r 394e6bda5a70 Lib/test/test_httpservers.py
--- a/Lib/test/test_httpservers.py      Sun Jun 29 15:56:21 2014 +0300
+++ b/Lib/test/test_httpservers.py      Sun Jun 29 19:27:16 2014 +0300
@@ -306,7 +306,7 @@
         response = self.request('/', method='FOO')
         self.check_status_and_reason(response, 501)
         # requests must be case sensitive,so this should fail too
-        response = self.request('/', method='get')
+        response = self.request('/', method='gets')
         self.check_status_and_reason(response, 501)
         response = self.request('/', method='GETs')
         self.check_status_and_reason(response, 501)
msg221860 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-06-29 16:29
Although it is incorrect, because the test specifically tests case sensitivity.
msg221862 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-06-29 16:37
What if you use 'custom' instead?
msg221863 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-06-29 16:52
It works with 'custom'.
msg221866 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-06-29 17:16
Both 'gets' and 'custom' work on my machine.
msg221893 - (view) Author: Jeff Allen (jeff.allen) * Date: 2014-06-29 20:06
Disabling the AV/firewall did not stop the symptoms when I was investigating originally. In order to get the unmodified test to pass, I had to stop the BFE (base filtering engine), which I think may have been given new rules or behaviours as a result of installing the AV solution ... or maybe it was a Windows upgrade that did it.

I did wonder if this might be a moving target, as the test deliberately includes server abuse, while the products want to stop that.

If I try test_httpservers.py as amended (http://hg.python.org/cpython/file/ffdd2d0b0049/Lib/test/test_httpservers.py) on my machine with CPython 3.4.1, I do not get the error Terry reports. (test_urlquote_decoding_in_cgi_check fails but it should.)
History
Date User Action Args
2014-06-29 20:06:38jeff.allensetmessages: + msg221893
2014-06-29 17:16:23terry.reedysetmessages: + msg221866
2014-06-29 16:52:07Claudiu.Popasetmessages: + msg221863
2014-06-29 16:37:45r.david.murraysetstatus: closed -> open
resolution: fixed ->
messages: + msg221862

stage: resolved -> needs patch
2014-06-29 16:29:51Claudiu.Popasetmessages: + msg221860
2014-06-29 16:28:50Claudiu.Popasetmessages: + msg221859
2014-06-29 16:12:07r.david.murraysetmessages: + msg221858
2014-06-28 16:45:50terry.reedysetmessages: + msg221803
2014-06-24 20:52:10r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg221499

stage: patch review -> resolved
2014-06-24 20:51:26python-devsetnosy: + python-dev
messages: + msg221498
2014-06-24 13:38:40Claudiu.Popasetmessages: + msg221458
2014-03-05 01:23:19r.david.murraysetmessages: + msg212755
2014-03-04 23:51:29jeff.allensetmessages: + msg212754
2014-03-03 18:49:10r.david.murraysetmessages: + msg212652
2014-02-23 20:53:49jeff.allensetfiles: + issue20155_py.patch

nosy: + fwierzbicki
messages: + msg212026

keywords: + patch
2014-02-23 16:13:29terry.reedysetnosy: + orsenthil, ezio.melotti, r.david.murray
messages: + msg212004

components: + Tests
type: behavior
stage: patch review
2014-02-23 15:25:35jeff.allensetmessages: + msg211999
2014-02-23 11:28:38Claudiu.Popasetnosy: + Claudiu.Popa
messages: + msg211990
2014-02-11 11:49:44terry.reedysetnosy: + terry.reedy

messages: + msg210926
versions: + Python 3.4
2014-01-07 00:19:11jeff.allencreate