Created on 2014-01-07 00:19 by jeff.allen, last changed 2014-06-29 20:06 by jeff.allen.
|issue20155_py.patch||jeff.allen, 2014-02-23 20:53||Patch for 2.7||review|
|msg207497 - (view)||Author: Jeff Allen (jeff.allen) *||Date: 2014-01-07 00:19|
When I run: start python -m test.test_httpservers test_request_line_trimming reports ERROR, and the test hangs at test_version_none. If I run a copy of the test in which the latter test is skipped with @unittest.skipIf(sys.platform == "win32", "..."), the error report is: ====================================================================== ERROR: test_request_line_trimming (__main__.BaseHTTPServerTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_httpservers.py", line 122, in test_request_line_trimming res = self.con.getresponse() File "C:\Python33\lib\http\client.py", line 1131, in getresponse response.begin() File "C:\Python33\lib\http\client.py", line 354, in begin version, status, reason = self._read_status() File "C:\Python33\lib\http\client.py", line 316, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "C:\Python33\lib\socket.py", line 297, in readinto return self._sock.recv_into(b) ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host This is essentially the same for Python 2.7.6 and for the current development tip of Jython 2.7b1+, which is actually where the problem first manifested. My machine is running 64-bit Windows 7 SP1, recently re-installed to a new, empty disk. Careful testing, elaborating the failing tests, shows that what is sent in PUT and GET operations is not quite what is received. Something tampers with the connection between the client and the server. (Identical traffic where the verbs are not PUT and GET arrives as sent.) Something fiddles with the forward message, for example "correcting" the spurious \n in test_request_line_trimming to a full \r\n, and holding back the payload of a PUT even when it was in the first packet. On the reverse path, it appears to act on the error response itself by closing the connection, without passing it to the client. Disabling the firewall (Windows Firewall and a commercial one), with the network cable unplugged, makes no difference. Nor does stopping anti-virus, anti-phishing, parental controls, etc.. However, stopping the Windows Basic Filtering Engine (BFE), makes the regression test run without error. Stopping the BFE takes out several dependent services, including Windows Firewall, but it seems likely the BFE itself is the culprit. Although the cause lies in the platform, not in Python, it seems to me still an "issue" for Python that the tests fail on a common platform practically out of the box. I'll work on this in the context of the Jython test and report back here.
|msg210926 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2014-02-11 11:49|
Installed 3.4.0c1 on Windows: test test_httpservers failed -- Traceback (most recent call last): File "C:\Programs\Python34.32\lib\test\test_httpservers.py", line 309, in test_invalid_requests self.check_status_and_reason(response, 501) File "C:\Programs\Python34.32\lib\test\test_httpservers.py", line 264, in check_status_and_reason self.assertEqual(response.status, status) AssertionError: 200 != 501 I know very little about http server stuff.
|msg211990 - (view)||Author: Claudiu Popa (Claudiu.Popa) *||Date: 2014-02-23 11:28|
Terry, I had the same problem with that failing test_httpservers.test_invalid_request using the latest build on Windows. After debugging, I found out that the problem was caused by my antivirus solution. Its http scanning engine caught malformed http requests, like the one in the test ("gEt / hTTP/1.0\r\nhost: 127.0.0.1:50340\r\nAccept-Encoding: identity\r\n\r\n"; for instance) and it modified the first line by uppercasing it, thus making the perfect condition for a failing test.
|msg211999 - (view)||Author: Jeff Allen (jeff.allen) *||Date: 2014-02-23 15:25|
Thanks for adding to the evidence here. As discussed above, disabling the security product (which is Bitdefender) on my PC didn't stop the problem for me, and I'm reluctant to uninstall. I narrowed it to the Windows Base Filtering Engine, but perhaps the behaviour of the BFE is extended by installing BD. If so, you could say this is not a Python problem, it is caused by BD "normalising" the HTTP. Or BD could say it is caused by expecting a defined result from abnormal HTTP. I took the view it were best fixed at our end. I found I could test the same thing (AFAICT), but modify the tests so they don't get interfered with. http://bugs.jython.org/issue2109 http://hg.python.org/jython/rev/6441fcfd940b Would a patch made from this be applicable to CPython?
|msg212004 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2014-02-23 16:13|
Except where specifically indicated otherwise, the Python test suite should be the same, and correct, for all implementations. If that change is correct for Jython, it should be correct for CPython. David, Ezio, or Senthil: does the change in the Jython patch look correct to any of you, enough that we should apply it?
|msg212026 - (view)||Author: Jeff Allen (jeff.allen) *||Date: 2014-02-23 20:53|
Actual patch for your convenience. I'm not set up to build CPython from source, so I've tested this with my installed CPython 2.7.6, and it's clean. [As for keeping the tests in sync, yes that's our aim. Jython's Lib contains only the customised versions, and everything else comes from a copy of CPython's in lib-python/2.7. I'm always looking for a chance to delete one (i.e. use the common file).]
|msg212652 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2014-03-03 18:49|
The second test looks correct to me. I don't understand what tset_request_line_trimming is testing (I haven't investigated), so I don't know if that one is correct, but it is certainly plausible.
|msg212754 - (view)||Author: Jeff Allen (jeff.allen) *||Date: 2014-03-04 23:51|
I worked out that the essence of the test is to insert an extra \n at the end of a GET request line. The request is syntactically invalid for HTTP. The \n\r\n appears like two blank lines, implying no headers, but the headers then follow where no data should be. The server is supposed to respond with status 501, because it does not, in fact, define a GET operation. To find the replacement test plausible you have to accept that, with a server that doesn't define GET, the verb may as well be XYZBOGUS. Since the security filter doesn't understand that verb either (unlike GET), it doesn't interfere in the test.
|msg212755 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2014-03-05 01:23|
Ah, OK. I was thinking that BaseHTTPRequestHandler defined GET, but I see that it doesn't. So yes, this patch looks fine to me.
|msg221458 - (view)||Author: Claudiu Popa (Claudiu.Popa) *||Date: 2014-06-24 13:38|
It would be nice if this could be committed. It's cumbersome to always have to deactivate the AV solution when running the tests on Windows, in order to avoid a failure of test_httpservers.
|msg221498 - (view)||Author: Roundup Robot (python-dev)||Date: 2014-06-24 20:51|
New changeset ffdd2d0b0049 by R David Murray in branch '3.4': #20155: use fake HTTP method names so windows doesn't hang the tests. http://hg.python.org/cpython/rev/ffdd2d0b0049 New changeset e67ad57eed26 by R David Murray in branch 'default': merge: #20155: use fake HTTP method names so windows doesn't hang the tests. http://hg.python.org/cpython/rev/e67ad57eed26 New changeset b0526da56c54 by R David Murray in branch '2.7': #20155: use fake HTTP method names so windows doesn't hang the tests. http://hg.python.org/cpython/rev/b0526da56c54
|msg221499 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2014-06-24 20:52|
Done. Thanks Jeff.
|msg221803 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2014-06-28 16:45|
After updating and rebuilding (32 bit VC express, on 64 bit Win 7), I get the same error as in msg210926, with or without -uall, even after turning my antivirus off. File "F:\Python\dev\4\py34\lib\test\test_httpservers.py", line 310, in test_invalid_requests self.check_status_and_reason(response, 501) File "F:\Python\dev\4\py34\lib\test\test_httpservers.py", line 265, in check_status_and_reason self.assertEqual(response.status, status) AssertionError: 200 != 501 Is this a failure of the patch or a different issue?
|msg221858 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2014-06-29 16:12|
If deactivating the AV/firewall doesn't change the behavior, it is presumably a different problem. But let's see what Claudiu has to say, since he was able to reproduce and then circumvent the original problem.
|msg221859 - (view)||Author: Claudiu Popa (Claudiu.Popa) *||Date: 2014-06-29 16:28|
Terry is right, this patch doesn't completely work. But with this fix the problem is solved: diff -r 394e6bda5a70 Lib/test/test_httpservers.py --- a/Lib/test/test_httpservers.py Sun Jun 29 15:56:21 2014 +0300 +++ b/Lib/test/test_httpservers.py Sun Jun 29 19:27:16 2014 +0300 @@ -306,7 +306,7 @@ response = self.request('/', method='FOO') self.check_status_and_reason(response, 501) # requests must be case sensitive,so this should fail too - response = self.request('/', method='get') + response = self.request('/', method='gets') self.check_status_and_reason(response, 501) response = self.request('/', method='GETs') self.check_status_and_reason(response, 501)
|msg221860 - (view)||Author: Claudiu Popa (Claudiu.Popa) *||Date: 2014-06-29 16:29|
Although it is incorrect, because the test specifically tests case sensitivity.
|msg221862 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2014-06-29 16:37|
What if you use 'custom' instead?
|msg221863 - (view)||Author: Claudiu Popa (Claudiu.Popa) *||Date: 2014-06-29 16:52|
It works with 'custom'.
|msg221866 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2014-06-29 17:16|
Both 'gets' and 'custom' work on my machine.
|msg221893 - (view)||Author: Jeff Allen (jeff.allen) *||Date: 2014-06-29 20:06|
Disabling the AV/firewall did not stop the symptoms when I was investigating originally. In order to get the unmodified test to pass, I had to stop the BFE (base filtering engine), which I think may have been given new rules or behaviours as a result of installing the AV solution ... or maybe it was a Windows upgrade that did it. I did wonder if this might be a moving target, as the test deliberately includes server abuse, while the products want to stop that. If I try test_httpservers.py as amended (http://hg.python.org/cpython/file/ffdd2d0b0049/Lib/test/test_httpservers.py) on my machine with CPython 3.4.1, I do not get the error Terry reports. (test_urlquote_decoding_in_cgi_check fails but it should.)
|2014-06-29 20:06:38||jeff.allen||set||messages: + msg221893|
|2014-06-29 17:16:23||terry.reedy||set||messages: + msg221866|
|2014-06-29 16:52:07||Claudiu.Popa||set||messages: + msg221863|
|2014-06-29 16:37:45||r.david.murray||set||status: closed -> open|
resolution: fixed ->
messages: + msg221862
stage: resolved -> needs patch
|2014-06-29 16:29:51||Claudiu.Popa||set||messages: + msg221860|
|2014-06-29 16:28:50||Claudiu.Popa||set||messages: + msg221859|
|2014-06-29 16:12:07||r.david.murray||set||messages: + msg221858|
|2014-06-28 16:45:50||terry.reedy||set||messages: + msg221803|
|2014-06-24 20:52:10||r.david.murray||set||status: open -> closed|
messages: + msg221499
stage: patch review -> resolved
messages: + msg221498
|2014-06-24 13:38:40||Claudiu.Popa||set||messages: + msg221458|
|2014-03-05 01:23:19||r.david.murray||set||messages: + msg212755|
|2014-03-04 23:51:29||jeff.allen||set||messages: + msg212754|
|2014-03-03 18:49:10||r.david.murray||set||messages: + msg212652|
nosy: + fwierzbicki
messages: + msg212026
keywords: + patch
+ orsenthil, ezio.melotti, r.david.murray|
messages: + msg212004
components: + Tests
stage: patch review
|2014-02-23 15:25:35||jeff.allen||set||messages: + msg211999|
messages: + msg211990
messages: + msg210926
versions: + Python 3.4