Multiple tests (test_socket, test_multiprocessing_*) fail due to incorrect recvmsg(2) buffer lengths, causing failures on FreeBSD CURRENT #78702
Comments
|
test_socket.RecvmsgIntoSCMRightsStreamTest fails on AMD64 FreeBSD CURRENT Debug 3.x: https://buildbot.python.org/all/#/builders/60/builds/248 ====================================================================== Traceback (most recent call last):
File "/usr/home/pablo/cpython/Lib/test/test_socket.py", line 2881, in testFDPassSeparateMinSpace
maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
File "/usr/home/pablo/cpython/Lib/test/test_socket.py", line 2798, in checkRecvmsgFDs
self.assertEqual(len(fds), numfds)
AssertionError: 1 != 2Ran 18 tests in 0.084s FAILED (failures=1) == Tests result: FAILURE == 1 test failed: I was able to reproduce the bug in the buildbot but this same test fails in 3.6 and 3.5 in the buildbot so it can be some configuration in the buildbot or some regression in FreeBSD. |
pablogsal
added
tests
|
This error happens a lot in the same buildbot in different tests: raceback (most recent call last): |
|
Copying in my original email to Pablo that elucidated the test failure cause and resolution, including upstream commit/review/issue references for the recvmsg(2) changes IN CURRENT. Full buildbot worker log attached While I'm here, update list of versions (branches) affected. ------------------ The test suite fails: File For these two tests (further up in the log [1]) ... test_fd_transfer Further up we see (in the initial test run) ... 4 tests failed: Then further up we see: ====================================================================== Traceback (most recent call last):
File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3186, in testFDPassSeparateMinSpace
self.checkRecvmsgFDs(2,
File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3107, in checkRecvmsgFDs
self.assertEqual(len(fds), numfds)
AssertionError: 1 != 2====================================================================== Traceback (most recent call last):
File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3186, in testFDPassSeparateMinSpace
self.checkRecvmsgFDs(2,
File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3107, in checkRecvmsgFDs
self.assertEqual(len(fds), numfds)
AssertionError: 1 != 2Ran 554 tests in 27.636s I looked for recvmsg() changes in FreeBSD recently, and found ... Commit: https://svnweb.freebsd.org/base?view=revision&revision=337423 The commit/bug/review details above point to recvmsg specific changes, Also, the commit date of Aug 7 16:36:48 2018 seems right around the I can't say yet whether or not a bug crept in with the change, *but*, It may also be that FreeBSD specific-behaviour isn't being It could also be the buffer sizes the socket code/tests use. ------------------ |
koobs
changed the title
|
Commit merged into master: commit 7291108 3.7: commit addef07 (upstream/3.7, 3.7) 3.6: commit fe48b6d (upstream/3.6, 3.6) (the commit message doesn't contain bpo-34521, so bots didn't added comments here about the commits) |
|
It looks like the test_socket is fixed but some related issues are still left in multiprocessing. |
|
Is test_socket of Python 2.7 affected by the issue? If not, I suggest to remove 2.7 from this issue and close it. |
|
To clarify affected tests/versions: Python 3.x - test_socket test_multiprocessing_spawn All of the issues are related to incorrect recvmsg(2) buffer length use, so I've amended the issue summary to describe the complete root problem. The be explicit,
|
koobs
changed the title
|
Pablo: I merged your PR on master, and backport it to 3.6 and 3.7. But I let you handle 2.7, since I expect a conflict :-) |
|
Shouldn't there be a NEWS entry for this change since the change seems to be to Lib/multiprocessing and not just its tests? |
|
Thanks, Pablo, I cherry-picked the NEWS entries into 3.7.1 and 3.6.7. So I guess the only remaining question here is what to do about 2.7. I'll also leave that for you! |
Some "FD transfer" tests of test_multiprocessing still fail on AMD64 FreeBSD CURRENT Shared 2.7: https://buildbot.python.org/all/#/builders/169/builds/36 Re-running test 'test_multiprocessing' in verbose mode Modules/_multiprocessing/multiprocessing.c uses CMSG_SPACE() *and* CMSG_LEN(). Are they used properly? I don't understand the code, but I propose: diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
index d192a074ba..eecace887e 100644
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -167,7 +167,7 @@ multiprocessing_recvfd(PyObject *self, PyObject *args)
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_RIGHTS;
- cmsg->cmsg_len = CMSG_LEN(sizeof(int));
+ cmsg->cmsg_len = CMSG_SPACE(sizeof(int));
msg.msg_controllen = cmsg->cmsg_len;
Py_BEGIN_ALLOW_THREADS |
|
I did not backport this still because the code is very different and it required more time to understand how it works on 2.7. I do not have access currently to koobs buildbot (the IP has changed) so I cannot check if that works :( |
|
Access to the buildbot has been restored (a while ago), and this issue being resolved (merge to 2.7) is the only test failing to get 2.7 back to green, which has been failing for months. Can we test with Victors proposed patch in msg330654 ? I don't believe the change/fix/merge will be difficult, given the very small change scope of the master commits. |
|
I am able to reproduce test_multiprocessing.test_fd_transfer() failure on the FreeBSD CURRENT buildbot: CURRENT-amd64% ./python -m test -m test_fd_transfer test_multiprocessing I converted my msg330654 patch into PR 15422: I confirm that this PR fix test_fd_transfer() and test_large_fd_transfer() of test_multiprocessing. |
|
Note: I also wrote PR 15421 to remove temporary files created by these 2 tests. |
|
"AMD64 FreeBSD CURRENT Shared 2.7 #159 is complete: Success" https://buildbot.python.org/all/#/builders/169/builds/159 The FreeBSD 2.7 buildbot is back to green, I close the issue. Thanks everyone was helped to get these issues to be fixed! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: