test_socket.RecvmsgIntoSCMRightsStreamTest fails on AMD64 FreeBSD CURRENT Debug 3.x:

https://buildbot.python.org/all/#/builders/60/builds/248


======================================================================
FAIL: testFDPassSeparateMinSpace (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/pablo/cpython/Lib/test/test_socket.py", line 2881, in testFDPassSeparateMinSpace
    maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
  File "/usr/home/pablo/cpython/Lib/test/test_socket.py", line 2798, in checkRecvmsgFDs
    self.assertEqual(len(fds), numfds)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 18 tests in 0.084s

FAILED (failures=1)
test test_socket failed
test_socket failed

== Tests result: FAILURE ==

1 test failed:
    test_socket


I was able to reproduce the bug in the buildbot but this same test fails in 3.6 and 3.5 in the buildbot so it can be some configuration in the buildbot or some regression in FreeBSD.

This error happens a lot in the same buildbot in different tests:

raceback (most recent call last):
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 3335, in child_access
    w = conn.recv()
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/connection.py", line 960, in rebuild_connection
    fd = df.detach()
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/reduction.py", line 185, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

Copying in my original email to Pablo that elucidated the test failure cause and resolution, including upstream commit/review/issue references for the recvmsg(2) changes IN CURRENT.

Full buildbot worker log attached

While I'm here, update list of versions (branches) affected.

------------------

The test suite fails:

  File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/multiprocessing/reduction.py",
line 164, in recvfds
    raise RuntimeError('received %d items of ancdata' %
RuntimeError: received 0 items of ancdata

For these two tests (further up in the log [1]) ...

test_fd_transfer
(test.test_multiprocessing_forkserver.WithProcessesTestConnection) ... FAIL
test_large_fd_transfer
(test.test_multiprocessing_forkserver.WithProcessesTestConnection) ... FAIL

Further up we see (in the initial test run) ...

4 tests failed:
    test_multiprocessing_fork test_multiprocessing_forkserver
    test_multiprocessing_spawn test_socket

Then further up we see:

======================================================================
FAIL: testFDPassSeparateMinSpace
(test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3186, in testFDPassSeparateMinSpace
    self.checkRecvmsgFDs(2,
  File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3107, in checkRecvmsgFDs
    self.assertEqual(len(fds), numfds)
AssertionError: 1 != 2
======================================================================
FAIL: testFDPassSeparateMinSpace
(test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3186, in testFDPassSeparateMinSpace
    self.checkRecvmsgFDs(2,
  File
"/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_socket.py",
line 3107, in checkRecvmsgFDs
    self.assertEqual(len(fds), numfds)
AssertionError: 1 != 2
----------------------------------------------------------------------
Ran 554 tests in 27.636s
FAILED (failures=2, skipped=57)

I looked for recvmsg() changes in FreeBSD recently, and found ...

Commit: https://svnweb.freebsd.org/base?view=revision&revision=337423
Review: https://reviews.freebsd.org/D16561
Bug ID: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=131876
Bug Desc:[socket] FD leak by receiving SCM_RIGHTS by recvmsg with small
control message buffer

The commit/bug/review details above point to recvmsg specific changes,
specifically, but maybe not *just*, for SCM_RIGHTS messages, which (in
my opinion) is way too similar to the test case specifics not to be related.

Also, the commit date of Aug 7 16:36:48 2018  seems right around the
time I updated the machine and the tests began to fail.

I can't say yet whether or not a bug crept in with the change, *but*,
the commit does say that it *fixes* bugs, which the python tests, or the
socket code in Python may be have been relying on.

It may also be that FreeBSD specific-behaviour isn't being
handled by the code and/or tests (truncated now where it wasn't before?)

It could also be the buffer sizes the socket code/tests use.

------------------

Commit merged into master:

commit 7291108d88ea31d205da4db19d202d6cbffc6d93
Author: Pablo Galindo <Pablogsal@gmail.com>
Date:   Thu Sep 27 10:25:03 2018 +0100

    Fix tests in test_socket to use correctly CMSG_LEN (GH-9594)
    
    After some failures in AMD64 FreeBSD CURRENT Debug 3.x buildbots
    regarding tests in test_socket that are using
    testFDPassSeparateMinSpace(), FreeBDS revision 337423 was pointed
    out to be the reason the test started to fail.
    
    A close examination of the manpage for cmsg_space(3) reveals that
    the number of file descriptors needs to be taken into account when
    using CMSG_LEN().
    
    This commit fixes tests in test_socket to use correctly CMSG_LEN, taking
    into account the number of FDs.

3.7:

commit addef07ca7d7b6971d59c062c3229e91a99e5f5e (upstream/3.7, 3.7)
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Thu Sep 27 06:30:47 2018 -0700

3.6:

commit fe48b6df101aac10dc846fa6fd1a41f877e77025 (upstream/3.6, 3.6)
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Thu Sep 27 06:30:55 2018 -0700

(the commit message doesn't contain bpo-34521, so bots didn't added comments here about the commits)

It looks like the test_socket is fixed but some related issues are still left in multiprocessing.

Is test_socket of Python 2.7 affected by the issue? If not, I suggest to remove 2.7 from this issue and close it.

To clarify affected tests/versions:

Python 3.x - test_socket test_multiprocessing_spawn
Python 2.7 - test_multiprocessing_spawn

All of the issues are related to incorrect recvmsg(2) buffer length use, so I've amended the issue summary to describe the complete root problem.

The be explicit,

- test_socket fix requires backporting to all supported 3.x branches
- test_multiprocessing_* test fixes (via multiprocessing.reduction fix) requires backporting to all supported 3.x branches *and* 2.7

New changeset 077061a7b24917aaf31057885c69919c5a553c88 by Victor Stinner (Pablo Galindo) in branch 'master':
bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613)
https://github.com/python/cpython/commit/077061a7b24917aaf31057885c69919c5a553c88

New changeset 5d33ee1595767d68800fda13bc68b7b01dba5117 by Miss Islington (bot) in branch '3.6':
bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613)
https://github.com/python/cpython/commit/5d33ee1595767d68800fda13bc68b7b01dba5117

New changeset 007fda436f707ac95c2fa8f8886efd9e09d5b630 by Victor Stinner (Miss Islington (bot)) in branch '3.7':
bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613) (GH-9619)
https://github.com/python/cpython/commit/007fda436f707ac95c2fa8f8886efd9e09d5b630

Pablo: I merged your PR on master, and backport it to 3.6 and 3.7. But I let you handle 2.7, since I expect a conflict :-)

Shouldn't there be a NEWS entry for this change since the change seems to be to Lib/multiprocessing and not just its tests?

New changeset bd036d3d15fc1310ccc32a43a3296b8c157ac221 by Pablo Galindo in branch 'master':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/bd036d3d15fc1310ccc32a43a3296b8c157ac221

New changeset d991ede16b34399c5ea9bd30e9a5c520bed6bea8 by Miss Islington (bot) in branch '3.7':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/d991ede16b34399c5ea9bd30e9a5c520bed6bea8

New changeset ae011e00189d9083dd84c357718264e24fe77314 by Miss Islington (bot) in branch '3.6':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/ae011e00189d9083dd84c357718264e24fe77314

New changeset d404ffa8adf252d49731fbd09b740360577274c8 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/d404ffa8adf252d49731fbd09b740360577274c8

New changeset ce3b5a80cc6ee4f9aff7b673f457294d7e054349 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-34521: Add NEWS entry for changes in GH-9613 (GH-9850)
https://github.com/python/cpython/commit/ce3b5a80cc6ee4f9aff7b673f457294d7e054349

Thanks, Pablo, I cherry-picked the NEWS entries into 3.7.1 and 3.6.7.  So I guess the only remaining question here is what to do about 2.7.  I'll also leave that for you!

> Is test_socket of Python 2.7 affected by the issue? If not, I suggest to remove 2.7 from this issue and close it.

Some "FD transfer" tests of test_multiprocessing still fail on AMD64 FreeBSD CURRENT Shared 2.7:

https://buildbot.python.org/all/#/builders/169/builds/36

Re-running test 'test_multiprocessing' in verbose mode
FAIL: test_fd_transfer (test.test_multiprocessing.WithProcessesTestConnection)
FAIL: test_large_fd_transfer (test.test_multiprocessing.WithProcessesTestConnection)

Modules/_multiprocessing/multiprocessing.c uses CMSG_SPACE() *and* CMSG_LEN(). Are they used properly? I don't understand the code, but I propose:

diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
index d192a074ba..eecace887e 100644
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -167,7 +167,7 @@ multiprocessing_recvfd(PyObject *self, PyObject *args)
     cmsg = CMSG_FIRSTHDR(&msg);
     cmsg->cmsg_level = SOL_SOCKET;
     cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
+    cmsg->cmsg_len = CMSG_SPACE(sizeof(int));
     msg.msg_controllen = cmsg->cmsg_len;
 
     Py_BEGIN_ALLOW_THREADS

I did not backport this still because the code is very different and it required more time to understand how it works on 2.7.

I do not have access currently to koobs buildbot (the IP has changed) so I cannot check if that works :(

Access to the buildbot has been restored (a while ago), and this issue being resolved (merge to 2.7) is the only test failing to get 2.7 back to green, which has been failing for months.

Can we test with Victors proposed patch in msg330654 ? I don't believe the change/fix/merge will be difficult, given the very small change scope of the master commits.

I am able to reproduce test_multiprocessing.test_fd_transfer() failure on the FreeBSD CURRENT buildbot:

CURRENT-amd64% ./python -m test -m test_fd_transfer test_multiprocessing 
(...)
Tests result: FAILURE

I converted my msg330654 patch into PR 15422: I confirm that this PR fix test_fd_transfer() and test_large_fd_transfer() of test_multiprocessing.

Note: I also wrote PR 15421 to remove temporary files created by these 2 tests.

New changeset c3642219090f2564c1790330cbf0ba31f19dcaf4 by Victor Stinner in branch '2.7':
bpo-34521: Fix FD transfer in multiprocessing on FreeBSD (GH-15422)
https://github.com/python/cpython/commit/c3642219090f2564c1790330cbf0ba31f19dcaf4

"AMD64 FreeBSD CURRENT Shared 2.7 #159 is complete: Success"

https://buildbot.python.org/all/#/builders/169/builds/159

The FreeBSD 2.7 buildbot is back to green, I close the issue.

Thanks everyone was helped to get these issues to be fixed!