classification
Title: New addition of vSockets to the python socket module
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Cathy Avery, berker.peksag, gregory.p.smith, kushal.das, ncoghlan, r.david.murray
Priority: normal Keywords: patch

Created on 2016-07-21 15:36 by Cathy Avery, last changed 2017-07-24 10:49 by Cathy Avery.

Files
File name Uploaded Description Edit
vsocket.patch Cathy Avery, 2016-07-21 15:36 vsockets for linux support review
vsock_rev2.patch Cathy Avery, 2016-11-09 18:07 vsockets for linux support revision 2 review
nc-vsock Cathy Avery, 2016-12-19 13:09
0001-bpo-27584-New-addition-of-vSockets-to-the-python-soc.patch Cathy Avery, 2017-06-27 18:36
REAME.txt Cathy Avery, 2017-06-27 18:38
Pull Requests
URL Status Linked Edit
PR 2489 open python-dev, 2017-06-29 14:59
Repositories containing patches
https://hg.python.org/cpython
Messages (31)
msg270935 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-07-21 15:36
I have added AF_VSOCK support to python's 3.6 socket module ( socketmodule.c socketmodule.h cloned from https://hg.python.org/cpython ). The implementation is very similar to AF_NETLINK.  AF_VSOCK requires the VMware-specific VMCI transport which is currently upstream or the virtio-vsock drivers developed by Stefan Hajnoczi at Red Hat. The virtio-vsock drivers are not upstream yet but more information with source and build instructions can be found at http://qemu-project.org/Features/VirtioVsock.

More information on vSocket programming can be found at https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/ws9_esx60_vmci_sockets.pdf

The VMCI transport supports SOCK_DGRAM and SOCK_STREAM on both Linux and Windows. Virtio-vsock currently supports SOCK_STREAM only on Linux.

My python addition supports SOCK_STREAM and SOCK_DGRAM calls on Linux only.

I have tested my implementation on both driver sets on Linux.

Attached is a diff file so you can see which files I've modified. These include a new configure.ac. I have already tested the new file generation by running autoreconf. 

Also included in the patch is an updated socket.rst file however I could not get the final html page to be double spaced.
msg272358 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-10 18:45
Looks like there's a missing versionadded directive in the doc patch.

Is it possible/sensible to add tests for the new feature?

(I haven't reviewed the patch in detail, hopefully someone with more experience with C socket programming than I have will do that.)
msg272363 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-08-10 19:16
Sure I can add tests. I would like to base them on the existing socket tests. Where are those?

I did add a version

+  .. versionadded:: 3.4

It just not may not be the right one.
msg272366 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-10 19:32
Ah, I see.  No, the versionadded will be 3.6, and should go *after* the documentation of the new socket type.

The existing socket tests are in Lib/test/test_socket.py.
msg272403 - (view) Author: Kushal Das (kushal.das) * (Python committer) Date: 2016-08-11 06:46
The patch can be applied, and build successfully. I have ran the current test suite[1]. The two failed tests do not seem to be have anything to do with this patch (read the end of the consoleText output). I think the thing remaining is the new test cases, and NEWS file update. 


[1] https://ci.centos.org/job/cPython-build-patch/24/consoleText
msg280423 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-11-09 18:07
Please forgive the long delay in providing this update. I got a little sidetracked. Attached is the patch for Python 3.7. It includes fixes suggested in rev 1 plus VSOCK tests in test_socket.py.

Thanks,

Cathy
msg282798 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-12-09 19:14
Is there anything else that is needed for this patch?

Thanks!
msg283534 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-12-18 03:31
The second patch seems to be missing the configure changes.  Also, the tests have some over-long lines (we limit line lengths to 79 characters).  I realize there are other long lines in that file, but no need to add more :)

There is trailing whitespace on a number of lines in your patch.

Since this is new, we may not want to accept it until the support hits upstream.  Specifically, it will be difficult to get a review if the reviewer has to build a custom kernel to test the code :)  You do say that the VMCI is upstream, but I don't know what that means.  Which upstream?

Note: I'm not familiar with the socket C code, so I haven't reviewed the C code changes.  The tests look fine to me.

For the docs, the proposal doesn't seem to follow the format of the existing docs.  I would expect only the first paragraph located where you have it.  The remaining constants should be in the 'module contents'/'constants' section, I think.  Yes, that means each one gets a '.. versionadded' label.  Presumably also an 'availablility' label with whatever the minimum kernel version is...another reason we may need to wait.
msg283535 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-12-18 03:37
Oh, I see, the ac changes are there, I was looking at the patch delta instead of the complete patch.
msg283607 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-12-19 11:55
Is there a format checker I could use on the patch?

VMCI and the vmw_vsock_vmci_transport kernel modules are located in the upstream linux tree at 

git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

They have been there for about years. These drivers are part of various downstream kernels such as RHEL. You will need a Vmware virtual machine in order to test it. 

Only the virtio-vsock driver is a new vsock application that needs to be custom built.
msg283608 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-12-19 11:56
Sorry about the typo the drivers have been there for about 4 years.
msg283612 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-12-19 12:12
'make patchcheck' will do whitespace checking because that's hard to eyeball (although many editors/IDEs do support making it visible nowadays).  We don't use any other checking tools other than eyeballs, since not all of the existing code conforms to PEP7/8 and for various reasons we aren't going to update most of the old code to conform.

So, if I'm running an ubuntu virtual machine under VMWare Fusion (which I already have set up) I should be able to get the tests to run?  Or does it need to be RedHat (or presumably CentOS)?
msg283620 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2016-12-19 13:09
First make sure the driver is in your kernel. It will be with RHEL. Look in /lib/modeles/"your kernel name"/kernel/net/vmw_vsock/vmw_vsock_vmci_transport. I have never tried it on vmware fusion. I have tested it on ESX. See if there is a VMCI option to enable on your VM's settings. Start the vm and do an lsmod to see if vmw_vsock_vmci_transport is loaded.

I've attached a little C program thats netcat for vsock. Its a quick confirmation that your transport is loaded correctly. It will show you your CID.

run ./nc-vsock

CID = 973033371
CID = 0x39ff4f9b
usage: ./nc-vsock [-l <port> [-t <dst> <dstport>] | <cid> <port>]
msg296410 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-06-20 10:51
The vsock code is now in the linux upstream kernel and qemu. I will be resubmitting my vsock patches for python. So from what I can tell the tip of the devel tree is for 3.7 and that the source control has switched to git. My question is do I need a github account or can I push git patches directly to this issues page? 

Thanks,

Cathy
msg296413 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-06-20 11:15
You don't need a GitHub account for contributing to CPython, but the pull request workflow is the preferred way. You can still attach your patches to this issue.
msg297064 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-06-27 18:36
I've attached the third version of VSOCK patch addressing the concerns of the last rev. I've also included a README file that lists instructions on how to setup a test environment.

Thanks
msg297065 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-06-27 18:38
Help file.
msg297276 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-06-29 15:18
I also issued a pull request

https://github.com/python/cpython/pull/2489

Let me know if I screwed it up.

Thanks,

Cathy
msg297285 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-06-29 17:10
The build and test bots failed but I don't understand why. The build could not find module _socket but that is not new and other modules failed.

The test could not import fcntl which I did add but should not fail as I have run this test many time and other tests import fcntl.

https://github.com/python/cpython/pull/2489
msg297516 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-07-02 13:04
I'm attempting to figure out whether or not we have a buildbot in the Buildbot fleet that will cover this test case.

Based on the pre-merge CI run, it seems Ubuntu 14.04 is too old to include the required kernel headers.

However, it looks like RHEL/CentOS are also currently still missing the userspace changes to fully enable AF_VSOCK support (as the Red Hat backport flow appears to have gone through the dedicated hypervisor variant first): https://bugzilla.redhat.com/show_bug.cgi?id=1315822

So it's looking to me like we're going to need either a recent Fedora, a non-LTS Ubuntu, or a Debian 9 system to be confident we have the right headers available.
msg297568 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-03 11:18
Fedora 25 has the proper headers.
msg297835 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-06 18:51
So I revised my code based on the reviews and I passed all the checks ... now what?

Thanks,

Cathy
msg298165 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-07-11 17:29
I think we are waiting on confirmation that we have a buildbot that has the necessary headers.
msg298166 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-11 17:31
OK thanks!
msg298167 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-07-11 17:35
I updated my PGO buildbot to Debian 9 which should have them.

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.x

~$ grep AF_VSOCK /usr/include/*/*/*
/usr/include/x86_64-linux-gnu/bits/socket.h:#define AF_VSOCK    PF_VSOCK
msg298169 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-11 17:50
You will also need linux/vm_sockets.h in order to build.
msg298171 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-07-11 17:59
yep, linux/vm_sockets.h exists.  I believe it's kernel headers are from 4.9.
msg298173 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-11 18:12
That should do it.
msg298174 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-07-11 18:14
Everything compiles successfully on this host - configure detects the header has HAVE_LINUX_VM_SOCKETS_H set to 1.

test_socket passes on this host.  However since i'm not running with a /dev/vsock, the unittests (correctly) skip the new tests.

testCreateSocket (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.'
testCrucialConstants (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.'
testSocketBufferSize (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.'
testVSOCKConstants (test.test_socket.BasicVSOCKTest) ... skipped 'VSOCK sockets required for this test.'
testStream (test.test_socket.ThreadedVSOCKSocketStreamTest) ... skipped 'VSOCK sockets required for this test.'
msg298805 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-21 14:26
Hi,

I believe I am waiting for a final review. Is there anything else I need to be doing at this point.

Thanks,

Cathy
msg298953 - (view) Author: Cathy Avery (Cathy Avery) * Date: 2017-07-24 10:49
There is an outstanding review on my pull request at https://github.com/python/cpython/pull/2489 as there is an red X at changes requested by kushaldas and I believe I have made the necessary changes. 

Again please let me know if there is anything that I need to do as I am new to this process.

Thanks,

Cathy
History
Date User Action Args
2017-07-24 10:49:27Cathy Averysetmessages: + msg298953
2017-07-21 14:26:04Cathy Averysetmessages: + msg298805
2017-07-11 18:14:09gregory.p.smithsetmessages: + msg298174
2017-07-11 18:12:07Cathy Averysetmessages: + msg298173
2017-07-11 17:59:05gregory.p.smithsetmessages: + msg298171
2017-07-11 17:50:21Cathy Averysetmessages: + msg298169
2017-07-11 17:35:58gregory.p.smithsetmessages: + msg298167
2017-07-11 17:31:54Cathy Averysetmessages: + msg298166
2017-07-11 17:29:34r.david.murraysetmessages: + msg298165
2017-07-06 18:51:20Cathy Averysetmessages: + msg297835
2017-07-03 11:18:10Cathy Averysetmessages: + msg297568
2017-07-02 13:04:08ncoghlansetnosy: + ncoghlan
messages: + msg297516
2017-06-29 17:10:44Cathy Averysetmessages: + msg297285
2017-06-29 15:18:02Cathy Averysetmessages: + msg297276
2017-06-29 14:59:43python-devsetpull_requests: + pull_request2548
2017-06-27 18:38:00Cathy Averysetfiles: + REAME.txt

messages: + msg297065
2017-06-27 18:36:58Cathy Averysetfiles: + 0001-bpo-27584-New-addition-of-vSockets-to-the-python-soc.patch

messages: + msg297064
2017-06-20 11:15:25berker.peksagsetnosy: + berker.peksag
messages: + msg296413
2017-06-20 10:51:36Cathy Averysetmessages: + msg296410
2016-12-19 13:09:07Cathy Averysetfiles: + nc-vsock

messages: + msg283620
2016-12-19 12:12:01r.david.murraysetmessages: + msg283612
2016-12-19 11:56:50Cathy Averysetmessages: + msg283608
2016-12-19 11:55:36Cathy Averysetmessages: + msg283607
2016-12-18 03:37:44r.david.murraysetmessages: + msg283535
2016-12-18 03:31:32r.david.murraysetmessages: + msg283534
2016-12-09 19:14:08Cathy Averysetmessages: + msg282798
2016-11-09 18:07:21Cathy Averysetfiles: + vsock_rev2.patch

messages: + msg280423
versions: + Python 3.7, - Python 3.6
2016-08-11 06:46:06kushal.dassetnosy: + kushal.das
messages: + msg272403
2016-08-10 19:32:34r.david.murraysetmessages: + msg272366
2016-08-10 19:16:29Cathy Averysetmessages: + msg272363
2016-08-10 18:45:17r.david.murraysetnosy: + r.david.murray

messages: + msg272358
stage: patch review
2016-07-21 15:36:30Cathy Averycreate