This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Passing a socket to a process (multiprocessing module)
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: asksol, jnoller, pen hill, pitrou
Priority: normal Keywords:

Created on 2011-02-04 19:24 by pen hill, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (3)
msg127922 - (view) Author: (pen hill) Date: 2011-02-04 19:24
When I run the following listing (server_multi.py) by using multiprocessing module of python, it runs successfully on a Linux machine. However, on a 64-bit windows machine I get the following error:

pickle.PicklingError: Can't pickle <built-in method recvfrom_into of _socket.socket object at 0x0000000002D2CF10>: it's not found as __main__.recvfrom_into

This module is from the book Foundations of Python Network Programming, 2nd Edition. I am pasting the code with the permission of the author. It seems that socket "listen_sock" cannot be pickled under 64-bit
Windows, however it can be pickled under Linux. How can we remove this issue on Windows? It is interesting that if I use the threading module, it works for both Windows and Linux. Thanks. 

server_multi.py :
#usage: $ python server_multi.py localhost process
#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 7 - server_multi.py
# Using multiple threads or processes to serve several clients in parallel.

import sys, time, lancelot
from multiprocessing import Process
from server_simple import server_loop
from threading import Thread

WORKER_CLASSES = {'thread': Thread, 'process': Process}
WORKER_MAX = 10

def start_worker(Worker, listen_sock):
    worker = Worker(target=server_loop, args=(listen_sock,))
    worker.daemon = True  # exit when the main process does
    worker.start()
    return worker

if __name__ == '__main__':
    if len(sys.argv) != 3 or sys.argv[2] not in WORKER_CLASSES:
        print >>sys.stderr, 'usage: server_multi.py interface thread|process'
        sys.exit(2)
    Worker = WORKER_CLASSES[sys.argv.pop()]  # setup() wants len(argv)==2

    # Every worker will accept() forever on the same listening socket.

    listen_sock = lancelot.setup()
    workers = []
    for i in range(WORKER_MAX):
        workers.append(start_worker(Worker, listen_sock))

    # Check every two seconds for dead workers, and replace them.

    while True:
        time.sleep(2)
        for worker in workers:
            if not worker.is_alive():
                print worker.name, "died; starting replacement worker"
                workers.remove(worker)
                workers.append(start_worker(Worker, listen_sock))

lancelot.py:

#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 7 - lancelot.py
# Constants and routines for supporting a certain network conversation.

import socket, sys

PORT = 1060
qa = (('What is your name?', 'My name is Sir Lancelot of Camelot.'),
      ('What is your quest?', 'To seek the Holy Grail.'),
      ('What is your favorite color?', 'Blue.'))
qadict = dict(qa)

def recv_until(sock, suffix):
    message = ''
    while not message.endswith(suffix):
        data = sock.recv(4096)
        if not data:
            raise EOFError('socket closed before we saw %r' % suffix)
        message += data
    return message

def setup():
    if len(sys.argv) != 2:
        print >>sys.stderr, 'usage: %s interface' % sys.argv[0]
        exit(2)
    interface = sys.argv[1]
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((interface, PORT))
    sock.listen(128)
    print 'Ready and listening at %r port %d' % (interface, PORT)
    return sock

server_simple.py:

#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 7 - server_simple.py
# Simple server that only serves one client at a time; others have to wait.

import lancelot

def handle_client(client_sock):
    try:
        while True:
            question = lancelot.recv_until(client_sock, '?')
            answer = lancelot.qadict[question]
            client_sock.sendall(answer)
    except EOFError:
        client_sock.close()

def server_loop(listen_sock):
    while True:
        client_sock, sockname = listen_sock.accept()
        handle_client(client_sock)

if __name__ == '__main__':
    listen_sock = lancelot.setup()
    server_loop(listen_sock)
msg127975 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-05 08:25
Well, sockets cannot be pickled on any platform:

>>> sock = socket.create_connection(("www.python.org", 80))
__main__:1: ResourceWarning: unclosed <socket.socket object, fd=3, family=2, type=1, proto=0>
>>> s = pickle.loads(pickle.dumps(sock))
>>> s.getpeername()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.error: getsockaddrlen: bad family
>>> s.fileno()
-1

The reason your code works under Linux is that multiprocessing uses fork() and therefore all objects and file handles are transparently inherited by the child. Windows doesn't have fork(), it instead spawns a new process to which it must marshal objects using pickle. You'll have to create your socket in the child for it to work at all.

By the way, multi-threading is much more appropriate than multi-processing when writing servers under Windows. Also, see the socketserver module for helpers to write both multi-threaded and multi-processed servers: http://docs.python.org/library/socketserver.html
msg127977 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-05 08:31
Opened issue11127 for raising a TypeError when trying to pickle any socket.
History
Date User Action Args
2022-04-11 14:57:12adminsetgithub: 55328
2011-02-05 08:31:28pitrousetstatus: pending -> closed
nosy: pitrou, jnoller, asksol, pen hill
messages: + msg127977
2011-02-05 08:25:06pitrousetstatus: open -> pending

nosy: + pitrou
messages: + msg127975

resolution: not a bug
2011-02-05 04:56:07r.david.murraysetnosy: + asksol, jnoller
type: crash -> behavior
components: + Library (Lib), - None
2011-02-04 19:24:22pen hillcreate