Issue11119
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2011-02-04 19:24 by pen hill, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (3) | |||
---|---|---|---|
msg127922 - (view) | Author: (pen hill) | Date: 2011-02-04 19:24 | |
When I run the following listing (server_multi.py) by using multiprocessing module of python, it runs successfully on a Linux machine. However, on a 64-bit windows machine I get the following error: pickle.PicklingError: Can't pickle <built-in method recvfrom_into of _socket.socket object at 0x0000000002D2CF10>: it's not found as __main__.recvfrom_into This module is from the book Foundations of Python Network Programming, 2nd Edition. I am pasting the code with the permission of the author. It seems that socket "listen_sock" cannot be pickled under 64-bit Windows, however it can be pickled under Linux. How can we remove this issue on Windows? It is interesting that if I use the threading module, it works for both Windows and Linux. Thanks. server_multi.py : #usage: $ python server_multi.py localhost process #!/usr/bin/env python # Foundations of Python Network Programming - Chapter 7 - server_multi.py # Using multiple threads or processes to serve several clients in parallel. import sys, time, lancelot from multiprocessing import Process from server_simple import server_loop from threading import Thread WORKER_CLASSES = {'thread': Thread, 'process': Process} WORKER_MAX = 10 def start_worker(Worker, listen_sock): worker = Worker(target=server_loop, args=(listen_sock,)) worker.daemon = True # exit when the main process does worker.start() return worker if __name__ == '__main__': if len(sys.argv) != 3 or sys.argv[2] not in WORKER_CLASSES: print >>sys.stderr, 'usage: server_multi.py interface thread|process' sys.exit(2) Worker = WORKER_CLASSES[sys.argv.pop()] # setup() wants len(argv)==2 # Every worker will accept() forever on the same listening socket. listen_sock = lancelot.setup() workers = [] for i in range(WORKER_MAX): workers.append(start_worker(Worker, listen_sock)) # Check every two seconds for dead workers, and replace them. while True: time.sleep(2) for worker in workers: if not worker.is_alive(): print worker.name, "died; starting replacement worker" workers.remove(worker) workers.append(start_worker(Worker, listen_sock)) lancelot.py: #!/usr/bin/env python # Foundations of Python Network Programming - Chapter 7 - lancelot.py # Constants and routines for supporting a certain network conversation. import socket, sys PORT = 1060 qa = (('What is your name?', 'My name is Sir Lancelot of Camelot.'), ('What is your quest?', 'To seek the Holy Grail.'), ('What is your favorite color?', 'Blue.')) qadict = dict(qa) def recv_until(sock, suffix): message = '' while not message.endswith(suffix): data = sock.recv(4096) if not data: raise EOFError('socket closed before we saw %r' % suffix) message += data return message def setup(): if len(sys.argv) != 2: print >>sys.stderr, 'usage: %s interface' % sys.argv[0] exit(2) interface = sys.argv[1] sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.bind((interface, PORT)) sock.listen(128) print 'Ready and listening at %r port %d' % (interface, PORT) return sock server_simple.py: #!/usr/bin/env python # Foundations of Python Network Programming - Chapter 7 - server_simple.py # Simple server that only serves one client at a time; others have to wait. import lancelot def handle_client(client_sock): try: while True: question = lancelot.recv_until(client_sock, '?') answer = lancelot.qadict[question] client_sock.sendall(answer) except EOFError: client_sock.close() def server_loop(listen_sock): while True: client_sock, sockname = listen_sock.accept() handle_client(client_sock) if __name__ == '__main__': listen_sock = lancelot.setup() server_loop(listen_sock) |
|||
msg127975 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-02-05 08:25 | |
Well, sockets cannot be pickled on any platform: >>> sock = socket.create_connection(("www.python.org", 80)) __main__:1: ResourceWarning: unclosed <socket.socket object, fd=3, family=2, type=1, proto=0> >>> s = pickle.loads(pickle.dumps(sock)) >>> s.getpeername() Traceback (most recent call last): File "<stdin>", line 1, in <module> socket.error: getsockaddrlen: bad family >>> s.fileno() -1 The reason your code works under Linux is that multiprocessing uses fork() and therefore all objects and file handles are transparently inherited by the child. Windows doesn't have fork(), it instead spawns a new process to which it must marshal objects using pickle. You'll have to create your socket in the child for it to work at all. By the way, multi-threading is much more appropriate than multi-processing when writing servers under Windows. Also, see the socketserver module for helpers to write both multi-threaded and multi-processed servers: http://docs.python.org/library/socketserver.html |
|||
msg127977 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2011-02-05 08:31 | |
Opened issue11127 for raising a TypeError when trying to pickle any socket. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:57:12 | admin | set | github: 55328 |
2011-02-05 08:31:28 | pitrou | set | status: pending -> closed nosy: pitrou, jnoller, asksol, pen hill messages: + msg127977 |
2011-02-05 08:25:06 | pitrou | set | status: open -> pending nosy: + pitrou messages: + msg127975 resolution: not a bug |
2011-02-05 04:56:07 | r.david.murray | set | nosy:
+ asksol, jnoller type: crash -> behavior components: + Library (Lib), - None |
2011-02-04 19:24:22 | pen hill | create |