classification
Title: multiprocessing example does not work on Windows
Type: behavior Stage:
Components: Documentation Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: multiprocessing example "pool of http servers " fails on windows
View: 5879
Assigned To: docs@python Nosy List: docs@python, jmaki, r.david.murray, sbt, terry.reedy, wolma
Priority: normal Keywords:

Created on 2014-04-12 01:50 by jmaki, last changed 2014-08-29 23:42 by terry.reedy. This issue is now closed.

Messages (10)
msg215957 - (view) Author: jmaki (jmaki) Date: 2014-04-12 01:50
Would you consider putting examples given in official documentation as part of release testing?  e.g. (from official python.org documentation):
----------------- snip ---------------------

An example of how a pool of worker processes can each run a SimpleHTTPServer.HttpServer instance while sharing a single listening socket.

#
# Example where a pool of http servers share a single listening socket
#
# On Windows this module depends on the ability to pickle a socket
# object so that the worker processes can inherit a copy of the server
# object.  (We import `multiprocessing.reduction` to enable this pickling.)
#
# Not sure if we should synchronize access to `socket.accept()` method by
# using a process-shared lock -- does not seem to be necessary.
#
# Copyright (c) 2006-2008, R Oudkerk
# All rights reserved.
#

import os
import sys

from multiprocessing import Process, current_process, freeze_support
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

if sys.platform == 'win32':
    import multiprocessing.reduction    # make sockets pickable/inheritable


def note(format, *args):
    sys.stderr.write('[%s]\t%s\n' % (current_process().name, format%args))


class RequestHandler(SimpleHTTPRequestHandler):
    # we override log_message() to show which process is handling the request
    def log_message(self, format, *args):
        note(format, *args)

def serve_forever(server):
    note('starting server')
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        pass


def runpool(address, number_of_processes):
    # create a single server object -- children will each inherit a copy
    server = HTTPServer(address, RequestHandler)

    # create child processes to act as workers
    for i in range(number_of_processes-1):
        Process(target=serve_forever, args=(server,)).start()

    # main process also acts as a worker
    serve_forever(server)


def test():
    DIR = os.path.join(os.path.dirname(__file__), '..')
    ADDRESS = ('localhost', 8000)
    NUMBER_OF_PROCESSES = 4

    print 'Serving at http://%s:%d using %d worker processes' % \
          (ADDRESS[0], ADDRESS[1], NUMBER_OF_PROCESSES)
    print 'To exit press Ctrl-' + ['C', 'Break'][sys.platform=='win32']

    os.chdir(DIR)
    runpool(ADDRESS, NUMBER_OF_PROCESSES)


if __name__ == '__main__':
    freeze_support()
    test()
----------------- snip ---------------------
does not work on windows...

Regards,
John
msg215994 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-13 04:11
Testing the documentation examples is a long term goal, which people occasionally contribute to.  I think there is an open issue about using the Sphinx doctest support, that for a long time was blocked by the documentation tool chain not using python3.  Some examples are very hard to run in an automated fashion, as well.  Perhaps the examples can be tweaked...but in some cases they can't, without losing their teaching value.  I have a feeling the multiprocessing examples fall into this category, as I seem to remember doing a pass of testing of them a (python3) release or two ago.  I did not test them on windows, though.
msg215999 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-04-13 07:15
jmaki: not sure what the issue is that you are reporting. I can see three different possible issues.
1. "Would you consider putting examples...". As R. David said: yes, we consider it. If you really wanted to know whether we consider it, we could close the issue as fixed.

2. "Please make examples ...". This would be a (near) duplicate of #15939, so I should close this issue as a duplicate.

3. "The specific example does not work on Windows". This issue should stay open until it is analysed an possibly fixed. If that is the issue, then it would be helpful if you could report how exactly the example fails.

I'd like to see a strict "one issue at a time" policy in this tracker, so please let us know how you would like us to proceed.
msg216135 - (view) Author: jmaki (jmaki) Date: 2014-04-14 17:49
loewis, r.david.murray: Please proceed in terms of #3--"The specific example does not work on Windows".  I am using python 2.7.6 on Windows 7.

Below is the failing message that happens immediately on startup, specifically, when hitting this line:
        Process(target=serve_forever, args=(server,)).start()

Thank you for the attention.

----- error message -----
Traceback (most recent call last):
  File "C:/Python27/httpthreadpool.py", line 75, in <module>
    test()
  File "C:/Python27/httpthreadpool.py", line 70, in test
    runpool(ADDRESS, NUMBER_OF_PROCESSES)
  File "C:/Python27/httpthreadpool.py", line 51, in runpool
    Process(target=serve_forever, args=(server,)).start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 725, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock
msg216186 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-04-14 19:26
Ah, Python 2.7. Note that the example is gone in Python 3, as part of the large change in issue8713.
msg216195 - (view) Author: jmaki (jmaki) Date: 2014-04-14 20:09
Upon further investigation, this may be related to:
http://bugs.python.org/issue1378
However, it seems the issue is not checked-in to Windows release for 2.x?

Regards,
John
msg216241 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-14 22:32
If you look at the source code for 2.7, it is clear that patch has been applied.
msg216242 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-14 22:35
To clarify, the commit in that issue is in 2.7.  So if there is something else that isn't in 2.7, it is a different issue.
msg216797 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-18 20:03
When quoting from the docs, it is helpful to give a link.
https://docs.python.org/2/library/multiprocessing.html#examples
That also identifies the version.

I verified that the example fails on my 2.7.6 Windows 7 with
PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock 

Since the example has been removed for 3.x, a possible minimal fix would be to say that it does not work on Windows and remove the statement that implies that it does.
  if sys.platform == 'win32': import multiprocessing.reduction
  # make sockets pickable/inheritable
msg226107 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-08-29 23:42
The same example failing is the subject of #5879.
History
Date User Action Args
2014-08-29 23:42:32terry.reedysetstatus: open -> closed
superseder: multiprocessing example "pool of http servers " fails on windows
resolution: duplicate
messages: + msg226107
2014-04-18 20:03:21terry.reedysetnosy: + terry.reedy
messages: + msg216797
2014-04-14 22:35:35r.david.murraysetmessages: + msg216242
2014-04-14 22:32:18r.david.murraysetmessages: + msg216241
2014-04-14 20:09:35jmakisetmessages: + msg216195
2014-04-14 19:27:02loewissetnosy: - loewis
2014-04-14 19:26:43loewissetnosy: loewis, r.david.murray, docs@python, sbt, wolma, jmaki
messages: + msg216186
2014-04-14 18:59:36loewissettitle: published examples don't work -> multiprocessing example does not work on Windows
2014-04-14 17:49:47jmakisetmessages: + msg216135
2014-04-13 07:15:40loewissetnosy: + loewis
messages: + msg215999
2014-04-13 04:11:08r.david.murraysetnosy: + docs@python, r.david.murray
messages: + msg215994

assignee: docs@python
components: + Documentation, - Cross-Build
2014-04-12 13:39:45berker.peksagsetnosy: + sbt
2014-04-12 13:12:45wolmasetnosy: + wolma
2014-04-12 01:50:40jmakicreate