Issue21204
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2014-04-12 01:50 by jmaki, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (10) | |||
---|---|---|---|
msg215957 - (view) | Author: jmaki (jmaki) | Date: 2014-04-12 01:50 | |
Would you consider putting examples given in official documentation as part of release testing? e.g. (from official python.org documentation): ----------------- snip --------------------- An example of how a pool of worker processes can each run a SimpleHTTPServer.HttpServer instance while sharing a single listening socket. # # Example where a pool of http servers share a single listening socket # # On Windows this module depends on the ability to pickle a socket # object so that the worker processes can inherit a copy of the server # object. (We import `multiprocessing.reduction` to enable this pickling.) # # Not sure if we should synchronize access to `socket.accept()` method by # using a process-shared lock -- does not seem to be necessary. # # Copyright (c) 2006-2008, R Oudkerk # All rights reserved. # import os import sys from multiprocessing import Process, current_process, freeze_support from BaseHTTPServer import HTTPServer from SimpleHTTPServer import SimpleHTTPRequestHandler if sys.platform == 'win32': import multiprocessing.reduction # make sockets pickable/inheritable def note(format, *args): sys.stderr.write('[%s]\t%s\n' % (current_process().name, format%args)) class RequestHandler(SimpleHTTPRequestHandler): # we override log_message() to show which process is handling the request def log_message(self, format, *args): note(format, *args) def serve_forever(server): note('starting server') try: server.serve_forever() except KeyboardInterrupt: pass def runpool(address, number_of_processes): # create a single server object -- children will each inherit a copy server = HTTPServer(address, RequestHandler) # create child processes to act as workers for i in range(number_of_processes-1): Process(target=serve_forever, args=(server,)).start() # main process also acts as a worker serve_forever(server) def test(): DIR = os.path.join(os.path.dirname(__file__), '..') ADDRESS = ('localhost', 8000) NUMBER_OF_PROCESSES = 4 print 'Serving at http://%s:%d using %d worker processes' % \ (ADDRESS[0], ADDRESS[1], NUMBER_OF_PROCESSES) print 'To exit press Ctrl-' + ['C', 'Break'][sys.platform=='win32'] os.chdir(DIR) runpool(ADDRESS, NUMBER_OF_PROCESSES) if __name__ == '__main__': freeze_support() test() ----------------- snip --------------------- does not work on windows... Regards, John |
|||
msg215994 - (view) | Author: R. David Murray (r.david.murray) * ![]() |
Date: 2014-04-13 04:11 | |
Testing the documentation examples is a long term goal, which people occasionally contribute to. I think there is an open issue about using the Sphinx doctest support, that for a long time was blocked by the documentation tool chain not using python3. Some examples are very hard to run in an automated fashion, as well. Perhaps the examples can be tweaked...but in some cases they can't, without losing their teaching value. I have a feeling the multiprocessing examples fall into this category, as I seem to remember doing a pass of testing of them a (python3) release or two ago. I did not test them on windows, though. |
|||
msg215999 - (view) | Author: Martin v. Löwis (loewis) * ![]() |
Date: 2014-04-13 07:15 | |
jmaki: not sure what the issue is that you are reporting. I can see three different possible issues. 1. "Would you consider putting examples...". As R. David said: yes, we consider it. If you really wanted to know whether we consider it, we could close the issue as fixed. 2. "Please make examples ...". This would be a (near) duplicate of #15939, so I should close this issue as a duplicate. 3. "The specific example does not work on Windows". This issue should stay open until it is analysed an possibly fixed. If that is the issue, then it would be helpful if you could report how exactly the example fails. I'd like to see a strict "one issue at a time" policy in this tracker, so please let us know how you would like us to proceed. |
|||
msg216135 - (view) | Author: jmaki (jmaki) | Date: 2014-04-14 17:49 | |
loewis, r.david.murray: Please proceed in terms of #3--"The specific example does not work on Windows". I am using python 2.7.6 on Windows 7. Below is the failing message that happens immediately on startup, specifically, when hitting this line: Process(target=serve_forever, args=(server,)).start() Thank you for the attention. ----- error message ----- Traceback (most recent call last): File "C:/Python27/httpthreadpool.py", line 75, in <module> test() File "C:/Python27/httpthreadpool.py", line 70, in test runpool(ADDRESS, NUMBER_OF_PROCESSES) File "C:/Python27/httpthreadpool.py", line 51, in runpool Process(target=serve_forever, args=(server,)).start() File "C:\Python27\lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump ForkingPickler(file, protocol).dump(obj) File "C:\Python27\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 548, in save_tuple save(element) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 725, in save_inst save(stuff) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 396, in save_reduce save(cls) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 748, in save_global (obj, module, name)) PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock |
|||
msg216186 - (view) | Author: Martin v. Löwis (loewis) * ![]() |
Date: 2014-04-14 19:26 | |
Ah, Python 2.7. Note that the example is gone in Python 3, as part of the large change in issue8713. |
|||
msg216195 - (view) | Author: jmaki (jmaki) | Date: 2014-04-14 20:09 | |
Upon further investigation, this may be related to: http://bugs.python.org/issue1378 However, it seems the issue is not checked-in to Windows release for 2.x? Regards, John |
|||
msg216241 - (view) | Author: R. David Murray (r.david.murray) * ![]() |
Date: 2014-04-14 22:32 | |
If you look at the source code for 2.7, it is clear that patch has been applied. |
|||
msg216242 - (view) | Author: R. David Murray (r.david.murray) * ![]() |
Date: 2014-04-14 22:35 | |
To clarify, the commit in that issue is in 2.7. So if there is something else that isn't in 2.7, it is a different issue. |
|||
msg216797 - (view) | Author: Terry J. Reedy (terry.reedy) * ![]() |
Date: 2014-04-18 20:03 | |
When quoting from the docs, it is helpful to give a link. https://docs.python.org/2/library/multiprocessing.html#examples That also identifies the version. I verified that the example fails on my 2.7.6 Windows 7 with PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock Since the example has been removed for 3.x, a possible minimal fix would be to say that it does not work on Windows and remove the statement that implies that it does. if sys.platform == 'win32': import multiprocessing.reduction # make sockets pickable/inheritable |
|||
msg226107 - (view) | Author: Terry J. Reedy (terry.reedy) * ![]() |
Date: 2014-08-29 23:42 | |
The same example failing is the subject of #5879. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:01 | admin | set | github: 65403 |
2014-08-29 23:42:32 | terry.reedy | set | status: open -> closed superseder: multiprocessing example "pool of http servers " fails on windows resolution: duplicate messages: + msg226107 |
2014-04-18 20:03:21 | terry.reedy | set | nosy:
+ terry.reedy messages: + msg216797 |
2014-04-14 22:35:35 | r.david.murray | set | messages: + msg216242 |
2014-04-14 22:32:18 | r.david.murray | set | messages: + msg216241 |
2014-04-14 20:09:35 | jmaki | set | messages: + msg216195 |
2014-04-14 19:27:02 | loewis | set | nosy:
- loewis |
2014-04-14 19:26:43 | loewis | set | nosy:
loewis, r.david.murray, docs@python, sbt, wolma, jmaki messages: + msg216186 |
2014-04-14 18:59:36 | loewis | set | title: published examples don't work -> multiprocessing example does not work on Windows |
2014-04-14 17:49:47 | jmaki | set | messages: + msg216135 |
2014-04-13 07:15:40 | loewis | set | nosy:
+ loewis messages: + msg215999 |
2014-04-13 04:11:08 | r.david.murray | set | nosy:
+ docs@python, r.david.murray messages: + msg215994 assignee: docs@python components: + Documentation, - Cross-Build |
2014-04-12 13:39:45 | berker.peksag | set | nosy:
+ sbt |
2014-04-12 13:12:45 | wolma | set | nosy:
+ wolma |
2014-04-12 01:50:40 | jmaki | create |