Issue6499
Created on 2009-07-17 02:20 by ezio.melotti, last changed 2009-12-13 17:24 by r.david.murray. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
test_docxmlrpc.py | kristjan.jonsson, 2009-07-26 21:15 | modified file with better error handling |
Messages (20) | |||
---|---|---|---|
msg90599 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-17 02:20 | |
I'm working on #6026 and I noticed that the patch for #6267 introduced an "import gzip" in Lib/xmlrpclib.py in r73638. gzip tries to import zlib, and if it's not available the import fails. This led to 3 new tests failures in the trunk: test_xmlrpc, test_docxmlrpc and test_multiprocessing. This also mean that the modules xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer (and possibly others) cannot be imported anymore if zlib is not available (they used to work on 2.6). xmlrpclib should check if the "import gzip" fails and disable the new gzip-related features (raising an error only when someone tries to use them). I don't know if this check can be moved directly on gzip but it seems unlikely. |
|||
msg90668 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-18 09:58 | |
I thought zlib was a builtin module? Why isn't it available? |
|||
msg90678 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-18 14:30 | |
I don't know why it's not installed here, but I didn't remove it intentionally. "here" is a Linux machine with Ubuntu 8.04.3 (hardy) LTS. One reason might be that I just have a limited number of program installed and zlib is probably a dependency of some popular packages and hence it's found in most of the machines. When I compiled Python I got this message, so I think that Python expects zlib to be already there: Python build finished, but the necessary bits to build these modules were not found: _dbm _gdbm _hashlib _sqlite3 _ssl _tkinter bz2 zlib |
|||
msg90709 - (view) | Author: Antoine Pitrou (pitrou) * ![]() |
Date: 2009-07-19 10:30 | |
Kristján, zlib is only built when the required development headers (.h files for the zlib library) are available. |
|||
msg90714 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-19 22:39 | |
submitted revision 74098 and revision 74099 which should fix your issue. Oh, and revision 74100 and revision 74101 as well (insufficient testing on my part, tsk. tsk.) |
|||
msg90715 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-19 22:45 | |
Thanks, now all the tests pass. However I had intermittent failures (with a really useful error message) on test_docxmlrpc: $ ./python Lib/test/regrtest.py -v test_docxmlrpc test_docxmlrpc test_autolink_dotted_methods (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) Test that selfdot values are made strong automatically in the ... ok test_autolinking (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) Test that the server correctly automatically wraps references to PEPS ... ok test_invalid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok test_lambda (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) Test that lambda functionality stays the same. The output produced ... ok test_system_methods (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) Test the precense of three consecutive system.* methods. ... ok test_valid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok ---------------------------------------------------------------------- Ran 6 tests in 0.392s OK 1 test OK. Unhandled exception in thread started by Error in sys.excepthook: Original exception was: |
|||
msg90751 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-21 08:44 | |
I found one problem in test_docxmlrpc.py, a missing "import socket" at the top (for the socket.timeout exception handler). Please add that, and see what happens. But there seems to be a problem with the error handing in your build. probably, sys.stderr is messed up somehow. You could try putting a breakpoint in line 444 of threadmodule.c and get to the bottom of it. But to short circuit that, try adding an "import sys" and then in the finally clause (around line 54) add "sys.stderr = sys.__stderr__" and see if you get better error output. |
|||
msg90782 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-21 23:52 | |
sys.stderr looks ok, I tried what you said and I also tried to put some "assert sys.stderr == sys.__stderr__" here and there and nothing changed. I commented out the tests one by one and I found out that the first test alone (test_valid_get_response) is enough to have that error message (the second alone doesn't seem to show any error). There's also a comment that says that response.read() is necessary but nothing changes if I remove that line. I also noticed that if I put a print at the end of the finally block the error message doesn't appear anymore. I didn't tried to debug threadmodule.c. |
|||
msg90784 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-22 00:10 | |
Well, I think the only way to move forward with this is to try to figure out what the actual error is. Something is wrong in printing it out to stderr, and it would be helpful to understand why it is not being output. Perhaps this is some buffering issue. You could try to redirect stderr to a file, using something like this i the finally clause: sys.stderr = open("/tmp/err.txt", "w", 0) |
|||
msg90788 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-22 00:24 | |
Same message in the terminal and nothing in the file. Can you reproduce it if you run the test several times? Some times I have to run "./python Lib/test/regrtest.py -v test_docxmlrpc" 15-20 or even more times before the message appears. |
|||
msg90808 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-22 10:00 | |
I can't reproduce this, no. I only have access to a windows machine and you appear to have a custom build (no zlib). I need your help to get the error message, so you need to try harder. (there are probably at least two problems, one causing an exeption and one causing the error output to fail) Try adding something like this into the file, after the except socket.timeout clause: except: import traceback traceback.print_exc(100, open(r"c:/tmp/err.txt", "a", 0)) raise |
|||
msg90834 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-22 23:17 | |
I tried that too, with no results. However using advanced debugging techniques I found something new. I added 3 prints in the finally, one before serv.server_close(), one after, and the last at the end, after evt.set(). The first two works, the third one sometimes doesn't (i.e. nothing is printed), even if no errors are displayed. I'm not 100% sure but at some point I think I saw the third message being printed at the beginning, before the first. But now I can't reproduce it anymore. |
|||
msg90835 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-07-22 23:42 | |
I tried to remove the first two prints (I reverted the file and the only difference with the original was the print after evt.set() (that here didn't print anything)) and I got something better: $ ./python Lib/test/regrtest.py -v test_docxmlrpc test_docxmlrpc test_valid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.063s OK 1 test OK. Exception in thread Thread-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/home/wolf/python-trunk/Lib/threading.py", line 524, in __bootstrap_inner File "/home/wolf/python-trunk/Lib/threading.py", line 477, in run File "/home/wolf/python-trunk/Lib/test/test_docxmlrpc.py", line 55, in server <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'write' Unhandled exception in thread started by Error in sys.excepthook: Original exception was: |
|||
msg90958 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-26 21:02 | |
Ah, so this is an interpreter shutdown issue, it seems. Something is causing this thread to not exit until the application exits and that can cause all sorts of weird race conditions. I wonder why that is happening. There must be an issue with test_valid_get_response |
|||
msg90960 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-07-26 21:15 | |
No, sorry. You have just introduced the race condition by putting a print after the evt.set(). Setting the event causes the main thread to exit python and so anything after that line will likely not work. In fact, this is probably the reason why we didn't get any sensible error output, because error of the evt.set in the finally. Please try the modified version attatched. |
|||
msg91916 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-08-24 11:43 | |
Any news on this? |
|||
msg92009 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-08-27 17:54 | |
Not yet, the machine I was using to work on this is currently broken and I couldn't test the new test_docxmlrpc yet. Once I've fixed the machine and tried it I'll let you know. |
|||
msg92560 - (view) | Author: Ezio Melotti (ezio.melotti) * ![]() |
Date: 2009-09-13 01:36 | |
I fixed the machine and tried the file that you uploaded, but nothing changed. This is what I used (traceback.print_exc() prints to stderr, so the traceback should be visible): $ while ./python Lib/test/regrtest.py -v test_docxmlrpc.py; do :; done > /dev/null Unhandled exception in thread started by Error in sys.excepthook: Original exception was: Unhandled exception in thread started by Error in sys.excepthook: Original exception was: ... |
|||
msg92567 - (view) | Author: Kristján Valur Jónsson (kristjan.jonsson) * ![]() |
Date: 2009-09-13 14:00 | |
Ok, this means that the exception is raised after the finally, when the thread is exiting. Now, at this point the process is exiting and therefore we have trouble printing the exception. (this is probably also the cause of the exception in the first place, some cleanup that fails because the process is exiting). Now, the traceback printing code is complex and it could fail in a multitude of places, but from the stuff that is output, it would appear that printing to sys.stderr is failing. To find out the error, we need to goad python into displaying the exception, even though the process is exiting. On a windows machiine, I would put a DebugBreak() call in t_bootstrap() in threadmodule_t and try to find out where the error output code is failing. I don't know if you can do just-in-time debugging on your machine. It would be best, of course, to singlestep through the code at line 452 in threadmodule.c. Then we could pinpoint where the output is failing and fix python to make it more resilient at process exit. Now, lines 452-455 don't produce any output. For starters, try to insert "file=0" before line 452 in threadmodule.c, so that PyObject_Print is called instead of PyFile_WriteObject() (which I think is failing.) If that produces output, then we can proceed to hardwire "stderr" into the traceback stuff. |
|||
msg96338 - (view) | Author: R. David Murray (r.david.murray) * ![]() |
Date: 2009-12-13 17:24 | |
Ezio, the original problem this ticket was opened for appears to be solved, so I'm going to close it. If you still want to work on the thread exception issue, please open a new ticket referencing this one. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2009-12-13 17:24:26 | r.david.murray | set | status: open -> closed versions: + Python 3.2 nosy: + r.david.murray messages: + msg96338 resolution: fixed stage: needs patch -> resolved |
2009-09-13 14:00:21 | kristjan.jonsson | set | messages: + msg92567 |
2009-09-13 01:36:42 | ezio.melotti | set | messages: + msg92560 |
2009-08-27 17:54:09 | ezio.melotti | set | messages: + msg92009 |
2009-08-24 11:43:39 | kristjan.jonsson | set | messages: + msg91916 |
2009-07-26 21:15:04 | kristjan.jonsson | set | files:
+ test_docxmlrpc.py messages: + msg90960 |
2009-07-26 21:02:02 | kristjan.jonsson | set | messages: + msg90958 |
2009-07-22 23:42:53 | ezio.melotti | set | messages: + msg90835 |
2009-07-22 23:17:02 | ezio.melotti | set | messages: + msg90834 |
2009-07-22 22:37:47 | ezio.melotti | unlink | issue6026 dependencies |
2009-07-22 10:00:38 | kristjan.jonsson | set | messages: + msg90808 |
2009-07-22 00:24:09 | ezio.melotti | set | messages: + msg90788 |
2009-07-22 00:10:52 | kristjan.jonsson | set | messages: + msg90784 |
2009-07-21 23:52:25 | ezio.melotti | set | messages: + msg90782 |
2009-07-21 08:44:52 | kristjan.jonsson | set | messages: + msg90751 |
2009-07-19 22:45:17 | ezio.melotti | set | messages: + msg90715 |
2009-07-19 22:39:34 | kristjan.jonsson | set | messages: + msg90714 |
2009-07-19 10:30:31 | pitrou | set | nosy:
+ pitrou messages: + msg90709 |
2009-07-18 14:30:32 | ezio.melotti | set | messages: + msg90678 |
2009-07-18 09:58:55 | kristjan.jonsson | set | messages: + msg90668 |
2009-07-17 02:40:28 | ezio.melotti | link | issue6026 dependencies |
2009-07-17 02:20:04 | ezio.melotti | create |