classification
Title: Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer when zlib is not available
Type: behavior Stage: resolved
Components: Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: kristjan.jonsson Nosy List: ezio.melotti, kristjan.jonsson, pitrou, r.david.murray
Priority: normal Keywords:

Created on 2009-07-17 02:20 by ezio.melotti, last changed 2009-12-13 17:24 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
test_docxmlrpc.py kristjan.jonsson, 2009-07-26 21:15 modified file with better error handling
Messages (20)
msg90599 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-17 02:20
I'm working on #6026 and I noticed that the patch for #6267 introduced
an "import gzip" in Lib/xmlrpclib.py in r73638.
gzip tries to import zlib, and if it's not available the import fails.
This led to 3 new tests failures in the trunk: test_xmlrpc,
test_docxmlrpc and test_multiprocessing.
This also mean that the modules xmlrpclib, DocXMLRPCServer and
SimpleXMLRPCServer (and possibly others) cannot be imported anymore if
zlib is not available (they used to work on 2.6).

xmlrpclib should check if the "import gzip" fails and disable the new
gzip-related features (raising an error only when someone tries to use
them).

I don't know if this check can be moved directly on gzip but it seems
unlikely.
msg90668 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-18 09:58
I thought zlib was a builtin module?  Why isn't it available?
msg90678 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-18 14:30
I don't know why it's not installed here, but I didn't remove it
intentionally.
"here" is a Linux machine with Ubuntu 8.04.3 (hardy) LTS.
One reason might be that I just have a limited number of program
installed and zlib is probably a dependency of some popular packages and
hence it's found in most of the machines.

When I compiled Python I got this message, so I think that Python
expects zlib to be already there:
Python build finished, but the necessary bits to build these modules
were not found: _dbm _gdbm _hashlib _sqlite3 _ssl _tkinter bz2 zlib
msg90709 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-07-19 10:30
Kristján, zlib is only built when the required development headers (.h
files for the zlib library) are available.
msg90714 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-19 22:39
submitted revision 74098 and revision 74099 which should fix your issue.
Oh, and revision 74100 and revision 74101 as well (insufficient testing 
on my part, tsk. tsk.)
msg90715 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-19 22:45
Thanks, now all the tests pass.
However I had intermittent failures (with a really useful error message)
on test_docxmlrpc:

$ ./python Lib/test/regrtest.py -v test_docxmlrpc
test_docxmlrpc
test_autolink_dotted_methods (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test that selfdot values are made strong automatically in the ... ok
test_autolinking (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test that the server correctly automatically wraps references to PEPS ... ok
test_invalid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
... ok
test_lambda (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test that lambda functionality stays the same.  The output produced ... ok
test_system_methods (test.test_docxmlrpc.DocXMLRPCHTTPGETServer)
Test the precense of three consecutive system.* methods. ... ok
test_valid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.392s

OK
1 test OK.
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
msg90751 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-21 08:44
I found one problem in test_docxmlrpc.py, a missing "import socket" at 
the top (for the socket.timeout exception handler).
Please add that, and see what happens.

But there seems to be a problem with the error handing in your build.  
probably, sys.stderr is messed up somehow.  You could try putting a 
breakpoint in line 444 of threadmodule.c and get to the bottom of it.

But to short circuit that,  
try adding an "import sys" and then in the finally clause (around line 
54) add "sys.stderr = sys.__stderr__" and see if you get better error 
output.
msg90782 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-21 23:52
sys.stderr looks ok, I tried what you said and I also tried to put some
"assert sys.stderr == sys.__stderr__" here and there and nothing changed.
I commented out the tests one by one and I found out that the first test
alone (test_valid_get_response) is enough to have that error message
(the second alone doesn't seem to show any error).
There's also a comment that says that response.read() is necessary but
nothing changes if I remove that line.
I also noticed that if I put a print at the end of the finally block the
error message doesn't appear anymore.
I didn't tried to debug threadmodule.c.
msg90784 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-22 00:10
Well, I think the only way to move forward with this is to try to 
figure out what the actual error is.  Something is wrong in printing it 
out to stderr, and it would be helpful to understand why it is not 
being output.  Perhaps this is some buffering issue.  You could try to 
redirect stderr to a file, using something like this i the finally 
clause:
sys.stderr = open("/tmp/err.txt", "w", 0)
msg90788 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-22 00:24
Same message in the terminal and nothing in the file.

Can you reproduce it if you run the test several times? Some times I
have to run "./python Lib/test/regrtest.py -v test_docxmlrpc" 15-20 or
even more times before the message appears.
msg90808 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-22 10:00
I can't reproduce this, no.  I only have access to a windows machine 
and you appear to have a custom build (no zlib).  I need your help to 
get the error message, so you need to try harder.  (there are probably 
at least two problems, one causing an exeption and one causing the 
error output to fail)

Try adding something like this into the file, after the except 
socket.timeout clause:
    except:
        import traceback
        traceback.print_exc(100, open(r"c:/tmp/err.txt", "a", 0))
        raise
msg90834 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-22 23:17
I tried that too, with no results.

However using advanced debugging techniques I found something new. I
added 3 prints in the finally, one before serv.server_close(), one
after, and the last at the end, after evt.set().
The first two works, the third one sometimes doesn't (i.e. nothing is
printed), even if no errors are displayed.

I'm not 100% sure but at some point I think I saw the third message
being printed at the beginning, before the first. But now I can't
reproduce it anymore.
msg90835 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-22 23:42
I tried to remove the first two prints (I reverted the file and the only
difference with the original was the print after evt.set() (that here
didn't print anything)) and I got something better:

$ ./python Lib/test/regrtest.py -v test_docxmlrpc
test_docxmlrpc
test_valid_get_response (test.test_docxmlrpc.DocXMLRPCHTTPGETServer) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.063s

OK
1 test OK.
Exception in thread Thread-1 (most likely raised during interpreter
shutdown):
Traceback (most recent call last):
  File "/home/wolf/python-trunk/Lib/threading.py", line 524, in
__bootstrap_inner
  File "/home/wolf/python-trunk/Lib/threading.py", line 477, in run
  File "/home/wolf/python-trunk/Lib/test/test_docxmlrpc.py", line 55, in
server
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
'write'
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
msg90958 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-26 21:02
Ah, so this is an interpreter shutdown issue, it seems.
Something is causing this thread to not exit until the application exits 
and that can cause all sorts of weird race conditions.  I wonder why that 
is happening.  There must be an issue with test_valid_get_response
msg90960 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-07-26 21:15
No, sorry.  You have just introduced the race condition by putting a print 
after the evt.set().  Setting the event causes the main thread to exit 
python and so anything after that line will likely not work.

In fact, this is probably the reason why we didn't get any sensible error 
output, because error of the evt.set in the finally.
Please try the modified version attatched.
msg91916 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-08-24 11:43
Any news on this?
msg92009 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-08-27 17:54
Not yet, the machine I was using to work on this is currently broken and
I couldn't test the new test_docxmlrpc yet. Once I've fixed the machine
and tried it I'll let you know.
msg92560 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-09-13 01:36
I fixed the machine and tried the file that you uploaded, but nothing 
changed. This is what I used (traceback.print_exc() prints to stderr, 
so the traceback should be visible):
$ while ./python Lib/test/regrtest.py -v test_docxmlrpc.py; do :; done 
> /dev/null
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
...
msg92567 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2009-09-13 14:00
Ok, this means that the exception is raised after the finally, when the 
thread is exiting.
Now, at this point the process is exiting and therefore we have trouble 
printing the exception.  (this is probably also the cause of the 
exception in the first place, some cleanup that fails because the 
process is exiting).
Now, the traceback printing code is complex and it could fail in a 
multitude of places, but from the stuff that is output, it would appear 
that printing to sys.stderr is failing.  To find out the error, we need 
to goad python into displaying the exception, even though the process is 
exiting.
On a windows machiine, I would put a DebugBreak() call in t_bootstrap() 
in threadmodule_t and try to find out where the error output code is 
failing.  I don't know if you can do just-in-time debugging on your 
machine.  It would be best, of course, to singlestep through the code at 
line 452 in threadmodule.c.  Then we could pinpoint where the output is 
failing and fix python to make it more resilient at process exit.

Now, lines 452-455 don't produce any output.  For starters, try to 
insert "file=0" before line 452 in threadmodule.c, so that 
PyObject_Print is called instead of PyFile_WriteObject() (which I think 
is failing.)
If that produces output, then we can proceed to hardwire "stderr" into 
the traceback stuff.
msg96338 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-12-13 17:24
Ezio, the original problem this ticket was opened for appears to be
solved, so I'm going to close it.  If you still want to work on the
thread exception issue, please open a new ticket referencing this one.
History
Date User Action Args
2009-12-13 17:24:26r.david.murraysetstatus: open -> closed

versions: + Python 3.2
nosy: + r.david.murray

messages: + msg96338
resolution: fixed
stage: needs patch -> resolved
2009-09-13 14:00:21kristjan.jonssonsetmessages: + msg92567
2009-09-13 01:36:42ezio.melottisetmessages: + msg92560
2009-08-27 17:54:09ezio.melottisetmessages: + msg92009
2009-08-24 11:43:39kristjan.jonssonsetmessages: + msg91916
2009-07-26 21:15:04kristjan.jonssonsetfiles: + test_docxmlrpc.py

messages: + msg90960
2009-07-26 21:02:02kristjan.jonssonsetmessages: + msg90958
2009-07-22 23:42:53ezio.melottisetmessages: + msg90835
2009-07-22 23:17:02ezio.melottisetmessages: + msg90834
2009-07-22 22:37:47ezio.melottiunlinkissue6026 dependencies
2009-07-22 10:00:38kristjan.jonssonsetmessages: + msg90808
2009-07-22 00:24:09ezio.melottisetmessages: + msg90788
2009-07-22 00:10:52kristjan.jonssonsetmessages: + msg90784
2009-07-21 23:52:25ezio.melottisetmessages: + msg90782
2009-07-21 08:44:52kristjan.jonssonsetmessages: + msg90751
2009-07-19 22:45:17ezio.melottisetmessages: + msg90715
2009-07-19 22:39:34kristjan.jonssonsetmessages: + msg90714
2009-07-19 10:30:31pitrousetnosy: + pitrou
messages: + msg90709
2009-07-18 14:30:32ezio.melottisetmessages: + msg90678
2009-07-18 09:58:55kristjan.jonssonsetmessages: + msg90668
2009-07-17 02:40:28ezio.melottilinkissue6026 dependencies
2009-07-17 02:20:04ezio.melotticreate