This patch is a first take at adding a recvall method
to the socket object, to mirror the existence of the
sendall method.

If the MSG_WAITALL flag is available, the recvall
method just calls recv() with that flag.

If it is not available, it uses an internal loop
(during the loop, threads are allowed, so this improves
concurrency).

Having this method makes Python code much simpler;
before you had to test for MSG_WAITALL yourself and
write your own loop in Python if the flag is not there
(on Windows for instance).
(also, having the loop in C improves performance and
concurrency compared to the same loop in Python)

Note: the patch hasn't been tested very well yet.

(code is based on a separate extension module found
here: http://www.it-ernst.de/python/ )

Logged In: YES 
user_id=21627

I like the feature (but see below). The patch is incomplete,
though:
- there are no changes to Doc/lib/libsocket.tex
- there are no changes to Lib/test/test_socket.py

Furthermore, the patch is also wrong: if a later recv call
fails, all data read so far are discarded. I believe this is
different from the WAITALL flag, which I hope will preserve
the data in the socket, for a subsequent recv call. As
keeping the data in the socket seems unimplementable, the
partial data should somehow be returned to the application.

A note on coding style: please omit the spaces after the
opening paren and before the closing in

while ( (bytes_got<total_size) && (n > 0) )

Irmen, do you want to update this patch for the current Python trunk, taking Martin's comments into account?

Sure, I'll give it another go.

I've not done any c-development for quite a while though, so I have to pick up the pieces and see how far I can get. Also, I don't have any compiler for Windows so maybe I'll need someone else to validate the patch on Windows for me, once I've got something together.

Ok I've looked at it again and think I can build an acceptable patch this time. However there are 2 things that I'm not sure of:

1) how to return the partial data to the application if the recv() loop fails before completion. Because the method will probably raise an exception on failure, as usual, it seems to me that the best place to put the partial data is inside the exception object. I can't think of another easy and safe way for the application to retrieve it otherwise. But, how is this achieved in code? I'll be using set_error() to return an error from my sock_recvall function I suppose.

2) the trunk is Python 2.7, should I make a separate patch for 3.x?

Ok I think I've got the code and doc changes ready. I added a recvall and a recvall_into method to the socket module. Any partially received data in case of errors is returned to the application as part of the args for a new exception, socket.partialdataerror.

Still need to work on some unit tests for these new methods.

Just a couple comments:

  * If MSG_WAITALL is defined and a signal interrupts recv, will a string shorter than requested will be returned by sock_recvall?
  * Since MSG_WAITALL is already exposed to Python (when the underlying platform provides it), I wonder if this could all be implemented more simply in pure Python.  Can you elaborate on the motivation to use C?

Someone should do another review when there are unit tests.

Currently if MSG_WAITALL is defined, recvall() just calls recv() internally with the extra flag. Maybe that isn't the smartest thing to do because it duplicates recv's behavior on errors. Which is: release the data and raise an error.
Would it be nicer to have recvall() release the data and raise an error, or to let it return the partial data? Either way, I think the behavior should be the same regardless of MSG_WAITALL being available. This is not yet the case.

Why C: this started out by making the (very) old patch that I once wrote for socketmodule.c up to date with the current codebase, and taking Martin's comments into account.
The old patch was small and straightforward. Unfortunately the new one turned out bigger and more complex than I thought. For instance I'm not particularly happy with the way recvall returns the partial data on fail. It uses a new exception for that but the code has some trickery to replace the socket.error exception that is initially raised. I'm not sure if my code is the right way to do this, it needs some review. I do think that putting it into the exception object is the only safe way of returning it to the application, unless the semantics on error are changed as mentioned above. Maybe it could be made simpler then.
In any case, it probably is a good idea to see if a pure python solution (perhaps just some additions to Lib/socket.py?) would be better. Will put some effort into this.

@Irmen if you do proceed with this it should be against the py3k trunk.