> I don't understand why you must put the socket in 
> non-blocking mode for sendfile().

I did that mainly because I'm using select() / poll() and it seems kind of "natural" to set the socket in non-blocking mode (proftpd does the same).
I'm not sure whether it actually makes any difference though (on Linux it works either way, blocking or not).
I'm removing the non-blocking mode in the new attached patch. We can take take a look at the buildbots later and see how they behave.

> Also I think the support code (_use_send() / _use_sendfile()) should be factored out somewhere.

Agreed and turned them into methods.

> There could even be a socket.sendfile() method with the appropriate fallbacks?

Perhaps. The only thing which is not clear is how to deal with blocking vs. non-blocking sockets.
Also, Windows should also be covered and expose TransmitFile. 
It's probably better to discuss this elsewhere.