Please do get rid of the asserts.

Regarding signed sizes, it's a historic accident (copied from Unix
system calls which tended to use plain ints or longs in the olden days),
but it may not be impossible to fix -- we've fixed other things
throughout the code (e.g. switching from int to Py_ssize_t).  It'll be a
monumental task though that's probably best tackled post-3.0.