The OSX manpage for setrlimit includes:

COMPATIBILITY
     setrlimit() now returns with errno set to EINVAL in places that histori-
     cally succeeded.  It no longer accepts "rlim_cur = RLIM_INFINITY" for
     RLIM_NOFILE.  Use "rlim_cur = min(OPEN_MAX, rlim_max)".

It's strange that this is not mentioned in the corresponding manpage on Ubuntu 12.04.

It seems the reason we can't use -1 for RLIMIT_NOFILE is because this requests 'unlimited' (RLIM_INFINITY), while there is always a limit on the number of open files (e.g., OPEN_MAX).

Looking at the code (Modules/resource.c), python is doing the sensible thing.  It seems that resource.setrlimit would have to be special-cased for RLIMIT_NOFILE to work as documented when specifying a "-1", but then python would be diverging from the behaviour of the underlying system call.

I therefore propose that the documentation simply be adjusted to note the observed behaviour (i.e., specifying -1 means 'unlimited', and does not work in general for RLIMIT_NOFILE).  I would be happy to provide a patch if this is desired.