This is a very simple module for accessing platform-
specific entropy sources.  

On Win32, it uses CryptGenRandom.  Otherwise, it tries 
to use /dev/urandom.  If it can't find that, it raises 
NotImplementedError.

>>> import entropy
>>> entropy.entropy(10)
'\x99/\xbd\x8e\xb0\xfbz/\xf6\xe9'

To compile for Win32, you have to link with "advapi32".  
The /dev/urandom code has not been tested.

Issues:

 - I believe this works with all versions of Windows later 
than Win95.  But I'm not sure.

 - there's overhead in opening/closing the source.  On 
Win32 at least, opening it is slower than reading from it -
 it seems to take around 1/5th of a millisecond (i.e. I 
can make 5000 calls per second).  I think this is okay; 
you can make fewer calls and buffer the results, or seed 
your own PRNG.  However, we could make it more 
object-based, where you open an entropy-source, and 
then repeatedly read from it.

 - It would be nice if there was some attribute that told 
you what source you were using, so you could display 
it, and in case you trust /dev/urandom but not Win32's 
CryptoAPI, for example.

Logged In: YES 
user_id=973611

Just a thought - this might make sense as a function within 
the 'os' module, instead of its own module.

Logged In: YES 
user_id=499

This patch would be tremendously valuable to me.  I've had
to maintain similar code for a couple of projects, and
having this functionality in Python would make my life one
step similar.

A few comments:
- According to MSDN, CryptGenRandom exists on Win98 and
later, and on Win95 OSR2, and on Win95 with IE 3.something
or later.

- It's necessary on some platforms, and for some
applications, to use a device other than /dev/urandom. 
(Some high-security code demands /dev/random; some OpenBSD
people swear by /dev/arandom; and so on.)

- Maybe it would be a good idea to only implement the
windows CryptGenRandom part in C, and implement the Unix
part in Python.  Then you could expose the windows code as
(say) "entopy.win_entropy(nBytes)", the unix part as (say)
"entropy.unix_entropy(nBytes, file='/dev/urandom')", and
have "entropy.entropy(nBytes)" be a cross-platform wrapper.
 This would cut down on the amount of C you need to add;
make it easier to switch entropy devices; provide better
errors when /dev/urandom is unreadable; and provide users
the option to trust only certain entropy sources.

- I believe that you're right not to worry too much about
the performance here.

- According to the MSDN documentation for
CryptAcquireContext, if your first call fails, you're
supposed to retry with a final argument of CRYPT_NEWKEYSET
before you report an error.  I don't know when/if this is
necessary, or whether there are hidden gotchas.

Once again, this patch is a great idea, and I heartily hope
that it gets in!

Logged In: YES 
user_id=973611


Thanks for the comments! - 

> - According to MSDN, CryptGenRandom exists on Win98 and
> later, and on Win95 OSR2, and on Win95 with IE 3.something
> or later.

I'm uploading a new version that fails gracefully on old
Win95s (that's the only change).

> - It's necessary on some platforms, and for some
> applications, to use a device other than /dev/urandom. 
> (Some high-security code demands /dev/random; some 
> OpenBSD people swear by /dev/arandom; and so on.)

My understanding is that /dev/random should only be used in
exceptionally rare cases, if at all.  You can turn up
several posts by David Wagner, a respected cryptographer,
about this.  For example:
http://tinyurl.com/2z2fx

In any case, if you really want /dev/random, or one of the
OpenBSD variants (arandom, srandom, prandom, etc.), it's
easy to do it yourself: open("/dev/random").read().

So I think we should ignore these and stick with
/dev/urandom, since it's the easiest-to-use (non-blocking)
and most portable (unless there are systems that don't offer
it?).

> - Maybe it would be a good idea to only implement the
> windows CryptGenRandom part in C, and implement the Unix
> part in Python.

That's not a bad idea - I sorta think this function should
be placed in the 'os' module, instead of its own module.  So
we could put the /dev/urandom code in 'os.py', and allow
more specific code in, e.g., posixmodule.c to override it.  

We could also add a variable 'os.entropySource' which would
return '/dev/urandom', or 'CryptoAPI', or whatever.

> - According to the MSDN documentation for
> CryptAcquireContext, if your first call fails, you're
> supposed to retry with a final argument of 
> CRYPT_NEWKEYSET before you report an error.

I'm pretty sure using CRYPT_VERIFYCONTEXT eliminates the
need for that:
http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q238187&ID=KB;EN-US;Q238187

Logged In: YES 
user_id=499

Thanks for the reply!

- As for /dev/*random -- yes, I believe you are right, and
/dev/urandom is almost always what you want.  I haven't been
able to find a platform that has one of the others, but
lacks /dev/urandom.

- I can't find a statement on the page you link about using
CRYPT_VERIFYCONTEXT that way, but you may well be right anway.

- One more important issue: It is a bad idea to use stdio
(C's 'fopen', Python's builtin 'open') to read from
/dev/urandom.  Most stdio implementation buffer data; on my
GNU/Linux box, when I call open('/dev/urandom').read(10), my
underlying fread() function sucks 4096 bytes into memory. 
(It does other weird stuff too, including calls to stat64,
mmap, and others.)  This has proved to be a problem in the
past, especially when running on systems with heavy user
process limits.  Instead, it is a better idea to use the
open syscall directly (open in C, os.open in Python).

Logged In: YES 
user_id=973611


> - I can't find a statement on the page you link about using
> CRYPT_VERIFYCONTEXT that way,

Note that retrying on NTE_BAD_KEYSET is only described under
the heading  "Private Key Operations Are Performed", in
which case you need to open/create a private key container.
 But if you use CRYPT_VERIFYCONTEXT it just creates a
temporary context in memory.  More corroboration:
http://tinyurl.com/2ct2o

For awhile I was distributing code without
CRYPT_VERIFYCONTEXT, and a user ran into the NTE_BAD_KEYSET
error.  But CRYPT_VERIFYCONTEXT fixed it, which is how I
stumbled on this...

> - One more important issue: It is a bad idea to use stdio
> (C's 'fopen', Python's builtin 'open') to read from
> /dev/urandom.

Good point.  I've tried to update the code to use syscalls.
 Is there any chance you could test this out, and see
whether the #includes look correct and portable?  I don't
have a UNIX box available.  If it needs fixes, feel free to
upload a new version.

Logged In: YES 
user_id=499

Your lastest version works fine for me (on Redhat Linux 9),
but I still think it would be a better idea to write the
/dev/urandom-ish code in Python, using os.open.  That way,
you could report better error messages if /dev/urandom is
inaccessable, instead of simply claiming that it "wasn't
found."  (Maybe it was found, but we don't have permission
to access it.)

Logged In: YES 
user_id=973611

I'm uploading a 4th version that does this as an
"os.urandom()" function.  The win32 code is in
posixmodule.c, and the /dev/urandom code is in os.py.

I think the name "urandom" is better than "entropy" - to
some people, "entropy" sounds more like /dev/random. 
"urandom" makes it clear that we're trying to provide
/dev/urandom-like functionality across platforms (i.e.: try
to use real entropy, but if you have to stretch it with a
cryptographic PRNG that's okay).

Also, I cleaned up error-handling, so it will always return
OSError (or an OSError subclass, like WindowsError).  

Finally I changed it to open the random source once, instead
of re-opening it every call.  At least on windows this is a
big speedup - from being able to make 5K calls per second it
can now make 250K.  It's probable that people will write
crypto code using this as their sole RNG source, so this is
probably worthwhile.  This introduces a small risk of
leaking file descriptors/handles if the os module is
reloaded, but I think that's okay. 

Logged In: YES 
user_id=499

This version looks exactly like what I need.  I really hope
it gets accepted for 2.4.

(I have no strong opinion on whether the fd/crypto context
should be held open.)

Logged In: YES 
user_id=21627

I also like the patch, but I think a number of changes are
still needed:
- there is no documentation
- there are no test cases
- Don't use Py_BuildValue, use PyString_FromStringAndSize
instead
- Don't stat /dev/urandom when importing os, instead, stat
on first usage, and raise NotImplementedError if no source
of randomness can be found.

As an option, support for PRNG could be added (e.g. by
checking an environment variable), but that can be left to
anybody who actually has and needs PRNG.

As for leaving the fd open: this is ok, I think, if it isn't
opened until first use.

Logged In: YES 
user_id=973611

Hi Martin,

Thanks for looking at this.  I uploaded a new patch (5th
version) with docs, tests, and your suggestions.

Logged In: YES 
user_id=21627

Thanks for the patch. Applied as

libos.tex 1.141
os.py 1.78
test_os.py 1.26
NEWS 1.1117
posixmodule.c 2.322