This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Ability to get random bytes from random.Random (as with os.urandom)
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: ideasman42, r.david.murray, rhettinger, serhiy.storchaka, tim.peters
Priority: normal Keywords:

Created on 2016-05-23 22:37 by ideasman42, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (7)
msg266202 - (view) Author: Campbell Barton (ideasman42) * Date: 2016-05-23 22:37
Currently, getting random bits can be done with os.urandom,

However recently I was faced with having to make a test (with reproducible behavior), so I needed to replace os.urandom with a random generator that took a seed value.

It turns out there are 3 ways (at least) to handle this, but none are really that great.

- Create a list, fill with Random.randint(0, 255), then convert to bytes with bytes.join.
- Call Random.getrandbits(), then int.to_bytes.
- Override urandom at a system level (possible on Linux [0]).

While these work, they are either slow (creating a list), non trivial (OS-level) or limited - Random.getrandbits hits internal limits of an int, and accidentally using the int (running repr on it for example, locks up Python), currently CPython will fail to create values above 256mb since bits is limited to INT_MAX [1].

In short, to keep tests running fast, and without messing about and accounting for internal limits of CPython, there isn't a convenient way to get random bits in Python.

Since bits are a primitive type and since its already supported by os.urandom, I think its reasonable the random module could support returning random bits.

If this is accepted, I can provide a patch for this, I'm just checking to know if the functionality would be accepted.

Suggest to call random.randbits(n).

----

[0]: http://stackoverflow.com/questions/26053875/bypass-dev-urandomrandom-for-testing
[1]: http://bugs.python.org/issue27072
msg266203 - (view) Author: Campbell Barton (ideasman42) * Date: 2016-05-23 22:47
Correction, meant to write: random.randbytes(n).
msg266204 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-05-23 23:07
It seems reasonable to me, but I'm not the one to sign off on something like this.  I've nosied someone who will have an opinion :)

Would doing:

    mybytes = bytes(random.randint(0, 255) for i in range(size))

be faster enough to be workable?  I suppose that might build a list under the covers...maybe preallocating a bytearray and filling it with ints would work better.
msg266206 - (view) Author: Campbell Barton (ideasman42) * Date: 2016-05-23 23:21
@r.david.murray, yes, this avoids list creation, but is still quite slow.

To create 1mb of data, you can compare the following:

    python -m timeit -s 'from os import urandom' 'print(len(urandom(1000000)))'

    python -m timeit -s 'from random import randint' 'print(len(bytes(randint(0, 255) for i in range(1000000))))'

On my system `os.urandom` is 0.04sec, using randint takes 2.24sec (approx 50x slower).
msg266236 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-24 06:42
$ ./python -m timeit -s 'from os import urandom' 'urandom(1000000)'
10 loops, best of 3: 132 msec per loop
$ ./python -m timeit -s 'from random import getrandbits' 'n = 1000000; getrandbits(n*8).to_bytes(n, "little")'
100 loops, best of 3: 14.1 msec per loop

With the purposed method this can be up to 3.4 times faster, but as I said in issue27072, I don't think API growing is worth the benefit.
msg266277 - (view) Author: Campbell Barton (ideasman42) * Date: 2016-05-24 21:01
@serhiy.storchaka, while a properly working function that uses getrandbits isn't so complex, its not trivial either.

It needs to create smaller chunks and join them (also check zero size case which raises an error if passed).

eg:

```
    def urandom_from_random(rng, length):
        if length == 0:
            return b''
    
        import sys
        chunk_size = 65535
        chunks = []
        while length >= chunk_size:
            chunks.append(rng.getrandbits(chunk_size * 8).to_bytes(chunk_size, sys.byteorder))
            length -= chunk_size
        if length:
            chunks.append(rng.getrandbits(length * 8).to_bytes(length, sys.byteorder))
        result = b''.join(chunks)
        return result
```
msg266367 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-05-25 14:55
Sorry Campbell, I concur with Serhiy that this isn't worth growing the API.
History
Date User Action Args
2022-04-11 14:58:31adminsetgithub: 71283
2016-05-25 14:55:09rhettingersetstatus: open -> closed
resolution: rejected
messages: + msg266367
2016-05-24 21:01:49ideasman42setmessages: + msg266277
2016-05-24 06:57:52rhettingersetnosy: + tim.peters
2016-05-24 06:42:10serhiy.storchakasetassignee: rhettinger

messages: + msg266236
nosy: + serhiy.storchaka
2016-05-23 23:21:41ideasman42setmessages: + msg266206
2016-05-23 23:07:40r.david.murraysetnosy: + rhettinger, r.david.murray

messages: + msg266204
title: Ability to get random bits from random.Random (as with os.urandom) -> Ability to get random bytes from random.Random (as with os.urandom)
2016-05-23 22:47:17ideasman42setmessages: + msg266203
2016-05-23 22:37:26ideasman42create