classification
Title: Document that the random module doesn't support fork
Type: Stage:
Components: Documentation Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: docs@python, haypo, pitrou, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-04-12 09:24 by haypo, last changed 2017-06-13 11:12 by serhiy.storchaka.

Messages (13)
msg291534 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-12 09:24
When reviewing the issue #30030, it reminded me that the random module "doesn't support fork": after fork, the parent and the child produce the same "random" number sequence.

I suggest to add a quick note about that: not a warning, just a note, as a reminder.

There is an exception: SystemRandom produces a different sequence after fork, since it uses os.urandom() (which runs in the kernel).

I am tempted to propose a solution in the note like using SystemRandom, but I'm not sure that it's a good idea. Some users may misunderstood the note and always use SystemRandom where random.Random is just fine for their needs.

Proposed note:

The random module doesn't support fork: the parent and the child process will produce the same number sequence.

Proposed solution:

If your code uses os.fork(), a workaround is to check if os.getpid() changes and in that case, create a new Random instance or reseed the RNG in the child process.

--

The tempfile module reminds the pid and instanciates a new RNG on fork.

Another option is to not add a note, but implement the workaround directly into the random module. But I don't think that it's worth it. Forking is a rare usecase, and calling os.getpid() may slowdown the random module. (I don't recall if os.getpid() requires a syscall or not, I'm quite sure that it's optimized at least on Linux to avoid a real syscall.)
msg291537 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-04-12 09:30
I'm not sure that's necessary.  fork() is a low-level primitive, people can/should use multiprocessing.Process instead, which does re-seed the PRNG.
msg291540 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-12 10:02
> I'm not sure that's necessary.

Do you mean that adding a note is not necessary? Or proposing a
solution in the note?
msg291541 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-04-12 10:04
I mean mentioning fork(), which people usually don't call directly, is not necessary, so, indeed, a note isn't necessary.
msg291544 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-12 10:36
I think it is worth documenting.

Even if multiprocessing reseeds the module-global random generator, it doesn't reseed other instances of Random.
msg291773 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-04-17 02:04
Several thoughts:

* AFAICT, in entire the history of this module, no user has ever reported this as being a source of confusion, so I don't think there is a real documentation issue here. 

* Regarding, "after fork, the parent and the child produce the same "random" number sequence", this is the expected behavior of a PRNG.  If it did something thing different, THAT would be a bug.

* There random module is likely the wrong place for a note.   It is more properly a topic about forking itself.  Perhaps there is room for a FAQ entry about forking elaborating on the broad range of state that is shared across forks (lock and file descriptors, etc).

* For those who need it, the API already supports reseeding and a way to make new instances of Random.  Both of those are the standard ways of doing it for people who need independent generators in different threads.
msg291788 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-17 07:36
os.fork() documentation contains a warning that refers to ssl documentation where the use of OpenSSL’s internal random number generator in forked processes is documented more detailed. I think the same should be added for the random module.
msg291797 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-04-17 11:46
> os.fork() documentation contains a warning that refers to ssl
> documentation where the use of OpenSSL’s internal random number
> generator in forked processes is documented more detailed. 
> I think the same should be added for the random module.

+1 It is reasonable to extend the docs for os.fork().
msg295889 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-13 10:33
Victor, this issue should be obsolete now that the global Random instance is reseeded after fork().
msg295894 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-13 10:59
Even if the global Random instance is reseeded after fork(), other instances are not reseeded.
msg295896 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-13 11:07
At least, it would be nice to document it for Python 3.6 and older.
msg295897 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-13 11:07
This is true, but it's by design.  If you want a specific random sequence (which is the primary use case for specific instances), you don't want fork() to mess with the random sequence.
msg295898 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-13 11:12
Yes, this is a feature, not a bug. But I think it is worth be explicitly documented.
History
Date User Action Args
2017-06-13 11:12:24serhiy.storchakasetmessages: + msg295898
2017-06-13 11:07:35pitrousetmessages: + msg295897
2017-06-13 11:07:29hayposetmessages: + msg295896
2017-06-13 10:59:54serhiy.storchakasetmessages: + msg295894
2017-06-13 10:33:32pitrousetmessages: + msg295889
2017-04-17 11:46:37rhettingersetmessages: + msg291797
2017-04-17 07:36:49serhiy.storchakasetmessages: + msg291788
2017-04-17 02:04:36rhettingersetmessages: + msg291773
2017-04-17 01:41:53rhettingersetassignee: docs@python -> rhettinger
2017-04-12 10:36:32serhiy.storchakasetmessages: + msg291544
2017-04-12 10:04:48pitrousetmessages: + msg291541
2017-04-12 10:02:53hayposetmessages: + msg291540
2017-04-12 09:30:03pitrousetnosy: + pitrou
messages: + msg291537
2017-04-12 09:26:23hayposetnosy: + rhettinger, docs@python, serhiy.storchaka

components: + Documentation
assignee: docs@python
2017-04-12 09:24:49haypocreate