classification
Title: Allow registering at-fork handlers
Type: enhancement Stage: resolved
Components: Extension Modules, Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: Arfrever, DLitz, aliles, amaury.forgeotdarc, asvetlov, christian.heimes, emptysquare, georg.brandl, grahamd, gregory.p.smith, haypo, ionelmc, jcea, lemburg, neologix, pitrou, rpcope1, sbt, serhiy.storchaka, socketpair, twouters, xupeng, yselivanov
Priority: normal Keywords: needs review, patch

Created on 2012-11-18 15:20 by christian.heimes, last changed 2017-05-29 23:23 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
pure-python-atfork.patch sbt, 2012-11-19 22:43 review
Pull Requests
URL Status Linked Edit
PR 1715 merged pitrou, 2017-05-22 12:10
PR 1834 merged gregory.p.smith, 2017-05-27 18:40
PR 1841 merged pitrou, 2017-05-28 08:48
PR 1843 merged pitrou, 2017-05-28 09:50
Messages (56)
msg175878 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-11-18 15:20
I propose the addition of an 'afterfork' module. The module shall fulfill a similar task as the 'atexit' module except that it handles process forks instead of process shutdown.

The 'afterfork' module shall allow libraries to register callbacks that are executed on fork() inside the child process and as soon as possible. Python already has a function that must be called by C code: PyOS_AfterFork(). The 'afterfork' callbacks are called as the last step in PyOS_AfterFork().

Use case example:
The tempfile module has a specialized RNG that re-initialized the RNG after fork() by comparing os.getpid() to an instance variable every time the RNG is accessed. The check can be replaced with an afterfork callback.

Open questions:
How should the afterfork() module handle exceptions that are raised by callbacks?

Implementation:
I'm going to use as much code from atexitmodule.c as possible. I'm going to copy common code to a template file and include the template from atexitmodule.c and afterforkmodule.c with some preprocessor tricks.
msg175892 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-11-18 17:34
pthread_atfork() allows the registering of three types of callbacks:

1) prepare callbacks which are called before the fork,
2) parent callbacks which are called in the parent after the fork
3) child callbacks which are called in the child after the fork.

I think all three should be supported.

I also think that a recursive "fork lock" should be introduced which is held during the fork.  This can be acquired around critical sections during which forks must not occur.

This is more or less a duplicate of #6923.  See also #6721.
msg175967 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-11-19 20:55
Thanks Richard!

My first reaction was YAGNI but after I read the two tickets I now understand the need for three different hooks. I suggest that we implement our own hooks like the http://linux.die.net/man/3/pthread_atfork function, especially the order of function calls:

The parent and child fork handlers shall be called in the order in which they were established by calls to pthread_atfork().  The prepare fork handlers shall be called in the opposite order.

I like to focus on three hooks + the Python API and leave the usage of the hooks to other developers.

Proposal:
* Introduce a new module called atfork (Modules/atforkmodule.c) that is build into the core.
* Move PyOS_AfterFork to Modules/atforkmodule.c.
* Add PyOS_BeforeFork() (or PyOS_PrepareFork() ?) and PyOS_AfterForkParent() 
* call the two new methods around the calls to fork() in the stdlib.

I'm not yet sure how to implement the Python API. I could either implement six methods:

  atfork.register_before_fork(callable, *args, **kwargs)
  atfork.register_after_fork_child(callable, *args, **kwargs)
  atfork.register_after_fork_parent(callable, *args, **kwargs)
  atfork.unregister_before_fork(callable)
  atfork.unregister_after_fork_child(callable)
  atfork.unregister_after_fork_parent(callable)

or two:

  atfork.register(prepare=None, parent=None, child=None, *args, **kwargs)
  atfork.unregister(prepare=None, parent=None, child=None)
msg175972 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-11-19 22:43
Note that Gregory P. Smith has written

    http://code.google.com/p/python-atfork/

I also started a pure python patch but did not get round it posting it.  (It also implements the fork lock idea.)  I'll attach it here.

How do you intend to handle the propagation of exceptions?  I decided that after

    atfork.atfork(prepare1, parent1, child1)
    atfork.atfork(prepare2, parent2, child2)
    ...
    atfork.atfork(prepareN, parentN, childN)

calling "pid = os.fork()" should be equivalent to

    pid = None
    prepareN()
    try:
        ...
            prepare2()
            try:
                prepare1()
                try:
                    pid = posix.fork()
                finally:
                    parent1() if pid != 0 else child1()
            finally:
                parent2() if pid != 0 else child2()
        ...
    finally:
        parentN() if pid != 0 else childN()
msg175973 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-11-19 23:59
I would not allow exceptions to propagate. No caller is expecting them.
msg175974 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-11-20 00:13
pthread_atfork() cannot be used to implement this. Another non-python
thread started by a C extension module or the C application that is
embedding Python within it is always free to call fork() on its own with
zero knowledge that Python even exists at all. It's guaranteed that fork
will be called while the Python GIL is held in this situation which would
cause any pre-fork thing registered by Python to deadlock.

At best, this can be implemented manually as we do with some of the before
and after fork stuff today but it must come with the caveat warning that it
cannot guarantee that these things are actually called before and after
fork() other than direct os.fork() calls from Python code or extremely
Python aware C extension modules that may call fork() (very rare, most C &
C++ libraries an extension module may be using assume that they've got the
run of the house).  ie: this problem is unsolvable unless you control 100%
of the code being used by your entire user application.

On Mon, Nov 19, 2012 at 3:59 PM, Gregory P. Smith <report@bugs.python.org>wrote:

>
> Gregory P. Smith added the comment:
>
> I would not allow exceptions to propagate. No caller is expecting them.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue16500>
> _______________________________________
>
msg175975 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-11-20 00:52
Meh! Exception handling takes all the fun of the API and is going to make it MUCH more complicated. pthread_atfork() ignores error handling for a good reason. It's going to be hard to get it right. :/

IFF we are going to walk the hard and rocky road of exception handling, then we are going to need at least four hooks and a register function that takres four callables as arguments: register(prepare, error, parent, child). Each prepare() call pushes an error handling onto a stack. In case of an exception in a prepare handler, the error stack is popped until all error handlers are called. This approach allows a prepare handler to actually prevent a fork() call from succeeding.

The parent and child hooks are always called no matter what. Exception are recorded and a warning is emitted when at least one hook fails. We might raise an exception but it has to be a special exception that ships information if fork() has succeeded, if the code runs in child or parent and about the child's PID.

I fear it's going to be *really* hard to get everything right.

Gregory made a good point, too. We can rely on pthread_atfork() as we are unable to predict how third party code is using fork(): "Take cover, dead locks ahead!" :) A cooperative design of the C API with three function is my preferred way, too. PyOS_AfterForkParent() should take an argument to signal a failed fork() call.
msg175980 - (view) Author: Amaury Forgeot d'Arc (Amaury.Forgeot.d'Arc) * Date: 2012-11-20 09:11
2012/11/20 Christian Heimes <report@bugs.python.org>

> IFF we are going to walk the hard and rocky road of exception handling,
> then we are going to need at least four hooks and a register function that
> takres four callables as arguments: register(prepare, error, parent,
> child). Each prepare() call pushes an error handling onto a stack. In case
> of an exception in a prepare handler, the error stack is popped until all
> error handlers are called. This approach allows a prepare handler to
> actually prevent a fork() call from succeeding.
>

FWIW, PyPy already has a notion of fork hooks:
https://bitbucket.org/pypy/pypy/src/b4e4017909bac6c102fbc883ac8d2e42fa41553b/pypy/module/posix/interp_posix.py?at=default#cl-682

Various subsystems (threads cleanup, import lock, threading.local...)
register their hook functions.

You may want to experiment from there :-)
A new "atfork" module would be easy to implement.
msg175997 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-11-20 15:44
> IFF we are going to walk the hard and rocky road of exception handling,
> then we are going to need at least four hooks and a register function that
> takres four callables as arguments: register(prepare, error, parent,
> child). Each prepare() call pushes an error handling onto a stack. In case
> of an exception in a prepare handler, the error stack is popped until all
> error handlers are called. This approach allows a prepare handler to
> actually prevent a fork() call from succeeding.

I think there are two main options if a prepare callback fails:

1) The fork should not occur and the exception should be raised
2) The fork should occur and the exception should be only be printed

I favour option 1 since, if they want, users can always wrap their prepare callbacks with

  try:
    ...
  except:
    sys.excepthook(*sys.exc_info())

With option 1 I don't see why error callbacks are necessary.  Just unwind the stack of imaginary try...finally... clauses and let any exceptions propagate out using exception chaining if necessary.  This is what pure-python-atfork.patch does.  Note, however, that if the fork succeeds then any subsequent exception is only printed.
msg176002 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-11-20 16:20
Amaury:
PyPy doesn't handle exceptions in hooks. Is there a reason why PyPy goes for the simplistic approach?

Richard:
An error callback has the benefit that the API can notice the hooks that some error has occurred. We may not need it, though.

I can think of six exception scenarios that must be handled:

(1) exception in a prepare hook -> don't call the remaining prepare hooks, run all related parent hooks in FILO order, prevent fork() call
(2) exception in parent hook during the handling of (1) -> print exception, continue with next parent hook
(3) exception in fork() call -> run parent hooks in FILO order
(4) exception in parent hook during the handling of (3) -> print exception, continue with next parent hook
(5) exception in parent hook when fork() has succeeded -> print exception, continue with next parent hook
(6) exception in child hook when fork() has succeeded  -> print exception, continue with next child hook

Do you agree?
msg176004 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-11-20 16:29
> PyPy doesn't handle exceptions in hooks.
> Is there a reason why PyPy goes for the simplistic approach?

Probably because nobody thought about it.
At the moment, there is only one 'before', one 'parent' hook (so the FILO order is simple), and three 'child' hooks.
And if the _PyImport_ReleaseLock call fails, you'd better not ignore the error...
msg176019 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-11-20 19:33
I think you are solving a non-problem if you want to expose exceptions from
such hooks. Nobody needs it.
msg176020 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-20 19:49
> I think you are solving a non-problem if you want to expose exceptions from
> such hooks. Nobody needs it.

Agreed.
msg176022 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-11-20 20:04
Your suggestion is that the hooks are called as:

for hook in hooks:
    try:
        hook()
    except:
        try:
            sys.excepthook(*sys.exc_info())
        except:
            pass

That makes the implementation much easier. :)
msg179838 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-01-12 23:37
"The tempfile module has a specialized RNG that re-initialized the RNG after fork() by comparing os.getpid() to an instance variable every time the RNG is accessed. The check can be replaced with an afterfork callback."

By the way, OpenSSL expects that its PRNG is reseed somehow (call RNG_add) after a fork. I wrote a patch for OpenSSL, but I don't remember if I sent it to OpenSSL.
https://bitbucket.org/haypo/hasard/src/4a1be69a47eb1b2ec7ca95a341d4ca953a77f8c6/patches/openssl_rand_fork.patch?at=default

Reseeding tempfile PRNG is useless (but spend CPU/memory/hang until we have enough entropy?) if the tempfile is not used after fork. I like the current approach.

--

I'm not saying that a new atfork module would not help, just that the specific case of tempfile should be discussed :-) I like the idea of a generic module to call code after fork.
msg179888 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-13 19:21
Might make sense to put this in atexit.atfork() to avoid small-module inflation?
msg179927 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-01-14 09:13
> Might make sense to put this in atexit.atfork() to avoid small-module inflation?

It sounds strange to mix "at exit" and "at fork" in the same module.
Both are very different.

2013/1/13 Arfrever Frehtes Taifersar Arahesis <report@bugs.python.org>:
>
> Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA@GMail.Com>:
>
>
> ----------
> nosy: +Arfrever
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue16500>
> _______________________________________
msg179945 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-01-14 15:49
On 13.01.2013 00:37, STINNER Victor wrote:
> By the way, OpenSSL expects that its PRNG is reseed somehow (call RNG_add) after a fork. I wrote a patch for OpenSSL, but I don't remember if I sent it to OpenSSL.
> https://bitbucket.org/haypo/hasard/src/4a1be69a47eb1b2ec7ca95a341d4ca953a77f8c6/patches/openssl_rand_fork.patch?at=default

Apparently not, and according to this thread, they don't think
this is an OpenSSL problem to solve:

http://openssl.6102.n7.nabble.com/recycled-pids-causes-PRNG-to-repeat-td41669.html

Note that you don't have to reseed the RNG just make sure that the
two forks use different sequences. Simply adding some extra data
in each process will suffice, e.g. by adding the PID of the new process
to the RNG pool. This is certainly doable without any major CPU
overhead :-)
msg179949 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-14 16:06
> It sounds strange to mix "at exit" and "at fork" in the same module.
> Both are very different.

That's true. The sys module would probably be the right place for both functionalities.
msg200767 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 12:30
Richard, do you have time to get your patch ready for 3.4?
msg200774 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-10-21 13:02
> Richard, do you have time to get your patch ready for 3.4?

Yes.  But we don't seem to have concensus on how to handle exceptions.  The main question is whether a failed prepare callback should prevent the fork from happenning, or just be printed.
msg200777 - (view) Author: Dwayne Litzenberger (DLitz) Date: 2013-10-21 13:42
> The main question is whether a failed prepare callback should prevent the fork from happenning

Yes, I think an exception should prevent the fork from happening.

It's fail-safe for the PRNG case (you can guarantee that a fork won't occur without properly re-seeding a PRNG), and it makes bugs easier to catch in unit testing.
msg200778 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 13:44
+1 for exception prevents fork
msg200779 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-10-21 13:48
I have a couple random remarks:
- now that FDs are non-inheritable by default, fork locks around
subprocess and multiprocessing shouldn't be necessary anymore? What
other use cases does the fork-lock have?
- the current implementation keeps hard-references to the functions
passed: so if one isn't careful, you can end up easily with a lot of
objects kept alive just because of those references, which can be a
problem
- also, since we're not sure about the API, and it's mostly intended
to be used for the interpreter/stdlib, how about making it private for
now, or at least "provisional' (I think that's the term)?
- I'm also +1 on exceptions in prepare hook preventing fork, but we'll
need to play a bit with actual fork hooks to see if that's a
reasonable approach
msg200780 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-10-21 13:51
"now that FDs are non-inheritable by default, fork locks around
subprocess and multiprocessing shouldn't be necessary anymore? What
other use cases does the fork-lock have?"

Well, on Windows, it's still not possible to inherit only one handle. If you mark temporary the handle as inheritable (os.set_handle_inheritable), it might be inherited by a diffrent child process if another Python thread spawn a process... It's probably unlikely, so it's one of the use case of such lock :-)
msg200789 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-10-21 14:41
> - now that FDs are non-inheritable by default, fork locks around
> subprocess and multiprocessing shouldn't be necessary anymore? What
> other use cases does the fork-lock have?

CLOEXEC fds will still be inherited by forked children.

> - the current implementation keeps hard-references to the functions
> passed: so if one isn't careful, you can end up easily with a lot of
> objects kept alive just because of those references, which can be a
> problem

True, but you could make the same complaint about atexit.register().

One can fairly easily create something like weakref.finalize which uses atfork but is smart about not creating hard refs.
msg200797 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-10-21 15:00
> Richard Oudkerk added the comment:
>
>> - now that FDs are non-inheritable by default, fork locks around
>> subprocess and multiprocessing shouldn't be necessary anymore? What
>> other use cases does the fork-lock have?
>
> CLOEXEC fds will still be inherited by forked children.

Hum, right, I was thinking only about subprocess-created children
(where an exec follows immediately).

>> - the current implementation keeps hard-references to the functions
>> passed: so if one isn't careful, you can end up easily with a lot of
>> objects kept alive just because of those references, which can be a
>> problem
>
> True, but you could make the same complaint about atexit.register().

Yeah, but atexit is usually used for process-lifetime resources (I
mean, there's 'exit' in the name).
One of the main use cases for atfork hooks would be the numerous
stdlib objects which have locks (or locks themselves): most of such
objects have arbitrary lifetimes (e.g. logging, events, open files,
etc).

The risk of leak is IMO much greater.
msg200826 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-21 18:50
> One of the main use cases for atfork hooks would be the numerous
stdlib objects which have locks (or locks themselves): most of such
objects have arbitrary lifetimes (e.g. logging, events, open files,
etc).
> The risk of leak is IMO much greater.

Well it is customary for callback-based APIs to hold strong references to their callbacks. If a library wants to avoid leaks, it should register a single callback which will then walk the current "live" resources and protect them.
(i.e. the resource lifetime issue should be solved by library or application code, not by the atfork module)

By the way, +0 to raising and aborting the fork when there's an exception. The only annoyance I can think about is a buggy library which would prevent the user from forking.
msg200843 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-10-21 20:47
> Well it is customary for callback-based APIs to hold strong references to their callbacks. If a library wants to avoid leaks, it should register a single callback which will then walk the current "live" resources and protect them.

I guess that the custom usage makes sense.
I'd just like a note in the doc, if possible ;-)
msg201892 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-11-01 12:49
Given PEP 446 (fds are now CLOEXEC by default) I prepared an updated patch where the fork lock is undocumented and subprocess no longer uses the fork lock.  (I did not want to encourage the mixing of threads with fork() without exec() by exposing the fork lock just for that case.)

But I found that a test for the leaking of fds to a subprocess started with closefds=False was somewhat regularly failing because the creation of CLOEXEC pipe fds is not atomic -- the GIL is not held while calling pipe().

It seems that PEP 446 does not really make the fork lock redundant for processes started using fork+exec.

So now I don't know whether the fork lock should be made public.  Thoughts?
msg201899 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-11-01 13:47
> Given PEP 446 (fds are now CLOEXEC by default) I prepared an updated patch where the fork lock is undocumented and subprocess no longer uses the fork lock.  (I did not want to encourage the mixing of threads with fork() without exec() by exposing the fork lock just for that case.)
>
> But I found that a test for the leaking of fds to a subprocess started with closefds=False was somewhat regularly failing because the creation of CLOEXEC pipe fds is not atomic -- the GIL is not held while calling pipe().

Was it on Linux? If yes, was it on an old kernel/libc? (I just want to
make sure that pipe2() is indeed used if available).

> It seems that PEP 446 does not really make the fork lock redundant for processes started using fork+exec.
>
> So now I don't know whether the fork lock should be made public.  Thoughts?

IMO it should be kept private for now: I think it's really only useful
in some corner cases, and if it turns out to be useful, we can expose
it later.
msg201909 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-01 14:49
The PEP 446 does not offer any warranty on the atomicity on clearing the
inheritable flag.
msg201912 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-11-01 16:28
It is a recent kernel and does support pipe2().

After some debugging it appears that a pipe handle created in Popen.__init__() was being leaked to a forked process, preventing Popen.__init__() from completing before the forked process did.

Previously the test passed because Popen.__init__() acquired the fork lock.
msg266590 - (view) Author: Марк Коренберг (socketpair) * Date: 2016-05-29 07:17
Also I have to add, maybe you also add automatic file descriptor closing on fork. We already have O_CLOEXEC, we have no O_CLOFORK. So, maybe it useful to add such flag ?
msg294141 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-22 12:12
I've posted https://github.com/python/cpython/pull/1715 which adds a single `os.register_at_fork` function to allow registering at-fork handlers.  Comments welcome.
msg294145 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-22 12:58
`atexit.register()` has different signature and allows to pass arbitrary positional and keyword arguments to the registered function. This is incompatible with the "when" argument. If we want to support registering arguments with the function, we need either three registering functions or make the "when" parameter the first and positional-only.
msg294146 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-22 13:02
API-wise, I went for the minimal route. This avoids any discussion of adding a separate module for a tiny functionality that is only going to be used by a couple of libraries (and probably no application code).

Comparisons with atexit are not really relevant, IMO, since the use cases are very different.

As for passing explicit arguments to the callable, people can use a lambda or functools.partial.  I don't want to complicate the C implementation with matters that are not really important.
msg294147 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-22 13:41
I agree that with lambdas and functools.partial the support of arguments is not needed. Unless someone has good reasons for supporting explicit passing of arguments I'm fine with your design.

If the user code manually calls the fork() and (now deprecated) PyOS_AfterFork(), it runs "child" functions, but not "before" and "parent" functions. Is it worth to emit a runtime warning if the "before" or "parent" lists are not empty? Or even immediately exit a child with error?
msg294148 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-22 13:49
I don't think exiting would be a good idea at all.  I'm not sure about emitting a warning: the problem is that the people seeing the warning (plain users) can't do anything about it; worse, they probably will not even understand what it is about, and will get confused.

The package maintainer should see a deprecation warning when compiling code using PyOS_AfterFork() (thanks to the Py_DEPRECATED attribute).

Of course, I'll also update the docs to add a deprecation once the design is settled.
msg294149 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-22 14:08
PyOS_AfterFork() is used by one code, but the "before" and "parent" handlers are registered by the other code (likely the code of other library). The author of the program that uses both libraries (the one that uses PyOS_AfterFork() and the one that registers handlers) can notice the warning and report the bug in the first library. I think silently skipping registered handlers would be worse.

Let allow the user to control the behavior by setting the warnings filter. If the corresponding warnings are ignored, nothing happen, if they are errors, the child is exited, otherwise the warning message is output on stderr as for other warnings.
msg294159 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-22 15:36
> If the corresponding warnings are ignored, nothing happen, if they are errors, the child is exited,

Right now PyOS_AfterFork() doesn't return an error code.  It is not obvious how the caller would react: simply print the error? raise a fatal error? Something else?

The only third-party use of PyOS_AfterFork() I found is in uwsgi (and I'm not sure they're using it correctly, since I don't know if the parent process is using Python at all...).
msg294597 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-27 15:50
New changeset 346cbd351ee0dd3ab9cb9f0e4cb625556707877e by Antoine Pitrou in branch 'master':
bpo-16500: Allow registering at-fork handlers (#1715)
https://github.com/python/cpython/commit/346cbd351ee0dd3ab9cb9f0e4cb625556707877e
msg294629 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-28 07:33
Could you please update the documentation Antoine?

* An entry in the C API section of Misc/NEWS.
* The "What's New" document needs changes in multiple places: the os module improvement, C API additions and deprecation, and may be in porting guide.
* Update references to PyOS_AfterFork() in the documentation and comments.
msg294633 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-28 09:22
Can threading._after_fork() be rewritten with using the new API?
msg294634 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-28 09:35
New changeset f7ecfac0c15f0c43ef5e6c0081eb3a059af9f074 by Antoine Pitrou in branch 'master':
Doc nits for bpo-16500 (#1841)
https://github.com/python/cpython/commit/f7ecfac0c15f0c43ef5e6c0081eb3a059af9f074
msg294635 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-28 09:38
Le 28/05/2017 à 11:22, Serhiy Storchaka a écrit :
> 
> Serhiy Storchaka added the comment:
> 
> Can threading._after_fork() be rewritten with using the new API?

It should be possible indeed. Let me see.
msg294639 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-28 11:20
Can multiprocessing.util.register_after_fork() be rewritten with using the new API?
msg294640 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-28 11:24
> Can multiprocessing.util.register_after_fork() be rewritten with using the new API?

It wouldn't benefit much from it, and there might be timing issue given the comments in BaseProcess._bootstrap():

            old_process = _current_process
            _current_process = self
            try:
                util._finalizer_registry.clear()
                util._run_after_forkers()
            finally:
                # delay finalization of the old process object until after
                # _run_after_forkers() is executed
                del old_process
msg294641 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-28 12:02
New changeset 4a8bcdf79cdb3684743fe1268de62ee88bada439 by Antoine Pitrou in branch 'master':
bpo-16500: Use register_at_fork() in the threading module (#1843)
https://github.com/python/cpython/commit/4a8bcdf79cdb3684743fe1268de62ee88bada439
msg294642 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-28 12:11
In PR 1834 Gregory proposes an alternate API:

    os.register_at_fork(*, before=None, after_in_parent=None, after_in_child=None)

Maybe open a new issue for this?
msg294651 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-05-28 18:00
I'll just reuse this issue, I started that work just to try and clean up the new API and docs added in this one to be more modern.  Thanks for the code review, good suggestions!
msg294702 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-29 16:35
The one of potential advantages of the API proposed by Gregory is that os.register_at_fork() can be made atomic. Either register all callbacks, or do nothing in the case of error. But current proposed implementation is not atomic. If resizing of some list is failed due to MemoryError (or may be KeyboardInterrupt), some callbacks can already be registered.

Is it worth to guarantee the atomicity of os.register_at_fork().
msg294703 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-29 16:36
Le 29/05/2017 à 18:35, Serhiy Storchaka a écrit :
> 
> Is it worth to guarantee the atomicity of os.register_at_fork().

I don't think so, honestly.
msg294705 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-05-29 17:03
New changeset 163468a766e16604bdea04a1ab808c0d3e729e5d by Gregory P. Smith in branch 'master':
bpo-16500: Don't use string constants for os.register_at_fork() behavior (#1834)
https://github.com/python/cpython/commit/163468a766e16604bdea04a1ab808c0d3e729e5d
msg294711 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-29 20:18
> New changeset 163468a766e16604bdea04a1ab808c0d3e729e5d by Gregory P. Smith in branch 'master':
> bpo-16500: Don't use string constants for os.register_at_fork() behavior (#1834)
> https://github.com/python/cpython/commit/163468a766e16604bdea04a1ab808c0d3e729e5d

Since there was a disagreement, FYI I also prefer this API. Just the
API, I didn't look at the implementation in detail.

Anyway, it's nice to see this very old issue finally fixed! It was
opened 5 years ago!

I also prefer to have a builtin registered callback in the random
module instead of having a custom callback in the multiprocessing
module to re-seed the RNG after fork!
msg294724 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-05-29 23:23
This may also allow us to do something reasonable for http://bugs.python.org/issue6721 as well.
History
Date User Action Args
2017-05-29 23:23:04gregory.p.smithsetmessages: + msg294724
2017-05-29 20:18:18hayposetmessages: + msg294711
2017-05-29 17:05:38gregory.p.smithsetstatus: open -> closed
stage: commit review -> resolved
2017-05-29 17:03:43gregory.p.smithsetmessages: + msg294705
2017-05-29 16:36:39pitrousetmessages: + msg294703
2017-05-29 16:35:56serhiy.storchakasetmessages: + msg294702
2017-05-28 18:00:12gregory.p.smithsetstatus: closed -> open
assignee: gregory.p.smith
messages: + msg294651

stage: resolved -> commit review
2017-05-28 12:11:02serhiy.storchakasetmessages: + msg294642
2017-05-28 12:02:39pitrousetstatus: open -> closed
2017-05-28 12:02:28pitrousetmessages: + msg294641
2017-05-28 11:24:09pitrousetmessages: + msg294640
2017-05-28 11:20:25serhiy.storchakasetmessages: + msg294639
2017-05-28 09:50:23pitrousetpull_requests: + pull_request1929
2017-05-28 09:38:19pitrousetmessages: + msg294635
2017-05-28 09:35:16pitrousetmessages: + msg294634
2017-05-28 09:22:59serhiy.storchakasetmessages: + msg294633
2017-05-28 08:48:51pitrousetpull_requests: + pull_request1928
2017-05-28 07:33:41serhiy.storchakasetstatus: closed -> open

messages: + msg294629
2017-05-27 18:40:17gregory.p.smithsetpull_requests: + pull_request1917
2017-05-27 15:52:57pitroulinkissue29828 superseder
2017-05-27 15:51:53pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-05-27 15:50:56pitrousetmessages: + msg294597
2017-05-22 15:36:22pitrousetmessages: + msg294159
2017-05-22 14:08:16serhiy.storchakasetmessages: + msg294149
2017-05-22 13:49:08pitrousetmessages: + msg294148
2017-05-22 13:41:22serhiy.storchakasetmessages: + msg294147
2017-05-22 13:02:34pitrousetmessages: + msg294146
2017-05-22 12:58:52serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg294145
2017-05-22 12:12:25pitrousettitle: Add an 'atfork' module -> Allow registering at-fork handlers
2017-05-22 12:12:19pitrousetstage: patch review
messages: + msg294141
versions: + Python 3.7, - Python 3.5
2017-05-22 12:10:27pitrousetpull_requests: + pull_request1805
2016-05-29 18:18:39emptysquaresetnosy: + emptysquare
2016-05-29 07:17:25socketpairsetmessages: + msg266590
2016-05-29 07:06:03socketpairsetnosy: + socketpair
2016-03-29 06:23:42rpcope1setnosy: + rpcope1
2015-10-19 20:22:21ionelmcsetnosy: + ionelmc
2015-05-26 19:38:54yselivanovsetnosy: + yselivanov
2013-12-26 10:00:27xupengsetnosy: + xupeng
2013-12-22 21:00:32pitrousetversions: + Python 3.5, - Python 3.4
2013-11-01 16:28:52sbtsetmessages: + msg201912
2013-11-01 14:49:12hayposetmessages: + msg201909
2013-11-01 13:47:14neologixsetmessages: + msg201899
2013-11-01 12:49:46sbtsetmessages: + msg201892
2013-10-21 20:47:56neologixsetmessages: + msg200843
2013-10-21 18:50:12pitrousetmessages: + msg200826
2013-10-21 15:00:22neologixsetmessages: + msg200797
2013-10-21 14:41:44sbtsetmessages: + msg200789
2013-10-21 13:51:36hayposetmessages: + msg200780
2013-10-21 13:48:19neologixsetmessages: + msg200779
2013-10-21 13:44:36christian.heimessetmessages: + msg200778
2013-10-21 13:42:46DLitzsetmessages: + msg200777
2013-10-21 13:02:12sbtsetmessages: + msg200774
2013-10-21 12:30:49christian.heimessetassignee: christian.heimes -> (no value)
messages: + msg200767
2013-10-16 08:58:47alilessetnosy: + aliles
2013-09-30 20:37:07DLitzsetnosy: + DLitz
2013-08-30 13:56:03pitrouunlinkissue18882 dependencies
2013-08-30 13:30:05asvetlovlinkissue18882 dependencies
2013-08-21 18:44:31neologixlinkissue6923 superseder
2013-08-15 12:44:16pitrousetnosy: + neologix
2013-01-14 16:06:08pitrousetmessages: + msg179949
2013-01-14 15:49:37lemburgsetnosy: + lemburg
messages: + msg179945
2013-01-14 09:13:28hayposetmessages: + msg179927
2013-01-13 21:43:00Arfreversetnosy: + Arfrever
2013-01-13 19:21:22georg.brandlsetnosy: + georg.brandl
messages: + msg179888
2013-01-12 23:37:35hayposetnosy: + haypo

messages: + msg179838
title: Add an 'afterfork' module -> Add an 'atfork' module
2012-11-27 05:56:18grahamdsetnosy: + grahamd
2012-11-24 00:35:44jceasetnosy: + jcea
2012-11-20 20:04:30christian.heimessetmessages: + msg176022
2012-11-20 19:49:23pitrousetnosy: + pitrou
messages: + msg176020
2012-11-20 19:33:38gregory.p.smithsetmessages: + msg176019
2012-11-20 16:29:10amaury.forgeotdarcsetmessages: + msg176004
2012-11-20 16:20:42christian.heimessetmessages: + msg176002
2012-11-20 15:44:34sbtsetmessages: + msg175997
2012-11-20 14:59:07amaury.forgeotdarcsetnosy: + amaury.forgeotdarc, - Amaury.Forgeot.d'Arc
2012-11-20 14:35:50asvetlovsetnosy: + asvetlov
2012-11-20 09:11:52Amaury.Forgeot.d'Arcsetnosy: + Amaury.Forgeot.d'Arc
messages: + msg175980
2012-11-20 00:52:11christian.heimessetmessages: + msg175975
2012-11-20 00:13:31gregory.p.smithsetmessages: + msg175974
2012-11-19 23:59:25gregory.p.smithsetmessages: + msg175973
2012-11-19 22:43:50sbtsetfiles: + pure-python-atfork.patch
keywords: + patch
messages: + msg175972
2012-11-19 20:55:20christian.heimessetnosy: + twouters, gregory.p.smith
messages: + msg175967
2012-11-18 17:34:43sbtsetmessages: + msg175892
2012-11-18 17:08:15pitrousetnosy: + sbt
2012-11-18 15:20:05christian.heimescreate