<< Fault handler for SIGSEGV, SIGFPE, SIGBUS and SIGILL signals: display the
Python backtrace and restore the previous handler. Allocate an alternate stack
for this handler, if sigaltstack() is available, to be able to allocate memory
on the stack, even on stack overflow (not available on Windows).

Import the module and call faulthandler.enable() to enable the fault handler.

The fault handler is called on catastrophic cases and so it can only use
signal-safe functions (eg. it doesn't allocate memory on the heap). That's why
the backtrace is limited: it only supports ASCII encoding (use the
backslashreplace error handler for non-ASCII characters), doesn't print
the source code in the backtrace (only the filename, the function name and the
line number), is limited to 100 frames and 100 threads.

By default, the Python backtrace is written to the standard error stream. Start
your graphical applications in a terminal and run your server in foreground to
see the backtrace, or pass a file to faulthandler.enable().

faulthandler is implemented in C using signal handlers to be able to dump a
backtrace on a crash or when Python is blocked (eg. deadlock). >>

https://github.com/haypo/faulthandler

faulthandler works on Python 2.5 .. 3.3.

--

faulthandler is an alternative to gdb7 + python-gdb.py to dump the backtrace. faulthandler already helped me to debug a PyQt crash on Windows: I don't know how to install gdb on Windows, especially with faulthandler!

But I didn't get a lot of feedbacks.

--

I don't know if the whole module should be included or not.

TODO list:

 * add something to enable faulthandler on the command line: an option on the command line (eg. python -x faulthandler script.py) or an environment variable (PYTHONFAULTHANDLER=y python script.py)?
 * decide what to do with child processes: disable the faulthandler after a fork?
 * use something else than Py_AtExit() to unload the module?
 * dump the Python backtrace on a fatal error (and/or maybe catch SIGABRT)
 * use maybe something else than alarm()+SIGARLM for dumpbacktrace_later() because it interrupts the current system call

--

History of this module.

I first proposed a segfault handler using setjmp/longjmp to raise a classic Python exception. So it was  possible to execute Python code after a segfault (including stack overflow). But the idea was rejected because the Python internal state may be corrupted if the segfault was an invalid memory write (buffer overflow?), and anyway we cannot warranty that the internal state is still consistent after a long jump.

=> http://bugs.python.org/issue3999 (sept. 2009)

After that, I proposed to just display the Python backtrace on segfault. The signal handler was enabled by default. Because some people was concerned about possible bugs or security vulnerabilities introduced by the signal handler, I proposed options to disable it. Finally, I proposed to disable it by default and add options to enable it, but it was too late for an inclusion into Python 3.2. Anyways, the project (a patch) was young and the API not correctly defined.

=> http://bugs.python.org/issue8863 (may 2010)

I converted my patch into a module. During this conversion, I realized that a module API is more natural than options using environment variables or functions in the sys module (I proposed sys.setfaulthandlerenabled(), an horrible name :-)). And it becomes more natural to enable the faulthandler by importing a module and calling a function (faulthandler.enable()).

Later I added functions to display the Python backtrace not only on a crash, but after a certain amount of time, or when an user signal is raised. You can also call the function directly to dump the backtrace. And finally, I added many option to configure how the backtrace is displayed: current thread of all threads, sys.stderr or another file, ...

--

I hope that this module will help to understand multiprocessing issues on the buildbots!

I tested faulthandler on Linux, FreeBSD and Windows XP: it works well. I suppose that it works on any operating systems.

You can also try it on know crashers: Lib/test/crashers/ (you have to modify the files to add: import faulthandler; faulthandler.enable()). Example:

----
$ ./python Lib/test/crashers/bogus_code_obj.py 
Fatal Python error: Segmentation fault

Traceback (most recent call first):
  File "", line 1 in 
  File "Lib/test/crashers/bogus_code_obj.py", line 20 in <module>
Erreur de segmentation
----

The backtrace is not revelant here because the file is very short, but faulthandler is more useful on an huge application.

It would be nice if it were enabled by default for fatal errors (and asserts perhaps?).

Various thoughts/nitpicking:
  - is it possible to indicate with a coding convention (e.g. comments) which parts of the code are intended to be called from a signal handler?  It seems worth making this explicit.  Or perhaps put it all in one file?
  - within tests.py, check_enabled and check_disabled seem to me to be misnamed; it's not at all clear what they do.
      I'd suggest renaming "get_output" to "run_code", perhaps (adding a docstring specifying the returned value)
      "check_enabled" seems to mean "assertCodeLeadsToOutput" or somesuch.

Within backtrace.py:
  - do all platforms supported by Python have a concept of numeric filedescriptors? I was wondering if FILE* might be a better abstraction here (with flushing), then read http://bugs.python.org/issue8863#msg124385 which gives the reason: fprintf etc are not signal-safe
  - all of the calls to "write" ignore the return code, leading to warnings from GCC.  I don't think there's any good way to handle errors from these calls, though.

Might be nice to also have SIGABRT (as per a c-level assertion failure), exposed 

NB: on Fedora/RHEL we also have a whole-system crash detection system (called "abrt": https://fedorahosted.org/abrt/ ), and in theory, that means that for me, crash reports get run using the gdb pretty-print hooks.

I'm wondering to what extent this would interract with whole-system crash-detection tools: would it e.g. mask a SIGSEGV, so that the crash is not seen by that system?

Le lundi 21 mars 2011 à 15:59 +0000, Dave Malcolm a écrit :
> Various thoughts/nitpicking:
>   - is it possible to indicate with a coding convention (e.g.
> comments) which parts of the code are intended to be called from a
> signal handler?  It seems worth making this explicit.  Or perhaps put
> it all in one file?

Ok, good idea, I will do that. I think that I will write it in the
function comment, something like: "This function is signal safe".

The following functions are signal safe:

faulthandler_fatal_error()
faulthandler_dump_backtrace()
faulthandler_dump_backtrace_threads()
faulthandler_user()

(.. and all helper functions, used by faulthandler_dump_backtrace():
reverse_string(), dump_decimal(), dump_hexadecimal(), dump_ascii(),
dump_frame().)

>   - within tests.py, check_enabled and check_disabled seem to me to be
> misnamed; it's not at all clear what they do.
>       I'd suggest renaming "get_output" to "run_code", perhaps (adding
> a docstring specifying the returned value)
>       "check_enabled" seems to mean "assertCodeLeadsToOutput" or
> somesuch.

Ok, I will try to find better names.

> Within backtrace.py:
>   - do all platforms supported by Python have a concept of numeric
> filedescriptors? I was wondering if FILE* might be a better
> abstraction here (with flushing), then read
> http://bugs.python.org/issue8863#msg124385 which gives the reason:
> fprintf etc are not signal-safe

Yes, I think that all platforms support numeric file descriptors. On
Windows, numeric file descriptors are not the native type for file, but
Windows provides a POSIX API.

And yes, FILE* API cannot be used because it uses buffers, and buffers
are not signal safe (at least, fwrite() is not signal safe).

I tested my module on Linux, FreeBSD and Windows. I don't have other
OS :-)

>   - all of the calls to "write" ignore the return code, leading to
> warnings from GCC.  I don't think there's any good way to handle
> errors from these calls, though.

Except exiting the signal handler, I don't see anything useful to do on
write() error. I think that it is safe to ignore write() errors, and I
prefer to keep the code simple.

I don't know how to make these warnings quiet.

> Might be nice to also have SIGABRT (as per a c-level assertion
> failure), exposed 

I think that it would do that while integrating faulthandler into Python
to check the interaction with Py_FatalError().

> NB: on Fedora/RHEL we also have a whole-system crash detection system
> (called "abrt": https://fedorahosted.org/abrt/ ), and in theory, that
> means that for me, crash reports get run using the gdb pretty-print
> hooks.
>
> I'm wondering to what extent this would interract with whole-system
> crash-detection tools: would it e.g. mask a SIGSEGV, so that the crash
> is not seen by that system?

faulthandler is compatible with gdb and abrt. For gdb, you get the
SIGSEGV signal before faulthandler: the "signal SIGSEGV" gdb command
will call faulthandler signal handler. For abrt, faulthandler does print
the backtrace, and then abrt is called. Execution order:

crash => gdb => faulthandler => abrt

(I think that gdb and abrt are exclusive)

I didn't try abrt, but I tried Ubuntu Apport which is smiliar (Apport
uses /proc/sys/kernel/core_pattern with a pipe).

You may test faulthandler on Fedora to tell me if it interacts correctly
with abrt :-)
https://github.com/haypo/faulthandler/wiki/

> Ok, good idea, I will do that. I think that I will write it in the
> function comment, something like: "This function is signal safe".

Done in 6685691dfcbd3644feffcb197491bce3168ff5de (git SHA-1)

While checking the usage of signal safe functions, I found a bug: dumpbacktrace_later() signal handler called Py_CLEAR() which is far from being signal safe. I tried to replace it be a pending call, but Py_AddPendingCall() doesn't look to be signal safe. So I just removed the call. cancel_dumpbacktrace_later() is already documented as being required to clear the reference to the file.

> >   - within tests.py, ...
> Ok, I will try to find better names.

Done in db99e17dea187bec4248aca9fc81f673d88dec93. I documented methods and renamed some of them.

I renamed some functions to conform to the PEP 8 (and have more readable function names). I prefer to do it today instead having to keep ugly names for years :-)

Functions are now:

 * enable(file=sys.stderr, all_threads=False)
 * disable()
 * is_enabled()

 * dump_backtrace(file=sys.stderr, all_threads=False)
 * dump_backtrace_later(delay, repeat=False, file=sys.stderr, all_threads=False)
 * cancel_dump_backtrace_later()

 * register(signum, file=sys.stderr, all_threads=False)
 * unregister(signum)

 * sigbus()
 * sigfpe()
 * sigill()
 * sigsegv()

Refer to the README file for the details.

> It would be nice if it were enabled by default for fatal errors
> (and asserts perhaps?).

That would mean that the module should be a builtin module, or that it is always loaded in memory.

I am maybe ok to enable it by default for debug builds, but not for release builds. At least, not in a first time.

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> It would be nice if it were enabled by default for fatal errors
>> (and asserts perhaps?).
> 
> That would mean that the module should be a builtin module, or that it is always loaded in memory.
> 
> I am maybe ok to enable it by default for debug builds, but not for release builds. At least, not in a first time.

FWIW: We've been using a similar approach in mxTools.safecall()
(which is an undocumented function only and only available if
mxTools is compiled with a special define; see the mxTools.c source
code for details).

Our use case is different, though: We use this function to
call a Python callable wrapped with a longjump environment and
instead of dumping a traceback we issue an exception, so that
the calling code can deal with the problem in a more flexible way.

The main reasoning here is proper cleanup, signal the problem
to managing code and providing more information about the
interpreter state at the time of the segfault.

The function is mostly used when calling out to (possibly buggy)
3rd party extensions hooked into a long running web server
application, where it enables the server process to write
request information about the problem to a log file and then
shuts down the process very quickly.

Note that we haven't bothered dealing with threads, since we
normally use single threaded server processes and because signals
and threads don't interact reliably anyway.

Perhaps you could consider adding a similar approach (raising
an exception instead of writing a traceback) to the module.
We could then port our code to use your module, which is
more advanced.

BTW: Why do you call the traceback functions "*backtrace*" instead
of "*traceback*" ?

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

Le mardi 22 mars 2011 à 09:01 +0000, Marc-Andre Lemburg a écrit :
> Perhaps you could consider adding a similar approach (raising
> an exception instead of writing a traceback) to the module.
> We could then port our code to use your module, which is
> more advanced.

I already wrote a patch implementing this idea, 2 years ago, and the
idea was rejected. Re-read the history of the module, described in the
first message of this issue:

<< History of this module.

I first proposed a segfault handler using setjmp/longjmp to raise a
classic Python exception. So it was  possible to execute Python code
after a segfault (including stack overflow). But the idea was rejected
because the Python internal state may be corrupted if the segfault was
an invalid memory write (buffer overflow?), and anyway we cannot
warranty that the internal state is still consistent after a long jump.

=> http://bugs.python.org/issue3999 (sept. 2009)

... >>

You may dig python-dev archives to learn more about this proposition.

> BTW: Why do you call the traceback functions "*backtrace*" instead
> of "*traceback*" ?

Oh... I don't know. Since Python uses "traceback" word, I will rename my
functions.

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Le mardi 22 mars 2011 à 09:01 +0000, Marc-Andre Lemburg a écrit :
>> Perhaps you could consider adding a similar approach (raising
>> an exception instead of writing a traceback) to the module.
>> We could then port our code to use your module, which is
>> more advanced.
> 
> I already wrote a patch implementing this idea, 2 years ago, and the
> idea was rejected. Re-read the history of the module, described in the
> first message of this issue:
> 
> << History of this module.
> 
> I first proposed a segfault handler using setjmp/longjmp to raise a
> classic Python exception. So it was  possible to execute Python code
> after a segfault (including stack overflow). But the idea was rejected
> because the Python internal state may be corrupted if the segfault was
> an invalid memory write (buffer overflow?), and anyway we cannot
> warranty that the internal state is still consistent after a long jump.
> 
> => http://bugs.python.org/issue3999 (sept. 2009)
> 
> ... >>
> 
> You may dig python-dev archives to learn more about this proposition.

I know that the state after a longjump in such a case cannot
be guaranteed, but from using the approach in practice over many
years, I know that it works quite well - most situations related
to programming errors in the 3rd party modules we plug into the
system (quant libs, ie. code that does pricing and risk analysis
of financial products) are segfaults related to NULL pointer
derefs and division by zero. Complete corruption of the Python
interpreter state is very rare.

Anyway, providing such an approach as option (with the default
being your implemented traceback printout), would leave the
decision to take this risk to the programmer and I think that's
a fair deal.

>> BTW: Why do you call the traceback functions "*backtrace*" instead
>> of "*traceback*" ?
> 
> Oh... I don't know. Since Python uses "traceback" word, I will rename my
> functions.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

Antoine Pitrou wrote:
> It would be nice if it were enabled by default for fatal errors (and asserts perhaps?).

I feel like a broken record. This code hardcodes fd=2 as a write target on crash, which is not safe thing to do at all. You can argue that adopters of Python 3.3 should have to deal with that fact, but it's obscure and there is no way to warn anyone about it except by putting a NEWS item, and if the PyCapsule discussion on python-dev have taught us anything: even well meaning programmers miss these things all the time.

I have stated this repeatedly on the other issues for this same discussion. I think creating a completely new issue for this same topic has segmented the discussion unfortunately. I wrote a much longer and more thoughtful explanation of why faulthandler writes to the wrong "thing" here:

http://bugs.python.org/msg124381

AFAICT, Victor has addressed my issue by giving programmers yet another interface to configure (that they may or may not be aware of). So, the only way this acceptable to me is if it's off by default and a programmer who wants this functionality opts-in and has taken care to make sure it does the right thing. My suggestion that faulthandler needs to find a way to be coupled to "sys.stderr" still stands.

> Antoine Pitrou wrote:
> > It would be nice if it were enabled by default for fatal errors (and asserts perhaps?).
> 
> I feel like a broken record. This code hardcodes fd=2 as a write target on crash,

For fatal errors, you needn't be async-safe, so the fatal error code
could read fileno(stderr) and give it to the traceback printing code.
What do you think, Victor?

> For fatal errors, you needn't be async-safe, so the fatal error code
> could read fileno(stderr) and give it to the traceback printing code.

My last patch for issue #8863 does exactly that:

##############
 void
 Py_FatalError(const char *msg)
 {
-    fprintf(stderr, "Fatal Python error: %s\n", msg);
-    fflush(stderr); /* it helps in Windows debug build */
-    if (PyErr_Occurred()) {
+    const int fd = fileno(stderr);
+
+    fputs("Fatal Python error: ", stderr);
+    fputs(msg, stderr);
+    fputc('\n', stderr);
+    fflush(stderr);
+
+    if (PyErr_Occurred())
         PyErr_PrintEx(0);
+    else {
+        fputc('\n', stderr);
+        fflush(stderr);
+        _Py_DumpBacktrace(fd);
     }
...
##############

Yes, call fileno() here is safe.

--

The main problem was on the SIGSEGV handler which was first proposed as enabled by default. Extract of my old patch:

+static void
+fault_handler(int signum)
+{
+    const int fd = 2; /* should be fileno(stderr) */
+    unsigned int i;
+    fault_handler_t *handler;
...

In the faulthandler module, the last call to faulthandler.enable() saves sys.stderr.fileno(). If this file descriptor is replaced by a critical file, we have a problem. It can occurs in two cases:
 - stderr was closed (after the call to enable) and a new file gets its file descriptor number
 - dup2() was used

Both cases may occur on a server application.

But I think that everybody agrees to disable the SIGSEGV handler by default.

--

I'm preparing the integration of faulthandler in the following Mercurial repo:
http://hg.python.org/features/faulthandler/

I didn't write the fatal error hook yet.

Let's try the "Remote hg repo" feature.

Ok, here is a full patch tested on Linux and Windows:
 - Add faulthandler module (code, doc, tests) as a builtin module
 - Add "-X faulthandler=1" command line option and PYTHONFAULTHANDLER=1 en var to enable the fault handler at startup
 - Add _Py_DumpTraceback() and _Py_DumpTracebackThreads() functions to Python/traceback.c
 - Py_FatalError() calls _Py_DumpTraceback() if there is not exception
 - Initialize/Unload faulthandler explicitly in Py_InitializeEx()/Py_Finalize() to do it in the right order

I added a section in the doc to explain the file descriptor issue.

I merged different .c and .h files into faulthandler.c, the code is easier to read and maintain like that.

I don't know if the following test is the best test to check if SIGARLM constant and alarm() function are available:

+#ifdef SIGALRM
+#  define FAULTHANDLER_LATER
+#endif

(The test works at least on Linux and Windows)

(Reminder for me: add something to Doc/whatsnew/3.3.rst and Misc/NEWS)

I've posted a review at http://bugs.python.org/review/11393/show.

I updated the (Hg repo and the) patch to fix all Antoine's remarks.

I don't know if it is important to document that some functions keep an internal reference to the file argument. For example, I wrote "It keeps a reference to file: use disable() to clear this reference." in faulthandler.enable() documentation. Is it necessary?

New patch 605422430234.diff:
 - Remove test_refcount(), it changed faulthandler internal state, and so it was no more possible to use faulthandler in regrtest.py
 - Use "#ifdef HAVE_ALARM" to decide if dump_traceback_later() is available, instead of "#ifdef SIGALRM"
 - Don't document _sig*() functions or _fatal_error() anymore

> I updated the (Hg repo and the) patch to fix all Antoine's remarks.

Can you make the suggested changes to the tests? Thank you.

I have pushed a new branch "faulthandler-thread" in http://hg.python.org/features/faulthandler/. It contains an implementation of dump_tracebacks_later() using a watchdog thread, instead of alarm().

It has two advantages:
- it works under Windows
- it won't disrupt use of alarm() or SIGALRM by user code (including the test suite)

It has one drawback: you can only display all threads, since the watchdog thread is not a Python thread.

I haven't fixed the tests for it, I'm waiting for Victor's changes first ;)

regrtest_timeout.patch: add --timeout option to regrtest.py to display the traceback and then exit if a test takes more than TIMEOUT seconds. It adds also a check on the subprocess exit code (regrtest.py -j): raise a exception if it is different than zero.

I regenerated the patch.

Last changes:

 - I merged Antoine's new implementation of faulthandler.dump_backtraces_later() using a thread
 - dump_backtraces_later() has a new exit option: if exit=True, call _exit()
 - disable (remove) register() and unregister() on Windows just because these functions are useless. No signal can be used. Only SIGSEGV and SIGBUS can be handled by the process, and these signals cannot be used with register(), only with enable()
 - fix register() and unregister() to be "signal safe": don't move the signal list in memory (because of realloc) or free memory (on unregister) because the signal handler may be running at the same time. Use a static list instead, with a "enabled" flag per signal. The code becomes simpler.
 - fix register(): don't reinstall the signal if it was already installed
 - speed up the test: because dump_backtraces_later() has now a subsecond resolution, we can use sleep of 50 ms instead of 1 sec

It tested the patch on Linux and Windows: it works. Cool, dump_backtraces_later() is now also supported on Windows, thanks to Antoine!

My TODO list is empty (the last item was "fix register() to be signal safe") so I think that the patch is ready to be commited.

Remainder for me: the patch contains some whitespace changes in Makefile.pre.in and Setup.dist.

dump_traceback(), enable(), dump_tracebacks_later(), register() flush the input file. It is not said in the doc. Should it be documented? For enable(), dump_tracebacks_later(), register(), it may be useless because the traceback is dumped later, and there may be new buffered data in the file. But I don't think that it hurts to flush the file.

>  - speed up the test: because dump_backtraces_later() has now a
> subsecond resolution, we can use sleep of 50 ms instead of 1 sec

This is too short, there may be random failures on some slow buildbots.
IMO, 0.5s is the minimum you can use.

+    def _check_dump_tracebacks_later(self, repeat, cancel, filename):
+        """
+        Call dump_tracebacks_later() two times, or three times if
repeat is True.
+        Check the output: the traceback may be written 1, 2 or 3 times
+        depending on repeat and cancel options.
+
+        Raise an error if the output doesn't match the expect format.
+        """

The docstring is outdated. It also seems the "cancel" option isn't
useful anymore, you could remove it and simplify the test.

+        process = script_helper.spawn_python('-c', code)
+        stdout, stderr = process.communicate()

Shouldn't you check the return code as well?

+        code = "\n".join(code)

Again, I think it would make the code simpler and more maintainable if
you used triple-quoted strings instead of lists/tuples.

When you launch a waiting thread in a subprocess, I think it's better to
set it in daemon mode so as to avoid blocking if the main thread raises
an exception.

You have a "#ifdef MS_WINDOWS" in check_signum() but that function is
not compiled under Windows (it is inside "#ifdef FAULTHANDLER_USER").

> >  - speed up the test: because dump_backtraces_later() has now a
> > subsecond resolution, we can use sleep of 50 ms instead of 1 sec
> 
> This is too short, there may be random failures on some slow buildbots.
> IMO, 0.5s is the minimum you can use.

Ok, changed.

> The docstring is outdated.

fixed

>  It also seems the "cancel" option isn't
> useful anymore, you could remove it and simplify the test.

cancel is used by test_dump_tracebacks_later_repeat_cancel() to check
that faulthandler.cancel_dump_tracebacks_later() did cancel the last
dump_backtraces_later() request.

> +        process = script_helper.spawn_python('-c', code)
> +        stdout, stderr = process.communicate()
> 
> Shouldn't you check the return code as well?

Yes, except for fatal errors, because the C library exits with a
non-zero exit code (e.g. 139 for a SIGSEGV on Linux). I added a check on
the exit code.

> +        code = "\n".join(code)
> 
> Again, I think it would make the code simpler and more maintainable if
> you used triple-quoted strings instead of lists/tuples.

Ok ok, done

> When you launch a waiting thread in a subprocess, I think it's better to
> set it in daemon mode so as to avoid blocking if the main thread raises
> an exception.

I doesn't know that. fixed

> You have a "#ifdef MS_WINDOWS" in check_signum() but that function is
> not compiled under Windows (it is inside "#ifdef FAULTHANDLER_USER").

Ah yes, the code became useless: removed

--

Other changes:

 - faulthandler_fatal_error() calls the previous signal handler using
raise(signum). Before, the signal was raised again because the same
instruction was executed again, and it raised the same fault. But if the
user sends manually a fatal signal to the process, the signal was
ignored because of the fault handler (but the traceback was displayed).
 - use SA_NODEFER and SA_RESTART flags in enable() and register()
 - faulthandler_get_fileno() accepts fileno()==0
 - cleanup the doc

> faulthandler_fatal_error() calls the previous signal handler
> using raise(signum)

It doesn't work as expected on Windows: Windows doesn't call its own signal handler anymore. Use the previous code (only on Windows).

I also added a test for stack overflow: it fails on FreeBSD because I called sigaltstack() with the wrong arguments. It is now fixed.

test_faulthandler pass on Linux, Windows and FreeBSD.

> My TODO list is empty (the last item was "fix register() to be signal
> safe") so I think that the patch is ready to be commited.

As I wrote, I did some tests on FreeBSD, found bugs and fixed them. I also fixed the weird behaviour if the user sends manually a fatal signal (like SIGSEGV): it is no more ignored. And I patched test_faulthandler to not create core files. On Windows, there is still a popup warning the user a fatal error occured. I don't know how to disable (temporary) this popup.

Here is the updated patch and I hope the final patch.

>  - speed up the test: because dump_backtraces_later() has now a
> subsecond resolution, we can use sleep of 50 ms instead of 1 sec

This is too short, there may be random failures on some slow buildbots.
IMO, 0.5s is the minimum you can use.

+    def _check_dump_tracebacks_later(self, repeat, cancel, filename):
+        """
+        Call dump_tracebacks_later() two times, or three times if
repeat is True.
+        Check the output: the traceback may be written 1, 2 or 3 times
+        depending on repeat and cancel options.
+
+        Raise an error if the output doesn't match the expect format.
+        """

The docstring is outdated. It also seems the "cancel" option isn't
useful anymore, you could remove it and simplify the test.

+        process = script_helper.spawn_python('-c', code)
+        stdout, stderr = process.communicate()

Shouldn't you check the return code as well?

+        code = "\n".join(code)

Again, I think it would make the code simpler and more maintainable if
you used triple-quoted strings instead of lists/tuples.

When you launch a waiting thread in a subprocess, I think it's better to
set it in daemon mode so as to avoid blocking if the main thread raises
an exception.

You have a "#ifdef MS_WINDOWS" in check_signum() but that function is
not compiled under Windows (it is inside "#ifdef FAULTHANDLER_USER").

I merged the faulthandler branch into the default branch. I removed __version__ field: the Python version should be enough. I also fixed an infinite loop raised by test_capi.

test_faulthandler pass on Solaris and OpenIndiana, but it fails on PPC:



======================================================================
FAIL: test_enable_file (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 166, in test_enable_file
    filename=filename)
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 84, in check_fatal_error
    self.assertEqual(lines, expected)
AssertionError: Lists differ: ['Fatal Python error: Bus erro... != ['Fatal Python error: Segmenta...

First differing element 0:
Fatal Python error: Bus error
Fatal Python error: Segmentation fault

- ['Fatal Python error: Bus error',
+ ['Fatal Python error: Segmentation fault',
   '',
   'Traceback (most recent call first):',
   '  File "<string>", line 4 in <module>']

======================================================================
FAIL: test_enable_threads (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 176, in test_enable_threads
    all_threads=True)
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 84, in check_fatal_error
    self.assertEqual(lines, expected)
AssertionError: Lists differ: ['Fatal Python error: Bus erro... != ['Fatal Python error: Segmenta...

First differing element 0:
Fatal Python error: Bus error
Fatal Python error: Segmentation fault

- ['Fatal Python error: Bus error',
+ ['Fatal Python error: Segmentation fault',
   '',
   'Current thread XXX:',
   '  File "<string>", line 3 in <module>']

======================================================================
FAIL: test_gil_released (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 154, in test_gil_released
    'Segmentation fault')
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 84, in check_fatal_error
    self.assertEqual(lines, expected)
AssertionError: Lists differ: ['Fatal Python error: Bus erro... != ['Fatal Python error: Segmenta...

First differing element 0:
Fatal Python error: Bus error
Fatal Python error: Segmentation fault

- ['Fatal Python error: Bus error',
+ ['Fatal Python error: Segmentation fault',
   '',
   'Traceback (most recent call first):',
   '  File "<string>", line 3 in <module>']

======================================================================
FAIL: test_sigfpe (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 104, in test_sigfpe
    'Floating point exception')
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 84, in check_fatal_error
    self.assertEqual(lines, expected)
AssertionError: Lists differ: [] != ['Fatal Python error: Floating...

Second list contains 4 additional elements.
First extra element 0:
Fatal Python error: Floating point exception

- []
+ ['Fatal Python error: Floating point exception',
+  '',
+  'Traceback (most recent call first):',
+  '  File "<string>", line 3 in <module>']

======================================================================
FAIL: test_sigsegv (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 93, in test_sigsegv
    'Segmentation fault')
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 84, in check_fatal_error
    self.assertEqual(lines, expected)
AssertionError: Lists differ: ['Fatal Python error: Bus erro... != ['Fatal Python error: Segmenta...

First differing element 0:
Fatal Python error: Bus error
Fatal Python error: Segmentation fault

- ['Fatal Python error: Bus error',
+ ['Fatal Python error: Segmentation fault',
   '',
   'Traceback (most recent call first):',
   '  File "<string>", line 3 in <module>']

----------------------------------------------------------------------

http://hg.python.org/features/faulthandler/rev/0943ef33495a should fix test_faulthandler to support SIGBUS on reading from NULL. But it doesn't fix the failure on SIGFPE yet. I'm running the buildbot to check if it's better or not.

New changeset b0680b5a5215 by Victor Stinner in branch 'default':
Issue #11393: Add the new faulthandler module
http://hg.python.org/cpython/rev/b0680b5a5215

New changeset df240014e72f by Victor Stinner in branch 'default':
Issue #11393: reenable all tests in regrtest.py (wooops, sorry Antoine)
http://hg.python.org/cpython/rev/df240014e72f

First failure: AMD64 Gentoo Wide 3.x.
http://www.python.org/dev/buildbot/all/builders/AMD64%20Gentoo%20Wide%203.x/builds/1363/

[1/1] test_faulthandler
test test_faulthandler failed -- Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.ochtman-gentoo-amd64/build/Lib/test/test_faulthandler.py", line 165, in test_stack_overflow
    '(?:Segmentation fault|Bus error)')
  File "/home/buildbot/buildarea/3.x.ochtman-gentoo-amd64/build/Lib/test/test_faulthandler.py", line 95, in check_fatal_error
    self.assertRegex(output, regex)
AssertionError: Regex didn't match: '^Fatal Python error: (?:Segmentation fault|Bus error)\n\nTraceback\\ \\(most\\ recent\\ call\\ first\\):\n  File "<string>", line 3 in <module>$' not found in ''

test_faulthandler failed also on AMD64 OpenIndiana 3.x: regrtest.py (only executing test_faulthandler) was killed after 11 min 20 sec.

New changeset 25a2aeecb34b by Victor Stinner in branch 'default':
Issue #11393: Disable test_stack_overflow of test_faulthandler
http://hg.python.org/cpython/rev/25a2aeecb34b

I disabled test_stack_overflow because the test fails on AMD64 Gentoo Wide 3.x, and maybe other buildbots (AMD64 OpenIndiana 3.x?). I fear that the test uses a lot of CPU, memory and/or swap.

stack_overflow.patch should fix AMD64 Gentoo Wide 3.x and AMD64 OpenIndiana 3.x failures. I schedule a test on their custom slaves, but these buildbots are still running test_faulthandler on their 3.x slaves. I interrupted the test, but it doesn't work (click on Cancel doesn't stop the tests). Schedule tests:

http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%20custom
http://www.python.org/dev/buildbot/all/builders/AMD64%20Gentoo%20Wide%20custom

I'm impatient to test regrtest_timeout.patch on x86 XP-4 3.x to try to learn more on test_multiprocessing timeout (1200 seconds)!

New changeset a6d2c703586a by Victor Stinner in branch 'default':
Issue #11393: Fix the documentation (cancel_dump_traceback_later)
http://hg.python.org/cpython/rev/a6d2c703586a

New changeset e289b64f355d by Victor Stinner in branch 'default':
Issue #11393: limit stack overflow test to 100 MB
http://hg.python.org/cpython/rev/e289b64f355d

test_faulthandler blocked AMD64 Gentoo Wide 3.x and AMD64 OpenIndiana 3.x buildbots because of the stack overflow test.

> New changeset e289b64f355d by Victor Stinner in branch 'default':
> Issue #11393: limit stack overflow test to 100 MB
> http://hg.python.org/cpython/rev/e289b64f355d

Ok, it fixed test_faulthandler on AMD64 OpenIndiana 3.x and AMD64 Gentoo Wide 3.x.

I opened #11727 for the patch on regrtest.py. All buildbots look happy (no more failure on test_faulthandler), so let's close this issue.

There is a failure on FreeBSD:

======================================================================
FAIL: test_dump_tracebacks_later_repeat (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_faulthandler.py", line 406, in test_dump_tracebacks_later_repeat
    self.check_dump_tracebacks_later(repeat=True)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_faulthandler.py", line 400, in check_dump_tracebacks_later
    self._check_dump_tracebacks_later(repeat, cancel, None)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_faulthandler.py", line 381, in _check_dump_tracebacks_later
    trace = self.get_output(code, True, filename)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_faulthandler.py", line 57, in get_output
    self.assertEqual(exitcode, 0)
AssertionError: -11 != 0

http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%203.x/builds/1333/steps/test/logs/stdio

New changeset aa2ac1581d23 by Victor Stinner in branch 'default':
Issue #11393: test_faulthandler checks the exitcode after the output
http://hg.python.org/cpython/rev/aa2ac1581d23

Same issue on PPC Leopard custom:

[224/354] test_faulthandler
test test_faulthandler failed -- Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 412, in test_dump_tracebacks_later_file
    self.check_dump_tracebacks_later(file=True)
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 398, in check_dump_tracebacks_later
    self._check_dump_tracebacks_later(repeat, cancel, filename)
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 381, in _check_dump_tracebacks_later
    trace = self.get_output(code, True, filename)
  File "/Users/buildbot/buildarea/custom.parc-leopard-1/build/Lib/test/test_faulthandler.py", line 57, in get_output
    self.assertEqual(exitcode, 0)
AssertionError: 1 != 0

Reopen the issue until this test is fixed.

New changeset e91de993964c by Victor Stinner in branch 'default':
Issue #11393: check that stdout is empty if we use a file
http://hg.python.org/cpython/rev/e91de993964c

New changeset 61626c3f3a54 by Victor Stinner in branch 'default':
Issue #11393: get more information on assertion error (test_faulthandler)
http://hg.python.org/cpython/rev/61626c3f3a54

Other failures on "x86 XP-4 3.x":
http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/4308

======================================================================
FAIL: test_dump_tracebacks_later (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 404, in test_dump_tracebacks_later
    self.check_dump_tracebacks_later()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 401, in check_dump_tracebacks_later
    self._check_dump_tracebacks_later(repeat, cancel, None)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 390, in _check_dump_tracebacks_later
    self.assertRegex(trace, '^%s$' % regex)
AssertionError: Regex didn't match: '^^Thread 0x[0-9a-f]+:\n  File "\\<string\\>", line 7 in func\\n  File "\\<string\\>", line 30 in \\<module\\>$$' not found in 'Thread 0x000007cc:\n  File "<string>", line 7 in func\n  File "<string>", line 30 in <module>\nTraceback (most recent call last):\n  File "<string>", line 30, in <module>\n  File "<string>", line 11, in func\nAssertionError'

======================================================================
FAIL: test_dump_tracebacks_later_file (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 413, in test_dump_tracebacks_later_file
    self.check_dump_tracebacks_later(file=True)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 399, in check_dump_tracebacks_later
    self._check_dump_tracebacks_later(repeat, cancel, filename)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 391, in _check_dump_tracebacks_later
    self.assertEqual(exitcode, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_dump_tracebacks_later_repeat (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 407, in test_dump_tracebacks_later_repeat
    self.check_dump_tracebacks_later(repeat=True)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 401, in check_dump_tracebacks_later
    self._check_dump_tracebacks_later(repeat, cancel, None)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 390, in _check_dump_tracebacks_later
    self.assertRegex(trace, '^%s$' % regex)
AssertionError: Regex didn't match: '^^Thread 0x[0-9a-f]+:\n  File "\\<string\\>", line 7 in func\\n  File "\\<string\\>", line 30 in \\<module\\>\nThread 0x[0-9a-f]+:\n  File "\\<string\\>", line 7 in func\\n  File "\\<string\\>", line 30 in \\<module\\>$$' not found in 'Thread 0x00000fd8:\n  File "<string>", line 7 in func\n  File "<string>", line 30 in <module>\nThread 0x00000fd8:\n  File "<string>", line 7 in func\n  File "<string>", line 30 in <module>\nTraceback (most recent call last):\n  File "<string>", line 30, in <module>\n  File "<string>", line 11, in func\nAssertionError'

======================================================================
FAIL: test_dump_tracebacks_later_repeat_cancel (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 410, in test_dump_tracebacks_later_repeat_cancel
    self.check_dump_tracebacks_later(repeat=True, cancel=True)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 401, in check_dump_tracebacks_later
    self._check_dump_tracebacks_later(repeat, cancel, None)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_faulthandler.py", line 390, in _check_dump_tracebacks_later
    self.assertRegex(trace, '^%s$' % regex)
AssertionError: Regex didn't match: '^^Thread 0x[0-9a-f]+:\n  File "\\<string\\>", line 7 in func\\n  File "\\<string\\>", line 30 in \\<module\\>\nThread 0x[0-9a-f]+:\n  File "\\<string\\>", line 7 in func\\n  File "\\<string\\>", line 30 in \\<module\\>$$' not found in 'Thread 0x0000082c:\n  File "<string>", line 7 in func\n  File "<string>", line 30 in <module>\nThread 0x0000082c:\n  File "<string>", line 7 in func\n  File "<string>", line 30 in <module>\nTraceback (most recent call last):\n  File "<string>", line 30, in <module>\n  File "<string>", line 11, in func\nAssertionError'


The 3 assertion errors mean that time.sleep(pause) takes less than pause seconds. 61626c3f3a54 adds more information about these failure.

test_dump_tracebacks_later_file() failure is the same than the PPC Leopard failure.

New changeset 69f58be4688a by Victor Stinner in branch 'default':
Issue #11393: test_faulthandler is more tolerant on inaccurate time
http://hg.python.org/cpython/rev/69f58be4688a

Cool, 69f58be4688a fixed all failures (all related to dump_backtraces_later()) on "x86 XP-4 3.x". So "PPC Leopard" should also be fixed.

New changeset 8b1341d51fe6 by Victor Stinner in branch 'default':
Issue #11393: Fix faulthandler_thread(): release cancel lock before join lock
http://hg.python.org/cpython/rev/8b1341d51fe6

> New changeset 8b1341d51fe6 by Victor Stinner in branch 'default':
> Issue #11393: Fix faulthandler_thread(): release cancel lock before join lock
> http://hg.python.org/cpython/rev/8b1341d51fe6

This is wrong, it should always be released, not only when explicitly
cancelled.

New changeset 0fb0fbd442b4 by Victor Stinner in branch 'default':
Issue #11393: New try to fix faulthandler_thread()
http://hg.python.org/cpython/rev/0fb0fbd442b4

New changeset 3558eecd84f0 by Victor Stinner in branch 'default':
Issue #11393: fix usage of locks in faulthandler
http://hg.python.org/cpython/rev/3558eecd84f0

The last commits fixed test_faulthandler on FreeBSD, there are no more faulthandler bugs. Close this issue gain.

New changeset e51d8a160a8a by Victor Stinner in branch 'default':
Issue #11393: fault handler uses raise(signum) for SIGILL on Windows
http://hg.python.org/cpython/rev/e51d8a160a8a

New changeset a4fa79b0d478 by Victor Stinner in branch 'default':
Issue #11393: The fault handler handles also SIGABRT
http://hg.python.org/cpython/rev/a4fa79b0d478

New changeset a27755b10448 by Victor Stinner in branch 'default':
Issue #11393: Fix faulthandler.disable() and add a test
http://hg.python.org/cpython/rev/a27755b10448

New changeset 7e3ed426962f by Victor Stinner in branch 'default':
Issue #11393: _Py_DumpTraceback() writes the header even if there is no frame
http://hg.python.org/cpython/rev/7e3ed426962f

New changeset e609105dff64 by Victor Stinner in branch 'default':
Issue #11393: signal of user signal displays tracebacks even if tstate==NULL
http://hg.python.org/cpython/rev/e609105dff64