Running the file couple of times will make the interpreter fail with: libgcc_s.so.1 must be installed for pthread_cancel to work

From what I've seen it is triggered from PyThread_delete_key (tries to load libgcc_s.so at that time).

How does it happen?

The main thread will close fd 4 (because fh object is getting dereferenced to 0) exactly at the same time libpthread will try to open and read libgcc_s.so with the same descriptor (4)

It's fairly obvious that the file handling in bug.py is a bug, but the interpreter should not crash like that !

This doesn't happen on python2.7. Also, python2.7 appears to be linked with libgcc_s.so.1 directly while the 3.x does not (I've tried 3.2 from ubuntu repos, and built 3.3 and 3.4 myself on ubuntu 12.04.2) - at least that's what ldd indicates.

Unfortunately, there's not much we can do about it: if dlsym() fails - which is the case here either because read() fails with EBADF, or because the file descriptor now points to another stream (i.e. not libgcc), the libc aborts (here upon pthread_exit(), not PyThread_delete_key()):
"""
libgcc_s.so.1 must be installed for pthread_cancel to work

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb7b0eb70 (LWP 17152)]
0xb7fe1424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fe1424 in __kernel_vsyscall ()
#1  0xb7e4e941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb7e51d72 in *__GI_abort () at abort.c:92
#3  0xb7e8ae15 in __libc_message (do_abort=1, fmt=0xb7f606f5 "%s") at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb7e8af44 in *__GI___libc_fatal (message=0xb7fc75ec "libgcc_s.so.1 must be installed for pthread_cancel to work\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:200
#5  0xb7fc4ffa in pthread_cancel_init () at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:65
#6  0xb7fc509d in _Unwind_ForcedUnwind (exc=0xb7b0edc0, stop=0xb7fc2bf0 <unwind_stop>, stop_argument=0xb7b0e454) at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:126
#7  0xb7fc2b98 in *__GI___pthread_unwind (buf=<optimized out>) at unwind.c:130
#8  0xb7fbcce0 in __do_cancel () at pthreadP.h:265
#9  __pthread_exit (value=0x0) at pthread_exit.c:30
#10 0x08132ced in PyThread_exit_thread () at Python/thread_pthread.h:266
#11 0x08137c37 in t_bootstrap (boot_raw=0x8318aa8) at ./Modules/_threadmodule.c:1023
#12 0xb7fbbc39 in start_thread (arg=0xb7b0eb70) at pthread_create.c:304
#13 0xb7ef978e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
"""

So if you're unlucky and end up closing the FD referring to libgcc used by dlopen/dlsym, you're pretty much screwed, and there's no way around this.

Note that you specific problem (upon PyThread_delete_key()) doesn't occur for python 2.7 because it uses and ad-hoc TLS implementation, whereas Python 3 uses the platform native TLS.
But as noted above, you can very well get an abort on pthread_exit(), and in likely many other places (pretty much everywhere libgcc can be dlopen'ed).

Unfortunately, we can't do much against this, so I'm tempted to close this as "wont fix".

Perhaps the only thing we could do would be try to "preload" libgcc by calling one of those APIs at startup? But I'm not sure it's a good idea to add such a platform-specific hack (for what is arguably an obscure and rare issue).

Maries Ionel Cristian> ubuntu 12.04.2

What is the version of your libc library? Try something like "dpkg -l libc6".

--

Could this issue be related to this glibc issue?
http://sourceware.org/bugzilla/show_bug.cgi?id=2644

I ran your script on Python 3.4 in a shell loop: I'm unable to reproduce the issue. I'm running Fedora 18 with the glibc 2.16.

Oh ok, I'm able to reproduce the issue with the system Python 3.3:

$ while true; do echo "loop"; python3.3 bug.py || break; done
loop
...
loop
libgcc_s.so.1 must be installed for pthread_cancel to work
Abandon (core dumped)

> Perhaps the only thing we could do would be try to "preload" libgcc by
> calling one of those APIs at startup?

Yeah, I was thinking about doing this in PyThread_init_thread() but...

> But I'm not sure it's a good idea to
> add such a platform-specific hack (for what is arguably an obscure and rare
> issue).

then thought the exact same thing.

Also, this exact problem can possibly show up anywhere dlopen() is
used (a grep through glibc code shows at least an occurrence in
nsswitch-related code).

> What is the version of your libc library? Try something like "dpkg -l
> libc6".
>

 2.15-0ubuntu10.4

I don't think it's that obscure ... uwsgi has this issue
https://www.google.com/search?q=libgcc_s.so.1+must+be+installed+for+pthread_cancel+to+work+uwsgi+site:lists.unbit.it-
they cause it probably different but the point is that it's not
obscure.

Correct link https://www.google.com/search?q=libgcc_s.so.1+must+be+installed+for+pthread_cancel+to+work+uwsgi+site:lists.unbit.it

2013/8/17 Maries Ionel Cristian <report@bugs.python.org>:
> I don't think it's that obscure ... uwsgi has this issue
> https://www.google.com/search?q=libgcc_s.so.1+must+be+installed+for+pthread_cancel+to+work+uwsgi+site:lists.unbit.it-
> they cause it probably different but the point is that it's not
> obscure.

The error message is maybe the same, the reason is completly different:
http://lists.unbit.it/pipermail/uwsgi/2013-July/006213.html
"IT WAS A limit-as issue"

The root cause is an arbitrary limit on the size of the virtual memory.

Here the bug is that a thread B closes a file descriptor opened in a
thread B by the glibc.

Well anyway, is there any way to preload libgcc ? Because in python2.x it wasn't loaded at runtime.

2013/8/18 Maries Ionel Cristian <report@bugs.python.org>:
> Well anyway, is there any way to preload libgcc ? Because in python2.x it wasn't loaded at runtime.

On Linux, you can try to set the LD_PRELOAD environment variable as a
workaround.

LD_PRELOAD=libgcc_s.so.1 python bug.py

You may need to specify the full path.

Alright ... would it be a very big hack to preload libgcc in the thread module (at import time) ? There is platform specific code there anyway, it wouldn't be such a big deal would it?

> On Linux, you can try to set the LD_PRELOAD environment variable as a
> workaround.
>
> LD_PRELOAD=libgcc_s.so.1 python bug.py
>
> You may need to specify the full path.

I don't think that'll work.
Despite its name, using LD_PRELOAD won't "preload" the library. It
will only be loaded upon dlopen(). It just makes sure that symbols
will be looked for in this library first, even before the libc.

> Because in python2.x it wasn't loaded at runtime.

Yes it was. As explained above, you can get the very same crash upon
pthread_exit().

> Alright ... would it be a very big hack to preload libgcc in the thread module (at import
> time) ?

IMO yes, but if someone writes a patch, I won't oppose to it :-)

I think this is the same issue I'm getting.
I'm hitting it when compiling python 3.6.2 compiled with gcc-4.8.4.
This wasn't occasional, it was every time I tried.
As a feeble workaround I was compiling in parallel, then in serial when it fails, although I'm now hitting the same issue when building uwsgi-2.0.15.

I'm building on a 56 core system so that might make me more likely to hit the issue.

Looks like I was mistaken. My cross compiler was trying to load libgcc_s.so.1 from the standard location and not liking the one it found. Fixed for now by setting LD_LIBRARY_PATH to point at the dir containing the right libgcc_s.so.1

I meant my cross compiled python, not my cross compiler.

I have observed this problem in a production application using Python 3.5 and 3.6 (system-packaged interpreters from Ubuntu) and I would like to mention that this is an effective workaround:

```
import ctypes
libgcc_s = ctypes.CDLL("libgcc_s.so.1")
```

provided you do it before creating any threads, and the variable `libgcc_s` remains live until after all threads are terminated.

I looked at this old issue. First of all, "bug.py" filename is accurate: this program contains a bug, but it's non-obvious.

* the main thread opens a file which gets the file descriptor 3
* a new thread is spawns which indirectly closes this file descriptor on: sys.stdout.close()
* the main thread is not aware that the file has been closed, and so will try to close the same file descriptor 3 sometime during Python shutdown
* then, sometimes, the glibc fails with an internal error because the main thread closed a file descriptor which was just opened by the other thread to implement thread cancellation: that logs "libgcc_s.so.1 must be installed for pthread_cancel to work" and calls abort()

Note: the Python file object remains alive until the "t" local variable is cleared, since "t.fh" contains a strong reference to the file object.

Again: it's a bug in the program. "libgcc_s.so.1 must be installed for pthread_cancel to work" is just *one* example of bad thing that can happen.

What can be do on the Python side? Well, stop to ignore silently I/O errors when a file is closed by its destructor.

I wrote a shy PR 12786 to only log these exceptions in development mode (-X dev). I'm not sure if we could stop to silence these exceptions *by default*.

Python has been modified last years to first raise an exception on I/O errors when file.close() is called explicitly, and then a similar change has been done for call to socket.socket.close().

New changeset 44235041f3b957abd36d3792450c3540aa09e120 by Victor Stinner in branch 'master':
bpo-18748: io.IOBase destructor now logs close() errors in dev mode (GH-12786)
https://github.com/python/cpython/commit/44235041f3b957abd36d3792450c3540aa09e120

To debug remaining "Exception ignored in:" issues, I'm using the following patch:

diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
index 972a4658b1..be38af3daa 100644
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -710,6 +710,7 @@ class TestCase(object):
 
             # clear the outcome, no more needed
             self._outcome = None
+            import gc; gc.collect()
 
     def doCleanups(self):
         """Execute all cleanup functions. Normally called for you after

And I run:

./python -X dev -u -m test test_io -v 2>&1|tee log

I wrote PR 12805 to silence IOBase destructor exceptions in test_io.

Is there someone interested to debug remaining "Exception ignored:" logs in test_urllib?

test_invalid_redirect (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24be0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
ok

test_read_bogus (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24cd0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
ok

test_redirect_limit_independent (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24cd0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24c30>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24460>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f24a50>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f247d0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52ea95f0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52ea99b0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f2f820>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f2fbe0>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <http.client.HTTPResponse object at 0x7fca52f2fd70>
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in close
    super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.
ok


It's unclear to be if it's a bug in http.client, a bug in the test... or something else.

Maybe the fact that close() exceptions are ignored silently in io.IOBase constructor should be better documented, but I'm not sure where it should be documented. If someone has an idea, please go ahead and write a pull request :-)

New changeset 472f794a33221ea835a2fbf6c9f12aa2bd66d1b0 by Victor Stinner in branch 'master':
bpo-18748: test_io: silence destructor errors (GH-12805)
https://github.com/python/cpython/commit/472f794a33221ea835a2fbf6c9f12aa2bd66d1b0

The ValueError warnings in test_urllib noted in msg340059  feels like an issue with the test where FakeSocket calls close by itself once it's internal io_refs counter is 0 and during destructor again close is called on an already closed object causing the ValueError.

* The test uses fakehttp() creating FakeSocket starting with io_refs as 1 [1]
* During the test in urlopen, sock.makefile() (io_refs += 1) is called increasing the io_refs to be 2.
* HTTPConnection.close calls sock.close (fakesocket.close) [3] calling io_refs -= 1 and to fakesocket.io_ref to be 1.
* This test uses raises status 302 with http_error_302 which calls self.redirect_internal where self.close is again called causing fp.io_refs == 0 and subsequently calling io.BytesIO.close(self) to close the object. [4]
* During the end of the test destructor is called on the above closed fakesocket object and trying to flush on a closed object causes the ValueError warnings .

Maybe a check could be added during flush to make sure the object is not closed by testing for self.closed but I am not sure if closed attribute is guaranteed to be present. Removing the manual call to close in fakesocket could help since the destructor should be taking care of it and I could see no test failures or warnings removing the close as io_refs gets to 0.

[1] https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/test/test_urllib.py#L61
[2] https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/test/test_urllib.py#L67
[3] https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/http/client.py#L919
[4] https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/urllib/request.py#L2145

I was wrong so removing the close call when io_refs gets to zero causes error on test_urllib2 [0] where close is asserted to be True and I only tested test_urllib initially. One another way would be to check for fp.closed in http.client before flushing but as mentioned before I am not sure of the attribute being always present and also to modify http.client code just for tests.

[0] https://github.com/python/cpython/blob/f12ba7cd0a7370631214ac0b337ab5455ce717b2/Lib/test/test_urllib2.py#L1685

Karthikeyan Singaravelan: Would you mind to open a separated issue for test_urllib warnings?

Okay, opened issue36918 . Raising PR against that issue.

New changeset bc2aa816620c5e02ad8e94d8514b7e8f3f551ca1 by Victor Stinner in branch 'master':
bpo-18748: _pyio.IOBase emits unraisable exception (GH-13512)
https://github.com/python/cpython/commit/bc2aa816620c5e02ad8e94d8514b7e8f3f551ca1

Related issue: bpo-37054 "Ignored exceptions in test_memoryio".

I created bpo-37223: test_io logs Exception ignored in: <function IOBase.__del__ at 0x7f2f3c73e4b0> warnings.

New changeset 4f6f7c5a611905fb6b81671547f268c226bc646a by Victor Stinner in branch 'master':
bpo-18748: Fix _pyio.IOBase destructor (closed case) (GH-13952)
https://github.com/python/cpython/commit/4f6f7c5a611905fb6b81671547f268c226bc646a

New changeset c15a682603a47f5aef5025f6a2e3babb699273d6 by Victor Stinner in branch '3.8':
bpo-37223: test_io: silence destructor errors (GH-14031)
https://github.com/python/cpython/commit/c15a682603a47f5aef5025f6a2e3babb699273d6

I closed the issue. I consider that the initial issue has been fixed and all related issues have been fixed as well:

* Python 3.8 now logs close() exception in file finalizer in development mode (-X dev) and debug mode (./configure --with-pydebug)
* I added sys.unraisablehook to Python 3.8 which allows to decide how these unraisable exceptions are handled
* Tests no longer log any "unraisable exception" in file finalizers (test_io and test_urllib have been fixed)
* regrtest now marks a test as ENV_CHANGED if it logs an unraisable exception (like the close() exception)

--

For Python 3.7, there is no known workaround.

It seems like Python 2.7 in debug mode logs an error:

$ python2.7-dbg bug.py 
close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor

Same problem using Python 3.6.8 on Ubuntu 18.04 LTS.

For now, solve this using

LD_PRELOAD=libgcc_s.so.1 python3 ...

For more details and pocs: https://github.com/WHK102/wss/issues/2

> Same problem using Python 3.6.8 on Ubuntu 18.04 LTS.

To be clear: this issue is NOT a bug in Python, but a bug in your application. You must fix your application. You can try to run it on Python 3.8 with python3.8 -X dev to get a log on the error. Good luck to debug it ;-)

The change that I made is only to add a log to help developers to find their own bugs ;-)

> To be clear: this issue is NOT a bug in Python

I don't think that's entirely true.  I think CPython needs to be linked against libgcc_s.so, so that this class of application bugs will no longer manifest as interpreter crashes.  I filed #37395 for that.

I hope that when an error occurs, python tells me what the problem is. The abort core error is a problem at a lower level than python because python is not able to recognize or handle the error.

The main problem is that I exceeded the maximum number of process threads supported by the kernel.

What I hope is that python throws an exception when it exceeds this limit or when it cannot access the pointer in memory of the process thread.

The problem is not if the script is right or wrong, but that Python is not able to recognize and handle the problem. A generic message saying that an error occurred without indicating where and how it occurred is a python bug.

> Running the file couple of times will make the interpreter fail with: libgcc_s.so.1 must be installed for pthread_cancel to work

See also bpo-44434: "_thread module: Remove redundant PyThread_exit_thread() call to avoid glibc fatal error: libgcc_s.so.1 must be installed for pthread_cancel to work".