classification
Title: stdout error at interpreter shutdown fails to return OS error status
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, georg.brandl, loewis, martin.panter, mkc, pitrou, python-dev, rbcollins, steve.dower, terry.reedy
Priority: normal Keywords: patch

Created on 2009-02-19 17:38 by mkc, last changed 2015-11-30 04:12 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
devfull.patch pitrou, 2010-08-04 23:35
finalize-error.patch martin.panter, 2014-12-20 08:55 review
finalize-error.v2.patch martin.panter, 2015-06-02 04:36 Py_Finalize() returns -1 review
Py_FinalizeEx.patch martin.panter, 2015-06-06 03:27 New API review
Py_FinalizeEx-120.patch martin.panter, 2015-08-31 02:41 Exit status 120 review
Messages (26)
msg82482 - (view) Author: Mike Coleman (mkc) Date: 2009-02-19 17:37
$ python2.5 -c 'print((1, 2, 3))' > /dev/full || echo error status
close failed: [Errno 28] No space left on device
$

The above sequence should also output 'error status' before the next
prompt.  That is, python should not be returning EXIT_SUCCESS in this
situation.
msg82484 - (view) Author: Mike Coleman (mkc) Date: 2009-02-19 17:43
Oops, I should have used the old python print syntax.  Nonetheless, the
behavior is the same:

$ python2.5 -c 'print 1, 2, 3' > /dev/full || echo error status
close failed: [Errno 28] No space left on device
$
msg112342 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-08-01 15:06
Python 2.6, from #5320:

$ python2.6 -c 'print 1, 2, 3' > /dev/full || echo error status
close failed in file object destructor:
Error in sys.excepthook:

Original exception was:
$

Python 3.1 doesn't output anything and also doesn't set an error status.
msg112919 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-04 23:04
Georg, are you saying that there is or is not a problem with 2.6 (which is beyond non-critical bug fixes) and/or 3.1?
msg112923 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-04 23:35
Here is a possible patch for 3.x.
It doesn't change the return status of the process, though.
msg113330 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-08 21:40
I've committed the py3k fix in r83854 (3.1 in r83855).
As for 2.6/2.7, I've finally diagnosed the problem: 2.6 and upwards use PyErr_Print() in the file destructor, but the sys.stderr and sys.excepthook objects are already None at that time (because of interpreter shutdown). Therefore the error can't be printed out.
2.5 and earlier used a direct fprintf() of the errno code to stderr.

I propose to ignore the 2.6/2.7 issue of the error not being printed properly, since it only happens at shutdown with a misbehaving stdout, which is not very common.

As for the issue of the stdout error not being reported in the process return code, this would be possible in 3.x, although it would involve changing the signature of Py_Finalize() to include a return status (it currently returns void).
msg148423 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-26 21:11
New changeset 150e096095e5 by Antoine Pitrou in branch '3.2':
Issue #13444: When stdout has been closed explicitly, we should not attempt to flush it at shutdown and print an error.
http://hg.python.org/cpython/rev/150e096095e5

New changeset 37300a1df7d7 by Antoine Pitrou in branch 'default':
Issue #13444: When stdout has been closed explicitly, we should not attempt to flush it at shutdown and print an error.
http://hg.python.org/cpython/rev/37300a1df7d7
msg232967 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-12-20 08:55
Here is a patch that changes Py_Finalize() to return -1 on error, and then sets the exit status to 1. It did not introduce any failures in the test suite for me. However I suspect it deserves more consideration about backwards compatibility etc, which is beyond my knowledge.
msg244647 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-02 04:36
Updated patch resolving minor merge conflicts with current code, a coding style fix, and some tweaks and syntax fixes to the documentation.
msg244662 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-06-02 10:08
I don't know how acceptable it is to add a return value to Py_Finalize(). Can it break the stable ABI?
msg244666 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-06-02 13:11
Going from void to int is fine for when the return value is ignored. However, those who check the return value will see garbage on earlier versions.

As always, safest to add a new function here. Or make the stable version continue to be void - it won't affect the exports.
msg244812 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-04 11:36
I guess this would involve:

* Making a new API called Py_Finalize2() or something that returns the status
* Redefine the existing Py_Finalize() to call Py_Finalize2() and ignore the return value
msg244817 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-06-04 12:41
We would probably call it Py_FinalizeEx(), but yes.
msg244891 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-06 03:27
Here is a patch adding a new Py_FinalizeEx() API. Please let me know if anything else needs to be added. Perhaps some sort of Py_LIMITED_API condition, adding to PC/python3.def, etc. I am not really familiar with these things.

I also changed the version notices to 3.6, assuming it is too late for 3.5.
msg248628 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-15 00:47
I think you need to make the following changes:

 - Py_Exit with a non-zero status code should probably preserve the code rather than replacing it with 1.
 - Ditto in Py_Main.
 - Add a defs entry for Py_FinalizeEx - should be a copy of the Py_Finalize entry with the return type modified.
msg249183 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-08-26 07:17
Thanks for the hints Robert. I will definitely add that defs entry.

I’m a bit unsure about preserving an existing non-zero exit status. It seems a slightly more complicated and unexpected way of do it to me. But I cannot remember why I was originally interested in this issue. Do you have a use case where keeping the exit status > 1 would be good?

Also need to add some entries to What’s New.
msg249187 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-26 07:40
def main():
   try:
       thing()
   except SomethingChanged:
       exit(1)
   except BinaryDiff:
       exit(2)
   except:
       exit(3)

This is the sort of pattern e.g. 'diff' has. 

Should exit(3) there become exit(1) ? exit(1) means 'succeeded and there were changes'. So no, 3,for this app, means 'broke entirely'.

Exiting with '1' is also poor/confusing for this app. So perhaps we should pick something rarely used - like 255. But anyhow the same logic applies: if an app signals 'error' with a different code, we shouldn't change the apps 'error' signal to be something else when we detect an error.

But OTOH preserving a non-zero not-error code when an error occurs is also wrong.

So I think, on reflection:
 1) pick an unusual code. Not 1. Not 77 (automake test skip code). Maybe 255? https://www.freebsd.org/cgi/man.cgi?query=sysexits&sektion=3&apropos=0&manpath=FreeBSD+10.2-RELEASE defines some codes we should avoid as well.
 2) We document that python itself may cause this exit code to be used and that apps written in python should never use that code
 3) we use your patch as it is - on shutdown error force it to this chosen value.

What do you think?
msg249191 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-08-26 09:07
I guess I like setting the status to 1 because it matches the status already used by the default exception handling, the value when you raise SystemExit("error message"), and the value used for most internal errors. However I did discover that the exit status is 2 if you give Python wrong CLI arguments or there is an OS error opening a script file.

I’m still rather unsure, only weakly in favour of my original proposal. Maybe see if anyone else has an opinion?
msg249192 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-26 09:16
Right. So I think one could also propose that the default exception handling should also change to this new number, but it is a different situation - programs don't conflict with that - they allow it to bubble up, but this situation is outside its control.

CLI handling of Python itself is clearly outside both situations and there's no need to reevaluate it now.
msg249356 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-08-30 11:20
Maybe 255 is a reasonable choice. Another option is some value slightly less than 126, since Posix shells have special meanings for 126, 127, and > 128 values.

My patches already document the exit status used under the Py_Exit() function: “If :c:func:`Py_FinalizeEx` indicates an error, the exit status is set to 1 [or 255].” Do you think it should be documented somewhere else as well, perhaps for the interpreter as a whole?
msg249370 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-30 20:01
It seems to me that how, and when, Python exits the process is important to folk who will never ever touch the C API.
msg249377 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-08-31 01:47
So far I haven’t noticed much documentation on the exit statuses generated by Python itself (e.g. for unhandled exceptions). I did find some discussion in <https://docs.python.org/3/library/sys.html#sys.exit>, so perhaps I will add to that.

Since it implies values beyond 127 may not work on some systems, I now propose to use the arbitrary value of 120 instead of 255. (Keeping clear of 126 and 127, which have a special meaning in Posix shells.) Although, that statement about 0–127 was added in 1998 (bbe92932a005), so may not be relevant these days.
msg249378 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-08-31 02:41
Here is a new patch setting the exit status to 120. I also added a What’s New entry, and some more documentation under sys.exit().

Robert, I added an entry to /PC/python3.def, following the lead of your patch in Issue 2786. You mentioned modifying the return value, but in that file I don’t see any return values. Should I ignore that bit, or am I looking in the wrong place?
msg255585 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-11-29 19:03
@Martin I was wrong re: the defs - they only cover function vs data, not return codes. So it looks fine to me.
msg255605 - (view) Author: Roundup Robot (python-dev) Date: 2015-11-30 03:29
New changeset 6b08429a3932 by Martin Panter in branch 'default':
Issue #5319: New Py_FinalizeEx() API to exit with status 120 on failure
https://hg.python.org/cpython/rev/6b08429a3932
msg255606 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-30 04:12
Thanks everyone for helping to get this right.
History
Date User Action Args
2015-11-30 04:12:14martin.pantersetstatus: open -> closed
resolution: fixed
messages: + msg255606

stage: patch review -> resolved
2015-11-30 03:29:49python-devsetmessages: + msg255605
2015-11-29 19:03:14rbcollinssetmessages: + msg255585
2015-08-31 02:41:51martin.pantersetfiles: + Py_FinalizeEx-120.patch

messages: + msg249378
2015-08-31 01:47:24martin.pantersetmessages: + msg249377
2015-08-30 20:01:27rbcollinssetmessages: + msg249370
2015-08-30 11:20:46martin.pantersetmessages: + msg249356
2015-08-26 09:16:23rbcollinssetmessages: + msg249192
2015-08-26 09:07:42martin.pantersetmessages: + msg249191
2015-08-26 07:40:51rbcollinssetmessages: + msg249187
2015-08-26 07:17:20martin.pantersetmessages: + msg249183
2015-08-15 00:47:45rbcollinssetnosy: + rbcollins
messages: + msg248628
2015-06-06 03:27:23martin.pantersetfiles: + Py_FinalizeEx.patch

messages: + msg244891
versions: - Python 3.5
2015-06-04 12:41:44pitrousetmessages: + msg244817
2015-06-04 11:36:08martin.pantersetmessages: + msg244812
2015-06-02 13:11:19steve.dowersetmessages: + msg244666
2015-06-02 10:08:55pitrousetnosy: + loewis, steve.dower
messages: + msg244662
2015-06-02 04:36:56martin.pantersetfiles: + finalize-error.v2.patch

stage: patch review
messages: + msg244647
versions: + Python 3.5, Python 3.6, - Python 3.1, Python 3.2
2014-12-20 08:55:12martin.pantersetfiles: + finalize-error.patch

messages: + msg232967
2014-12-20 05:00:17martin.pantersetnosy: + martin.panter
2011-11-27 00:29:42Arfreversetnosy: + Arfrever
2011-11-26 21:11:58python-devsetnosy: + python-dev
messages: + msg148423
2010-08-08 22:06:04skrahlinkissue5321 superseder
2010-08-08 21:40:35pitrousettitle: I/O error during one-liner fails to return OS error status -> stdout error at interpreter shutdown fails to return OS error status
messages: + msg113330
versions: - Python 2.7
2010-08-04 23:35:06pitrousetfiles: + devfull.patch
versions: + Python 3.1, Python 3.2
nosy: + pitrou

messages: + msg112923

keywords: + patch
2010-08-04 23:04:13terry.reedysetnosy: + terry.reedy

messages: + msg112919
versions: + Python 2.7, - Python 2.5
2010-08-01 15:07:10georg.brandllinkissue5320 superseder
2010-08-01 15:06:53georg.brandlsetnosy: + georg.brandl
messages: + msg112342
2009-02-19 17:43:11mkcsetmessages: + msg82484
components: + Interpreter Core
2009-02-19 17:38:00mkccreate