Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raising an exception that cannot be unpickled causes hang in ProcessPoolExecutor #69088

Open
filmor mannequin opened this issue Aug 20, 2015 · 4 comments
Open

Raising an exception that cannot be unpickled causes hang in ProcessPoolExecutor #69088

filmor mannequin opened this issue Aug 20, 2015 · 4 comments

Comments

@filmor
Copy link
Mannequin

filmor mannequin commented Aug 20, 2015

BPO 24900
Nosy @pitrou, @avassalotti, @agronholm, @serhiy-storchaka

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-08-20.07:21:49.990>
labels = []
title = 'Raising an exception that cannot be unpickled causes hang in ProcessPoolExecutor'
updated_at = <Date 2017-08-30.23:21:01.986>
user = 'https://bugs.python.org/filmor'

bugs.python.org fields:

activity = <Date 2017-08-30.23:21:01.986>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2015-08-20.07:21:49.990>
creator = 'filmor'
dependencies = []
files = []
hgrepos = []
issue_num = 24900
keywords = []
message_count = 4.0
messages = ['248878', '248879', '249531', '250385']
nosy_count = 5.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'filmor', 'alex.gronholm', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue24900'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6']

@filmor
Copy link
Mannequin Author

filmor mannequin commented Aug 20, 2015

When raising an exception that is unpicklable in a worker process. This used to be the case for all exceptions that have a non-trivial constructor but this was fixed by http://bugs.python.org/issue1692335.

I now have the concrete problem with cx_Oracle in which the error type cx_Oracle._Error doesn't define tp_new which results in a

UnpicklingError: NEWOBJ class argument has NULL tp_new

I have already reported this fact at https://bitbucket.org/anthony_tuininga/cx_oracle/issues/23.

However, I think that the worker thread should either be able to recover from this (which should be quite easy) or, at the very least, fail cleanly. Currently, when this condition is triggered the worker processes are not killed alongside the parent.

There is an existing report and a patch in the backports project: agronholm/pythonfutures#30 (comment)

@filmor
Copy link
Mannequin Author

filmor mannequin commented Aug 20, 2015

Actually I think that this may consist of multiple issues:

  1. http://bugs.python.org/issue8323 (closed as wont-fix): multiprocessing.Queue should throw a sensible error if a Queue element can't be unpickled, however the user managed to create that
  2. The worker process should prevent unpicklable ResultItems to be put on the Queue
  3. The parent thread in ProcessPoolExecutor should handle the exceptions by setting its broken flag and at least trying to cleanly shut down

@filmor filmor mannequin changed the title Raising an exception that is unable to be unpickled causes hang in ProcessPoolExecutor Raising an exception that cannot be unpickled causes hang in ProcessPoolExecutor Aug 20, 2015
@serhiy-storchaka
Copy link
Member

The patch for bpo-22995 disallows default pickling objects with tp_new == NULL.

@filmor
Copy link
Mannequin Author

filmor mannequin commented Sep 10, 2015

That would touch 2, but as this is happening in the exception handling already that would need to be catched and handled by itself. However, one could then do without a "pickle -> unpickle" dance.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Status: No status
Development

No branches or pull requests

2 participants