classification
Title: future_factory argument for Thread/ProcessPoolExecutor
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, bquinlan, pitrou, stefanhoelzl, tomMoral
Priority: normal Keywords: patch

Created on 2019-04-02 19:52 by stefanhoelzl, last changed 2019-05-11 11:26 by stefanhoelzl.

Pull Requests
URL Status Linked Edit
PR 12668 open stefanhoelzl, 2019-04-02 20:04
Messages (6)
msg339364 - (view) Author: Stefan Hölzl (stefanhoelzl) * Date: 2019-04-02 19:52
adding a future_factory argument to Thread/ProcessPoolExecutor to control which type of Future should be created by Executor.submit
msg341341 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-03 15:06
What is your use case?
msg342079 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-10 17:14
I'd like to know about the use case too :-)
msg342114 - (view) Author: Stefan Hölzl (stefanhoelzl) * Date: 2019-05-10 17:57
It would allow to use Futures with a customized interface for a specific domain.
e.g. to not only save the result of a task but also some context informations
or provide properties/methods which are result specific.

But when subclassing Future the builtin Thread/ProcessExecutor cannot be reused anymore, because
the returned class on submit cannot be customized to be the subclassed Future.

With my change it would be possible to customize the Future object returned by the Executor.

As it is now the Future class has to be wrapped and the Executor subclassed
to return a wrapped Future object. 
The Future object cannot be extended without completely wrapping it.

This change would make the Executor class more versatile.

It would allow something like this:

class CustomExecutor:
	...

custom_executor = CustomExecutor()
custom_future = custom_executor.submit(workload, context=context_information)
...
assert custom_future.context_has_some_property()
assert custom_future.result_has_some_property()


factories are also used in other places in standard library:
logging.setLogFactory
asyncio.loop.set_task_factory
msg342178 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-11 09:24
Sorry, but in the provided snippet I see a class with an interface which looks close to executor.

There is no clear relationship with standard executors (e.g. is it inherited from ThreadExecutor?). Proposed `future_factory` is also absent.

Please, provide a real example if you want to convince us that the requested feature is important and desired.
msg342184 - (view) Author: Stefan Hölzl (stefanhoelzl) * Date: 2019-05-11 11:26
Here is a complete example.
Please consider this is a very simple example just to demonstrate what would be possible with my proposed changes.

In the following example you can see two things:
 * saving informations about the context in which a workload was submitted to the Executor in the Future
 * having a custom methods on the Future returned by the Executor

from concurrent.futures import Future, ThreadPoolExecutor


def workload(n):
    # does some heavy calculations
    return n


class Context:
    def __init__(self, name):
        self.name = name


class CustomFactory(Future):
    def __init__(self):
        super().__init__()
        self.context = None

    def is_from_context(self, ctx):
        return self.context.name == ctx

    def processed_result(self):
        # processes the result
        return self.result()


class ContextExecutor(ThreadPoolExecutor):
    def __init__(self):
        super().__init__(future_factory=CustomFactory)

    def submit(self, workload, arg, context):
        future = super().submit(workload, arg)
        future.context = context
        return future


context_executor = ContextExecutor()
futures = [
    context_executor.submit(workload, 0, context=Context("A")),
    context_executor.submit(workload, 1, context=Context("B")),
    context_executor.submit(workload, 2, context=Context("A")),
]

futures_from_context_a = [f for f in futures if f.is_from_context("A")]
if all(f.done() for f in futures_from_context_a):
    print("All Futures in context A are done")
    processed_results = [f.processed_result() for f in futures_from_context_a]
    print("Results:", processed_results)
History
Date User Action Args
2019-05-11 11:26:15stefanhoelzlsetmessages: + msg342184
2019-05-11 09:24:48asvetlovsetmessages: + msg342178
2019-05-10 17:57:31stefanhoelzlsetmessages: + msg342114
2019-05-10 17:14:59pitrousetmessages: + msg342079
2019-05-10 17:14:42pitrousetnosy: + tomMoral
2019-05-03 15:06:24asvetlovsetnosy: + asvetlov
messages: + msg341341
2019-04-03 03:36:56xtreaksetnosy: + bquinlan, pitrou
2019-04-02 20:04:34stefanhoelzlsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12596
2019-04-02 19:52:53stefanhoelzlcreate