classification
Title: Promise the long-time truth that `args=list` works
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, rhettinger, serhiy.storchaka, tim.peters
Priority: low Keywords: easy, patch

Created on 2021-11-06 04:02 by tim.peters, last changed 2021-11-11 03:22 by tim.peters.

Pull Requests
URL Status Linked Edit
PR 29437 closed rhettinger, 2021-11-06 05:01
Messages (4)
msg405846 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2021-11-06 04:02
A number of contexts allow specifying a tuple of arguments to be passed later to a function. The Thread constructor is a fine example, and happened to come up (again! for me) here today:

https://stackoverflow.com/questions/69858950/why-do-we-have-to-add-comma-in-args-in-python-multithreading/69859068

This often confuses especially newbies, because the function they intend to parallelize often takes only a single argument, and Python's syntax for a 1-element tuple actually _requires_ parentheses in the context of an argument list, with a naked trailing comma:

t = threading.Thread(target=access, args=(thread_number,))

It "looks weird" to people.

I'm not suggesting to change that, but instead to officially bless the workaround I've seen very often in real code: use a list instead.

t = threading.Thread(target=access, args=[thread_number])

Nobody scratches their head over what that means.

CPython's implementations typically couldn't care less what kind of sequence is used, and none that I'm aware of verify that it's specifically a tuple. The implementations just go on to do some simple variation of

    self.target(*self.args)

Tuple or list makes no real difference. I'm not really keen to immortalize the "any sequence type whatsoever that just happens to work" implementation behavior, but am keen to promise that a list specifically will work. A lot of code already relies on it.
msg405851 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-11-06 09:09
There is a difference if you modify the arguments list after creating a thread.

    args = [1]
    t = threading.Thread(target=access, args=args)
    args[0] = 2
    t.start()

Would it call access(1) or access(2)?
msg405867 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2021-11-06 16:58
Serhiy, we haven't documented such stuff, and, indeed, I've been burned by it but much more often in the case of multiprocessing.Process. But note that I'm SWAPPING the order of your last two lines. In the original, you mutated the argument _before_ starting any parallel work, so "of course" the new worker will see the mutation:

    def access(xs):
        print(xs)

    args = ([1],)
    t = multiprocessing.Process(target=access, args=args)
    t.start() # start parallel work before mutating
    args[0][0] = 2



Does that print [1] or [2]? Passing a tuple in no way prevents mutations to mutable objects the tuple contains.

When the docs are silent, "implementation defined" rules. Whether you use threading or multiprocessing in the altered example above, the result printed simply isn't defined - it's a race between the main thread doing the mutation and the "parallel part" accessing the mutated object.

This is subtler in the multiprocessing context, though, because the relevant "parallel part" is really the hidden thread that pickles the argument list to send to the worker. That effectively makes a deep copy. But it's still a race, just not one visible from staring at the Python code. In the threading case, no copies are made.
msg406145 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2021-11-11 03:22
Changed stage back to "needs patch", since Raymond appears to have closed his PR. Raymond, what's up with that?
History
Date User Action Args
2021-11-11 03:22:04tim.peterssetmessages: + msg406145
stage: patch review -> needs patch
2021-11-06 16:58:17tim.peterssetmessages: + msg405867
2021-11-06 09:09:32serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg405851
2021-11-06 05:12:19rhettingersetmessages: - msg405847
2021-11-06 05:01:09rhettingersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request27691
2021-11-06 04:44:44rhettingersetnosy: + rhettinger
messages: + msg405847
2021-11-06 04:02:32tim.peterssettitle: Promise that the long-time truth that `args=list` works -> Promise the long-time truth that `args=list` works
2021-11-06 04:02:01tim.peterscreate