classification
Title: Weird behavior with generators with self-referencing output.
Type: behavior Stage:
Components: Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: PyryP, amaury.forgeotdarc
Priority: normal Keywords:

Created on 2011-12-13 19:25 by PyryP, last changed 2011-12-14 02:09 by PyryP. This issue is now closed.

Messages (3)
msg149403 - (view) Author: Pyry Pakkanen (PyryP) Date: 2011-12-13 19:25
The following self-referencing generator has incorrect output:

def ab_combinations():
    #'', 'a', 'b', 'aa', 'ab', 'ba', 'bb', 'aaa', ...
    def _deferred_output():
        yield ""
        tees = tee(output)

        #This definition works fine: '', 'a', 'b', 'aa', 'ab', 'ba', ...
        l = [(item+"a" for item in tees[0]), (item+"b" for item in tees[1])]

        #This definition results in: '', 'b', 'b', 'bb', 'bb', 'bb', ...
        #l = [(item+label for item in t) for t, label in zip(tees,"ab")]
        
        while True:
            for g in l:
                yield next(g)

    result, output = tee(_deferred_output())
    return result
msg149412 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-12-13 22:33
This is expected, and is due to the late binding of the "label" variable in the "item+label" expression.  Look at the example below:

>>> l = [lambda item: item + label for label in "ab"]
>>> f1, f2 = l
>>> print f1(''), f2('')
b b

For the lambda function, 'label' is a free variable, whose value is fetched from the global environment at runtime.  But at the time the second time is executed, 'label' has only one value, the last one from the for loop, and both functions only see 'b'.
A Generator Expression also defines a code block, and the late binding also applies.

Now to fix your code, if 'ab' can be of arbitrary length, I can't find a simpler way without yet another inline function:

l = [(lambda lbl:(item + lbl for item in t))(label) for t, label in zip(tees,"ab")]

'lbl' is still a free variable, but now 'lbl' comes from the lambda function, and there is one *different* *function* per iteration of the loop; so they are in fact distinct 'lbl' variables, and the generator expressions will correctly see different labels.
msg149416 - (view) Author: Pyry Pakkanen (PyryP) Date: 2011-12-14 02:09
Oh, I was sure it had to do with binding issues. I just couldn't put my finger on it because the behavior seemed so counterintuitive.
Thanks for clearing things up. I can now work around this feature.
History
Date User Action Args
2011-12-14 02:09:43PyryPsetmessages: + msg149416
2011-12-13 22:33:08amaury.forgeotdarcsetstatus: open -> closed

nosy: + amaury.forgeotdarc
messages: + msg149412

resolution: works for me
2011-12-13 19:25:39PyryPcreate