This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: itertools.zip_longest behaves strangely with an iterable class
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: daniel.urban, georg.brandl, ocean-city, rhettinger
Priority: normal Keywords: patch

Created on 2009-10-31 09:07 by daniel.urban, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
zip_longest_test_case.py daniel.urban, 2009-10-31 09:07 Sample code, which shows this strange behavior
fix_izip_longest.patch ocean-city, 2009-10-31 10:12
improve_test_itertools.patch ocean-city, 2009-11-01 09:32
Messages (11)
msg94746 - (view) Author: Daniel Urban (daniel.urban) * (Python triager) Date: 2009-10-31 09:07
I'm trying to write an iterable class, and it behaves strangely with
itertools.zip_longest. The following example demonstrates this:

class Repeater: # this class is similar to itertools.repeat
   def __init__(self, o, t):
       self.o = o
       self.t = int(t)
   def __iter__(self): # its iterator is itself
       return self
   def __next__(self):
       if self.t > 0:
           self.t -= 1
           return self.o
       else:
           raise StopIteration

(Of course this is a trivial class, which could be substituted with
itertools.repeat, but I wanted to keep it simple for this example.)

The following code shows my problem:
>>> r1 = Repeater(1, 3)
>>> r2 = Repeater(2, 4)
>>> for i, j in zip_longest(r1, r2, fillvalue=0):
...     print(i, j)
...
1 2
1 2
1 2
0Traceback (most recent call last):
 File "<stdin>", line 2, in <module>
 File "zip_longest_test_case.py", line 30, in __next__
   raise StopIteration
StopIteration
>>>

It seems, that zip_longest lets through the StopIteration exception,
which it shouldn't. 

The strange thing is, that if I use the python implementation of
zip_longest, as it is in the documentation [1], I get the expected
result:

# zip_longest as it is in the documentation:
def zip_longest(*args, fillvalue=None):
   # zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-
   def sentinel(counter = ([fillvalue]*(len(args)-1)).pop):
       yield counter()         # yields the fillvalue, or raises IndexError
   fillers = repeat(fillvalue)
   iters = [chain(it, sentinel(), fillers) for it in args]
   try:
       for tup in zip(*iters):
           yield tup
   except IndexError:
       pass

Test code again:
>>> r1 = Repeater(1, 3)
>>> r2 = Repeater(2, 4)
>>> for i, j in zip_longest(r1, r2, fillvalue=0):
...     print(i, j)
...
1 2
1 2
1 2
0 2

I would think, that this is the expected behaviour.

Also, Matthew Dixon Cowles discovered, that if using list(), the C
implementation of itertools.zip_longest also works fine:

>>> r1=Repeater(1,3)
>>> r2=Repeater(2,5)
>>> list(itertools.zip_longest(r1,r2,fillvalue=0))
[(1, 2), (1, 2), (1, 2), (0, 2), (0, 2)]

This is strange, and I think it really shouldn't work this way. 
(Thanks for Matthew Dixon Cowles' help on the python-help mailing list.)

I'm attaching a test script, which tries all 4 variations (library
zip_longest with and without list(), and the documentation's zip_longest
impplementation with and without list()). 

And another thing: it works fine in 2.6.4 (with izip_longest).
msg94748 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-10-31 09:52
I saw strange thing with following code + release26-maint/trunk.

from itertools import *

class Repeater(object): # this class is similar to itertools.repeat
   def __init__(self, o, t):
       self.o = o
       self.t = int(t)
   def __iter__(self): # its iterator is itself
       return self
   def next(self):
       if self.t > 0:
           self.t -= 1
           return self.o
       else:
           raise StopIteration

r1 = Repeater(1, 3)
r2 = Repeater(2, 4)
for i, j in izip_longest(r1, r2, fillvalue=0):
    print(i, j)

Be care that Repeater is using new-style class. (it's default on py3k) I
couldn't see any problem with officially released windows binary, but I
could see following error with VC6 debug build.

(1, 2)
(1, 2)
(1, 2)
(0, 2)
XXX undetected error
Traceback (most recent call last):
  File "a.py", line 20, in <module>
    print(i, j)
  File "a.py", line 15, in next
    raise StopIteration
StopIteration

# Still there is possibility this is VC6 bug though.
msg94749 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-10-31 10:12
I hope an attached patch will fix this issue. (this patch is for trunk)
I think PyErr_Clear() is needed to clear StopIteration there.
msg94751 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-10-31 10:31
The patch is incorrect; tp_iternext can raise exceptions other than
StopIteration which must be let through.
msg94754 - (view) Author: Daniel Urban (daniel.urban) * (Python triager) Date: 2009-10-31 11:35
> I saw strange thing with following code + release26-maint/trunk.

I tried your code (with the new-style class) with Python 2.6.4 and 2.7a0
(trunk), and both worked fine. I built them with GCC 4.2.4 on Ubuntu 8.04.
msg94761 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-10-31 16:51
> I tried your code (with the new-style class) with Python 2.6.4 and 2.7a0
> (trunk), and both worked fine. I built them with GCC 4.2.4 on Ubuntu 8.04.

The problem seems to only show up in debug builds on 2.x, but it is there.
msg94763 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-10-31 17:13
I've got it from here.
msg94783 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-11-01 09:32
I created the patch to improve test which was checked in r76004. This
patch checks if correct elements are returned even when RuntimeError is
raised. Could you take a look?
msg94784 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-11-01 09:37
> correct elements are returned even when RuntimeError is
raised.

Or maybe it is not guaranteed. :-)
msg94799 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-11-01 18:48
Added r76018 to use Hirokazu's test for the RuntimeError case and to
redirect stdout to a file for the StopIteration case.  Also, fixed-up
weird indentation in the C code.
msg94806 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-11-01 21:04
Forward-ported in r76025 r76026 and r76027.
History
Date User Action Args
2022-04-11 14:56:54adminsetgithub: 51493
2009-11-01 21:04:15rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg94806
2009-11-01 18:48:15rhettingersetmessages: + msg94799
2009-11-01 18:46:37rhettingersetmessages: - msg94794
2009-11-01 18:46:27rhettingersetmessages: - msg94793
2009-11-01 17:51:59rhettingersetmessages: + msg94794
2009-11-01 17:41:02rhettingersetmessages: + msg94793
2009-11-01 09:37:40ocean-citysetmessages: + msg94784
2009-11-01 09:32:53ocean-citysetfiles: + improve_test_itertools.patch

messages: + msg94783
2009-10-31 17:13:57rhettingersetmessages: + msg94763
2009-10-31 16:51:14georg.brandlsetmessages: + msg94761
2009-10-31 11:35:02daniel.urbansetmessages: + msg94754
2009-10-31 10:31:02georg.brandlsetassignee: rhettinger

messages: + msg94751
nosy: + rhettinger, georg.brandl
2009-10-31 10:12:55ocean-citysetfiles: + fix_izip_longest.patch
keywords: + patch
messages: + msg94749

versions: + Python 2.6, Python 2.7, Python 3.2
2009-10-31 09:52:49ocean-citysetnosy: + ocean-city
messages: + msg94748
2009-10-31 09:07:57daniel.urbancreate