Issue7244
Created on 2009-10-31 09:07 by durban, last changed 2009-11-01 21:04 by rhettinger.
|
msg94746 - (view) |
Author: Daniel Urban (durban) |
Date: 2009-10-31 09:07 |
|
I'm trying to write an iterable class, and it behaves strangely with
itertools.zip_longest. The following example demonstrates this:
class Repeater: # this class is similar to itertools.repeat
def __init__(self, o, t):
self.o = o
self.t = int(t)
def __iter__(self): # its iterator is itself
return self
def __next__(self):
if self.t > 0:
self.t -= 1
return self.o
else:
raise StopIteration
(Of course this is a trivial class, which could be substituted with
itertools.repeat, but I wanted to keep it simple for this example.)
The following code shows my problem:
>>> r1 = Repeater(1, 3)
>>> r2 = Repeater(2, 4)
>>> for i, j in zip_longest(r1, r2, fillvalue=0):
... print(i, j)
...
1 2
1 2
1 2
0Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "zip_longest_test_case.py", line 30, in __next__
raise StopIteration
StopIteration
>>>
It seems, that zip_longest lets through the StopIteration exception,
which it shouldn't.
The strange thing is, that if I use the python implementation of
zip_longest, as it is in the documentation [1], I get the expected
result:
# zip_longest as it is in the documentation:
def zip_longest(*args, fillvalue=None):
# zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-
def sentinel(counter = ([fillvalue]*(len(args)-1)).pop):
yield counter() # yields the fillvalue, or raises IndexError
fillers = repeat(fillvalue)
iters = [chain(it, sentinel(), fillers) for it in args]
try:
for tup in zip(*iters):
yield tup
except IndexError:
pass
Test code again:
>>> r1 = Repeater(1, 3)
>>> r2 = Repeater(2, 4)
>>> for i, j in zip_longest(r1, r2, fillvalue=0):
... print(i, j)
...
1 2
1 2
1 2
0 2
I would think, that this is the expected behaviour.
Also, Matthew Dixon Cowles discovered, that if using list(), the C
implementation of itertools.zip_longest also works fine:
>>> r1=Repeater(1,3)
>>> r2=Repeater(2,5)
>>> list(itertools.zip_longest(r1,r2,fillvalue=0))
[(1, 2), (1, 2), (1, 2), (0, 2), (0, 2)]
This is strange, and I think it really shouldn't work this way.
(Thanks for Matthew Dixon Cowles' help on the python-help mailing list.)
I'm attaching a test script, which tries all 4 variations (library
zip_longest with and without list(), and the documentation's zip_longest
impplementation with and without list()).
And another thing: it works fine in 2.6.4 (with izip_longest).
|
|
msg94748 - (view) |
Author: Hirokazu Yamamoto (ocean-city) |
Date: 2009-10-31 09:52 |
|
I saw strange thing with following code + release26-maint/trunk.
from itertools import *
class Repeater(object): # this class is similar to itertools.repeat
def __init__(self, o, t):
self.o = o
self.t = int(t)
def __iter__(self): # its iterator is itself
return self
def next(self):
if self.t > 0:
self.t -= 1
return self.o
else:
raise StopIteration
r1 = Repeater(1, 3)
r2 = Repeater(2, 4)
for i, j in izip_longest(r1, r2, fillvalue=0):
print(i, j)
Be care that Repeater is using new-style class. (it's default on py3k) I
couldn't see any problem with officially released windows binary, but I
could see following error with VC6 debug build.
(1, 2)
(1, 2)
(1, 2)
(0, 2)
XXX undetected error
Traceback (most recent call last):
File "a.py", line 20, in <module>
print(i, j)
File "a.py", line 15, in next
raise StopIteration
StopIteration
# Still there is possibility this is VC6 bug though.
|
|
msg94749 - (view) |
Author: Hirokazu Yamamoto (ocean-city) |
Date: 2009-10-31 10:12 |
|
I hope an attached patch will fix this issue. (this patch is for trunk)
I think PyErr_Clear() is needed to clear StopIteration there.
|
|
msg94751 - (view) |
Author: Georg Brandl (georg.brandl) |
Date: 2009-10-31 10:31 |
|
The patch is incorrect; tp_iternext can raise exceptions other than
StopIteration which must be let through.
|
|
msg94754 - (view) |
Author: Daniel Urban (durban) |
Date: 2009-10-31 11:35 |
|
> I saw strange thing with following code + release26-maint/trunk.
I tried your code (with the new-style class) with Python 2.6.4 and 2.7a0
(trunk), and both worked fine. I built them with GCC 4.2.4 on Ubuntu 8.04.
|
|
msg94761 - (view) |
Author: Georg Brandl (georg.brandl) |
Date: 2009-10-31 16:51 |
|
> I tried your code (with the new-style class) with Python 2.6.4 and 2.7a0
> (trunk), and both worked fine. I built them with GCC 4.2.4 on Ubuntu 8.04.
The problem seems to only show up in debug builds on 2.x, but it is there.
|
|
msg94763 - (view) |
Author: Raymond Hettinger (rhettinger) |
Date: 2009-10-31 17:13 |
|
I've got it from here.
|
|
msg94783 - (view) |
Author: Hirokazu Yamamoto (ocean-city) |
Date: 2009-11-01 09:32 |
|
I created the patch to improve test which was checked in r76004. This
patch checks if correct elements are returned even when RuntimeError is
raised. Could you take a look?
|
|
msg94784 - (view) |
Author: Hirokazu Yamamoto (ocean-city) |
Date: 2009-11-01 09:37 |
|
> correct elements are returned even when RuntimeError is
raised.
Or maybe it is not guaranteed. :-)
|
|
msg94799 - (view) |
Author: Raymond Hettinger (rhettinger) |
Date: 2009-11-01 18:48 |
|
Added r76018 to use Hirokazu's test for the RuntimeError case and to
redirect stdout to a file for the StopIteration case. Also, fixed-up
weird indentation in the C code.
|
|
msg94806 - (view) |
Author: Raymond Hettinger (rhettinger) |
Date: 2009-11-01 21:04 |
|
Forward-ported in r76025 r76026 and r76027.
|
|
| Date |
User |
Action |
Args |
| 2009-11-01 21:04:15 | rhettinger | set | status: open -> closed resolution: fixed messages:
+ msg94806
|
| 2009-11-01 18:48:15 | rhettinger | set | messages:
+ msg94799 |
| 2009-11-01 18:46:37 | rhettinger | set | messages:
- msg94794 |
| 2009-11-01 18:46:27 | rhettinger | set | messages:
- msg94793 |
| 2009-11-01 17:51:59 | rhettinger | set | messages:
+ msg94794 |
| 2009-11-01 17:41:02 | rhettinger | set | messages:
+ msg94793 |
| 2009-11-01 09:37:40 | ocean-city | set | messages:
+ msg94784 |
| 2009-11-01 09:32:53 | ocean-city | set | files:
+ improve_test_itertools.patch
messages:
+ msg94783 |
| 2009-10-31 17:13:57 | rhettinger | set | messages:
+ msg94763 |
| 2009-10-31 16:51:14 | georg.brandl | set | messages:
+ msg94761 |
| 2009-10-31 11:35:02 | durban | set | messages:
+ msg94754 |
| 2009-10-31 10:31:02 | georg.brandl | set | assignee: rhettinger
messages:
+ msg94751 nosy:
+ rhettinger, georg.brandl |
| 2009-10-31 10:12:55 | ocean-city | set | files:
+ fix_izip_longest.patch keywords:
+ patch messages:
+ msg94749
versions:
+ Python 2.6, Python 2.7, Python 3.2 |
| 2009-10-31 09:52:49 | ocean-city | set | nosy:
+ ocean-city messages:
+ msg94748
|
| 2009-10-31 09:07:57 | durban | create | |
|