This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Sumudu.Fernando
Recipients Sumudu.Fernando, eric.araujo, falsetru, rhettinger, terry.reedy
Date 2012-01-18.11:01:32
SpamBayes Score 0.0
Marked as misclassified No
Message-id <1326884494.62.0.671988482658.issue10109@psf.upfronthosting.co.za>
In-reply-to
Content
I don't agree with the response to this.

It is true that as implemented (at least in 2.7, I don't have 3.x handy to check) itertools.product requires finite iterables.  However this seems to be simply a consequence of the implementation and not part of the "spirit" of the function, which as falsetru pointed out is stated to be "equivalent to nested for-loops in a generator expression".

Indeed, implementing product in Python (in a recursive way) doesn't have this problem.

Perhaps a more convincing set of testcases to show why this could be considered a problem:

>>> import itertools
>>> itertools.product(xrange(100))
<itertools.product object at 0xb7ed334c>
>>> itertools.product(xrange(1000000))
<itertools.product object at 0xb7ed620c>
>>> itertools.product(xrange(1000000000))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

Note that I'm not even using an infinite iterable, just a really big one.  The issue is that creating the iterator fails with a MemoryError, before I've even asked for any values.  Consider the following:

for (i, v) in enumerate(itertools.product(a, b, c)):
    if i < 1000:
        print v
    else:
        break

When a, b, and c are relatively small, finite iterables, this code works fine.  However, if *any* of them are too large (or infinite), we see a MemoryError before the loop even starts, even though only 1000 elements are required.  I think it's conceivable that we might want something like "a = itertools.cycle(xrange(5))", and even that will break this loop.

That said, in all such cases I could think of, we can always either truncate big iterators before passing them to product, or use zip/comprehensions to add their values into the tuple (or some combination of those).  So maybe it isn't a huge deal.

I've attached my implementation of product which deals with infinite iterators by leveraging enumerate and itertools.cycle, and is pretty much a direct translation of the "odometer" idea.  This doesn't support the "repeat" parameter (but probably could using itertools.tee).  One thing that should be changed is itertools.cycle shouldn't be called / doesn't need to be called on infinite iterators, but I couldn't figure out how to do that.  Maybe there is some way to handle it in the C implementation?)

In summary: the attached implementation of product can accept any mix of infinite / finite iterators, returning a generator intended for partial consumption.  The existing itertools.product doesn't work in this case.
History
Date User Action Args
2012-01-18 11:01:34Sumudu.Fernandosetrecipients: + Sumudu.Fernando, rhettinger, terry.reedy, falsetru, eric.araujo
2012-01-18 11:01:34Sumudu.Fernandosetmessageid: <1326884494.62.0.671988482658.issue10109@psf.upfronthosting.co.za>
2012-01-18 11:01:33Sumudu.Fernandolinkissue10109 messages
2012-01-18 11:01:32Sumudu.Fernandocreate