This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author nneonneo
Recipients Lucas Wiman, Sumudu.Fernando, eric.araujo, falsetru, nneonneo, rhettinger, terry.reedy, yegle
Date 2016-09-17.20:12:01
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1474143121.85.0.451534395535.issue10109@psf.upfronthosting.co.za>
In-reply-to
Content
I think this _is_ a bug. Most of the itertools are quite thrifty with memory, and exceptions are explicitly documented.

The itertools generally only require memory proportional to the number of iterations consumed. However, `itertools.product` requires an enormous up-front memory cost even if only a few iterations are performed. There are good reasons to extract only a few iterations from a `product` - for example, a search procedure where only the first item in the product is infinite:

    for year, day in product(count(2010), range(365)):
        if rare_event(year, day):
            break

Or, in an application I recently tried, to count occurrences of something up to a limit:

    prod = product(count(), range(n))
    filtered = ifilter(pred, prod)
    want = list(islice(filtered, 101))
    if len(want) > 100:
        print "Too many matches"

Having `product` greedily consume all of its input iterators is rather antithetical to the notion of itertools being lazy generators, and it breaks efficient composability with the other itertools.

The fix should be fairly straightforward: like tee, maintain lists of items already generated as the input iterators are exhausted, and add a note that "product"'s memory usage may slowly grow over time (as opposed to growing to maximum size immediately).
History
Date User Action Args
2016-09-17 20:12:01nneonneosetrecipients: + nneonneo, rhettinger, terry.reedy, falsetru, eric.araujo, Sumudu.Fernando, yegle, Lucas Wiman
2016-09-17 20:12:01nneonneosetmessageid: <1474143121.85.0.451534395535.issue10109@psf.upfronthosting.co.za>
2016-09-17 20:12:01nneonneolinkissue10109 messages
2016-09-17 20:12:01nneonneocreate