This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Henry Carscadden
Recipients Dennis Sweeney, Henry Carscadden, rhettinger, tim.peters
Date 2020-04-10.04:10:15
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1586491815.75.0.176154052275.issue40230@roundup.psfhosted.org>
In-reply-to
Content
Hey, Tim, I just wanted a note of clarification. I was working on an approximation algorithm for a network science tool that is being released soon. Initially, I relied on itertools.product(), but when I moved to testing on non-trivial graphs, I immediately had Out of Memory Errors. This was even on the nodes with large memories on the computing cluster that I was using. The issue is a lot of extra lists are added as the number of iterables passed product grows although the actual number of elements in the iterables is relatively small. 
Here is the workaround that I used to handle many arguments:

def product(*args) -> Iterable:
    '''
    Takes in several iterables and returns a generator
    that yields the cartesian product of the elements in the iterables
    :param args: sets which you would like the cartesian product of
    :return: generator
    '''
    if len(args) == 1 and type(args) == list:
        return args
    numArgs = len(args)
    indices = np.zeros(numArgs, dtype=np.uint)
    checkLen = len(args[0])
    while indices[0] < checkLen:
        curr_term = [args[0][indices[0]]]
        for i in range(1, numArgs):
            curr_term.append(args[i][indices[i]])
        indices[-1] += np.uint(1)
        for i in range(1, numArgs):
            updated = False
            if indices[-i] >= len(args[-i]):
                indices[-i] = np.uint(0)
                indices[-i - 1] += np.uint(1)
                updated = True
            if not updated:
                break
        yield curr_term
History
Date User Action Args
2020-04-10 04:10:15Henry Carscaddensetrecipients: + Henry Carscadden, tim.peters, rhettinger, Dennis Sweeney
2020-04-10 04:10:15Henry Carscaddensetmessageid: <1586491815.75.0.176154052275.issue40230@roundup.psfhosted.org>
2020-04-10 04:10:15Henry Carscaddenlinkissue40230 messages
2020-04-10 04:10:15Henry Carscaddencreate