classification
Title: Doc for itertools recipe consume is complicated and less efficient
Type: performance Stage: commit review
Components: Documentation Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: 7778 Superseder:
Assigned To: rhettinger Nosy List: Muhammad.Alkarouri, ezio.melotti, flox, georg.brandl, peter.otten, rhettinger, zuo
Priority: normal Keywords:

Created on 2010-01-23 13:52 by Muhammad.Alkarouri, last changed 2010-04-03 03:24 by rhettinger. This issue is now closed.

Files
File name Uploaded Description Edit
consume_timeit.py Muhammad.Alkarouri, 2010-01-23 13:52 timings of the various implementation
Messages (11)
msg98187 - (view) Author: Muhammad Alkarouri (Muhammad.Alkarouri) Date: 2010-01-23 13:52
Based on the discussion at:
http://groups.google.co.uk/group/comp.lang.python/browse_thread/thread/c1ae3513a31eb63e/d0701a9902732c67?hl=en#d0701a9902732c67

In the documentation of itertools, one of the functions provided in the recipes section (10.7.3) is consume:

def consume(iterator, n):
    "Advance the iterator n-steps ahead. If n is none, consume entirely."
    collections.deque(islice(iterator, n), maxlen=0)

A clearer implementation is:

def consume(n, items):
    if n == 0:
        return
    next(islice(items, n-1, None), None)

It uses no fancy tricks and is thus more suitable for a documentation. Moreover, the second implementation is actually more efficient. Some timings are provided in the thread linked above. As an example, here are the timings on my machine (Python 2.6.1, x86_64, OS X 10.6:

consume_deque #old implementation
    10: 1.2913839817
   100: 3.18093585968
  1000: 21.6316840649

consume_forloop #using a straight for loop
    10: 0.658184051514
   100: 2.85271406174
  1000: 24.6730420589

consume_islice #the suggested implementation
    10: 0.688861131668
   100: 1.15058612823
  1000: 5.52356886864

The script computing these timings is attached. Thanks to Peter Otten for coming up both with the function and the timing script.
msg98189 - (view) Author: Peter Otten (peter.otten) * Date: 2010-01-23 17:29
As noted on comp.lang.python the implementation can be simplified to

def consume(items, n):
    next(islice(items, n, n), None)

When I suggested the above I wasn't aware that 

consume(items, None)

should exhaust the entire iterator. Unfortunately I've not found an elegant way to add that functionality.
msg98193 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-01-23 18:38
Will switch to the version using next() instead of deque().
msg98207 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-01-24 03:35
See r77721
msg98220 - (view) Author: Muhammad Alkarouri (Muhammad.Alkarouri) Date: 2010-01-24 12:57
Excellent solution and comments. Many thanks.
msg98237 - (view) Author: Jan Kaliszewski (zuo) Date: 2010-01-24 17:06
(sorry! typed into a wrong field)
msg98242 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-01-24 17:29
Changeset r77721 should be ported to trunk, and py3k probably.
msg98247 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-01-24 19:41
Remain calm.  I left the status as Open for a reason.  When I'm satisfied that people are reacting well to the new update, will port to other versions.
msg98260 - (view) Author: Muhammad Alkarouri (Muhammad.Alkarouri) Date: 2010-01-25 01:19
Mea culpa. Not knowing the conventions here, I closed the ticket, which probably resulted in florent's feedback. I will leave you to it then.
msg98277 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-01-25 09:31
No problem.
Will forward port when I get a chance.
msg98289 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-25 15:11
Raymond, before porting you should check the typos reported in issue #7778.
History
Date User Action Args
2010-04-03 03:24:18rhettingersetstatus: open -> closed
resolution: out of date
2010-01-25 15:11:16ezio.melottisetnosy: + ezio.melotti
messages: + msg98289

dependencies: + Typo(s) in ``itertools`` documentation reST marker
stage: needs patch -> commit review
2010-01-25 09:31:55rhettingersetmessages: + msg98277
versions: - Python 2.6
2010-01-25 01:19:32Muhammad.Alkarourisetmessages: + msg98260
2010-01-24 19:41:31rhettingersetmessages: + msg98247
2010-01-24 17:29:44floxsetstatus: closed -> open
nosy: + flox
messages: + msg98242

2010-01-24 17:06:21zuosetnosy: + zuo

messages: + msg98237
versions: + Python 2.6, Python 2.7, Python 3.2
2010-01-24 17:05:16zuosettitle: dictview -> Doc for itertools recipe consume is complicated and less efficient
2010-01-24 17:04:46zuosettitle: Doc for itertools recipe consume is complicated and less efficient -> dictview
versions: - Python 2.6, Python 2.7, Python 3.2
2010-01-24 12:57:40Muhammad.Alkarourisetstatus: open -> closed

messages: + msg98220
2010-01-24 03:35:24rhettingersetmessages: + msg98207
2010-01-23 18:38:59rhettingersetmessages: + msg98193
2010-01-23 17:29:10peter.ottensetnosy: + peter.otten
messages: + msg98189
2010-01-23 14:07:23georg.brandlsetassignee: georg.brandl -> rhettinger
2010-01-23 13:56:20ezio.melottisetpriority: normal
nosy: + rhettinger
versions: + Python 2.6, Python 3.1, Python 2.7, Python 3.2

stage: needs patch
2010-01-23 13:52:59Muhammad.Alkarouricreate