There is a useful Scanner class in the re module which is undocumented.

See:

http://mail.python.org/pipermail/python-dev/2003-April/035075.html
http://www.evanfosmark.com/2009/02/sexy-lexing-with-python/

Any opinions on whether this was intended to be exposed?

The class is commented as being "experimental", and the interface
probably could use improvement (it's a bit awkward to subclass Scanner),
but since it's been around for so long, I guess enough people will be
using it that we have to keep it that way and document it.

Opinions?

I'm sure people are already rely on the intimate details of this class,
so why not?

FYI, I did tidy up the class and add a 'scaniter' method when I was
working on issue #2636; it might yet see the light of day if it gets the
go ahead!

OK, so we'll wait for that.

Whether this should be exposed is up to effbot.  It's his code.  He
knows its limitations and he made the original decision to leave it
undocumented.

One of the limitations is that it identifies what matched by using
capture groups, so if the expressions provided contain captures then it
gets confused! :-)

I handled that by 1) rejecting named captures and 2) changing unnamed
captures into non-captures.

So far effbot hasn't said anything about this. So Frederik, was it
intended to be exposed or not?

Whether it was intended to be exposed or not it is known and used - and
therefore we can't change the API without going through the usual
deprecation process. As it is used and useful it should be documented.

OK, clear, then I'll see what I can do.

This is very old code, without a mature API.  Will take a look at
competing recipes to see if this is still the best way of doing this. 
Maybe it should return a generator instead of a list.  Perhaps there
should be some checking for re's that don't combine well.  Also, would
like to see it exercised on a number of common parsing tasks to see how
well it holds up.  Better to do this work than to expose what is there
now and get hit with bug reports, feature requests, deprecations, and
cases where is it almost the right tool for the job.

Looking back at the original thread, good reasons were expresses for not
documenting this code but just leaving it in as an example.