CPython provides a Python-level API to the parser, but not to the
tokenizer itself.  Somewhat annoyingly, it does provide a nice C API,
but that's not properly exposed for external modules.  

To fix this, the tokenizer.h file should be moved from the Parser
directory to the Include directory, and the (semi-public) functions that
already available must be flagged with PyAPI_FUNC, as shown below.

The PyAPI_FUNC fix should be non-intrusive enough to go into 2.6 and
3.0; moving stuff around is perhaps better left for a later release
(which could also include a Python binding).

Index: tokenizer.h
===================================================================
--- tokenizer.h (revision 514)
+++ tokenizer.h (working copy)
@@ -54,10 +54,10 @@
        const char* str;
 };

-extern struct tok_state *PyTokenizer_FromString(const char *);
-extern struct tok_state *PyTokenizer_FromFile(FILE *, char *, char *);
-extern void PyTokenizer_Free(struct tok_state *);
-extern int PyTokenizer_Get(struct tok_state *, char **, char **);
+PyAPI_FUNC(struct tok_state *) PyTokenizer_FromString(const char *);
+PyAPI_FUNC(struct tok_state *) PyTokenizer_FromFile(FILE *, char *,
char *);
+PyAPI_FUNC(void) PyTokenizer_Free(struct tok_state *);
+PyAPI_FUNC(int) PyTokenizer_Get(struct tok_state *, char **, char **);

 #ifdef __cplusplus
 }

IMO the "struct tok_state" should not be part of the API, it contains
too many implementation details. Or maybe as an opaque structure.

There are a few things in the struct that needs to be public, but that's
nothing that cannot be handled by documentation.  No need to complicate
the API just in case.

Sorry for the terribly dumb question about this.

Are you meaning that, at this stage, all that is required is:

 1. the application of the PyAPI_FUNC macro
 2. move the file to the Include directory
 3. update Makefile.pre.in to point to the new location

Just I have read this now 10 times or so and keep thinking more must be
involved :-) [certainly given my embarrassing start to the Python dev
community re:asynchronous thread exceptions :-| ]

I have attached a patch that does this. Though at this time it is
lacking any documentation that will state what parts of "struct
tok_state" are private and public. I will need to trawl the code some
more to do that.

I have executed:

 - ./configure
 - make
 - make test

And all proceed well.

That's should be all that's needed to expose the existing API, as is. 
If you want to verify the build, you can grab the pytoken.c and setup.py
files from this directory, and try building the module.

http://svn.effbot.org/public/stuff/sandbox/pytoken/

Make sure you remove the local copy of "tokenizer.h" that's present in
that directory before you build.  If that module builds, all's well.

Did that and it builds fine.

So my test procedure was:

 - checkout clean source
 - apply patch as per guidelines
 - remove the file Psrser/tokenizer.h (*)
 - ./configure
 - make
 - ./python setup.py install

Build platform: Ubuntu , gcc 4.2.3

All works fine.

thanks for the extra test files.

* - one question though. I removed the file using 'svn remove' but the
diff makes it an empty file not removed why is that? (and is it correct?)

It would be nice if this same C API was used to implement the 'tokenize' module.  Issues like issue2180 will potentially require bug fixes in two places :-/