This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author effbot
Recipients effbot
Date 2008-07-14.11:32:09
SpamBayes Score 1.4654182e-05
Marked as misclassified No
Message-id <1216035196.11.0.15913194841.issue3353@psf.upfronthosting.co.za>
In-reply-to
Content
CPython provides a Python-level API to the parser, but not to the
tokenizer itself.  Somewhat annoyingly, it does provide a nice C API,
but that's not properly exposed for external modules.  

To fix this, the tokenizer.h file should be moved from the Parser
directory to the Include directory, and the (semi-public) functions that
already available must be flagged with PyAPI_FUNC, as shown below.

The PyAPI_FUNC fix should be non-intrusive enough to go into 2.6 and
3.0; moving stuff around is perhaps better left for a later release
(which could also include a Python binding).

Index: tokenizer.h
===================================================================
--- tokenizer.h (revision 514)
+++ tokenizer.h (working copy)
@@ -54,10 +54,10 @@
        const char* str;
 };

-extern struct tok_state *PyTokenizer_FromString(const char *);
-extern struct tok_state *PyTokenizer_FromFile(FILE *, char *, char *);
-extern void PyTokenizer_Free(struct tok_state *);
-extern int PyTokenizer_Get(struct tok_state *, char **, char **);
+PyAPI_FUNC(struct tok_state *) PyTokenizer_FromString(const char *);
+PyAPI_FUNC(struct tok_state *) PyTokenizer_FromFile(FILE *, char *,
char *);
+PyAPI_FUNC(void) PyTokenizer_Free(struct tok_state *);
+PyAPI_FUNC(int) PyTokenizer_Get(struct tok_state *, char **, char **);

 #ifdef __cplusplus
 }
History
Date User Action Args
2008-07-14 11:33:16effbotsetspambayes_score: 1.46542e-05 -> 1.4654182e-05
recipients: + effbot
2008-07-14 11:33:16effbotsetspambayes_score: 1.46542e-05 -> 1.46542e-05
messageid: <1216035196.11.0.15913194841.issue3353@psf.upfronthosting.co.za>
2008-07-14 11:32:15effbotlinkissue3353 messages
2008-07-14 11:32:13effbotcreate