As stated in PEP-617, the old parser will be removed in Python 3.10

New changeset 9727694 by Lysandros Nikolaou in branch 'master':
bpo-40939: Generate keyword.py using the new parser (GH-20800)
9727694

>>> __new_parser__
  File "<stdin>", line 1
    __new_parser__
    ^
SyntaxError: You found it!

"new", "ex" or "ng" are not really future proof names. Can we rename the keyword to "__peg_parser__"?

Can we rename the keyword to "__peg_parser__"?

I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after #64967 is there.

I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after #64967 is there.

I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

New changeset 961edf7 by Miss Islington (bot) in branch '3.9':
bpo-40939: Generate keyword.py using the new parser (GH-20800)
961edf7

I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

Ok then! On it.

I did'nt ask to remove the easter egg. I'm just asking to avoid the "new" name. In my experience, each time that a "new" thing happens, later we have to use "new extended", "new_v2" or worse name :-)

Oh, if the name changes, please change it in 3.9 as well.

Look at this amazing names of the 5 flavors of functions parsing a string:

PyParser_ParseString()
PyParser_ParseStringFlags()
PyParser_ParseStringFlagsFilename()
PyParser_ParseStringFlagsFilenameEx() <= public!
PyParser_ParseStringObject()

Same for parsing a file:

PyParser_ParseFile()
PyParser_ParseFileFlags()
PyParser_ParseFileFlagsEx() <= public!
PyParser_ParseFileObject()

Or PyRun functions:

PyRun_String()
PyRun_AnyFile()
PyRun_AnyFileEx() <= public!
PyRun_AnyFileFlags()
PyRun_SimpleString()
PyRun_SimpleFile()
PyRun_SimpleFileEx() <= public!
PyRun_InteractiveOne()
PyRun_InteractiveLoop()
PyRun_File()
PyRun_FileEx() <= public!
PyRun_FileFlags()

ceval.c:

PyEval_EvalCode()
PyEval_EvalCodeEx() <= public!
_PyEval_EvalCodeWithName()
_PyEval_EvalCode()

I cannot count the number of "Ex" functions that we have :-)

Py_Finalize() -> :Py_FinalizeEx() <= public!
PyErr_Print() -> PyErr_PrintEx() <= public!
PySys_SetArgv() -> PySys_SetArgvEx() <= public!
PyErr_WarnEx() <= public!

_PyBytes_FormatEx()
_PyDict_MergeEx()
_Py_DecodeLocaleEx(), _Py_EncodeLocaleEx()
struct PyMemAllocatorEx
_Py_DecodeUTF8Ex(), _Py_EncodeUTF8Ex()
etc.

Honestly I see no reason to keep that easter egg. Can we remove it please?

New changeset 961edf7 by Miss Islington (bot) in branch '3.9':
bpo-40939: Generate keyword.py using the new parser (GH-20800)

This change broke this buildbot:

AMD64 Arch Linux VintageParser 3.9:
https://buildbot.python.org/all/#/builders/765/builds/67
1 test failed: test_keyword

Ok, let's remove it. Lysandros can you modify PR 20802 to remove it?

Lysandros can you modify PR 20802 to remove it?

Yup!

This change broke this buildbot:

#65001 fixes this breakage.

New changeset bcd7dee by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove PEG parser easter egg (new_parser) (bpo-20802)
bcd7dee

New changeset 1ed83ad by Pablo Galindo in branch 'master':
bpo-40939: Remove the old parser (GH-20768)
1ed83ad

A few remaining references to the good old times of the old parser:

Programs/_testembed.c:488: putenv("PYTHONOLDPARSER=1");
Programs/_testembed.c:676: putenv("PYTHONOLDPARSER=1");
Tools/scripts/run_tests.py:29: if 'PYTHONOLDPARSER' not in os.environ:

A few remaining references to the good old times of the old parser:

Thanks Victor, opened #20815 to address those.

New changeset 436b648 by Pablo Galindo in branch 'master':
bpo-40939: Remove some extra references to PYTHONOLDPARSER (GH-20815)
436b648

New changeset 3782497 by Pablo Galindo in branch '3.9':
[3.9] bpo-40939: Fix test_keyword for the old parser (GH-20814)
3782497

New changeset 756180b by Pablo Galindo in branch 'master':
bpo-40939: Clean and adapt the peg_generator directory after deleting the old parser (GH-20822)
756180b

Shouldn't the following files be deleted too?

Include/bitset.h
Include/grammar.h
Include/graminit.h
Include/parsetok.h
Include/node.h
Python/graminit.c
Parser/node.c

Also declarations:
PyNode_Compile in Include/compile.h
PyParser_SimpleParse* in Include/pythonrun.h

And PyParser_ASTFrom* API need new implementations.

I'm currently testing a commit that removes all these files on my fork, before I push it upstream. A question that I'm not 100% sure about is if we can already remove the symbol module. I guess it's okay since it got deprecated in 3.9 (bpo-40759) and the old parser is also out, but just to make sure.

You can delete symbol.py -- it has no use now that the old parser is gone.

We should probably also update the regeneration targets in the Makefile. (At least review them.)

New changeset 314858e by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove the old parser (Part 2) (GH-21005)
314858e

I will submit a PR today

Pablo, have you already started on this? I didn't see your comment earlier and I've got a PR ready.

Pablo, have you already started on this? I didn't see your comment earlier and I've got a PR ready.

Yeah, but don't worry: submit your PR and I will review it :)

New changeset c26d591 by Lysandros Nikolaou in branch 'master':
bpo-40939: Document removal of the old parser in 3.10 whatsnew (GH-23321)
c26d591

Victor, Miro, both removal of the parser module and of all the C API functions are now documented in the 3.10 whatsnew document. Do you feel that this is enough for us to close this issue again?

Thanks. I feel like the What's new document should teach people what to do when they are hit by the removals. The removals are documented, but the developers who are affected have no clue what to do. What do you think?

(Sorry I wasn't able to provide this feedback before the PR was merged.)

The removals are documented, but the developers who are affected have no clue what to do. What do you think?

Here is difficult to recommend a canonical Path because as I mentioned, there is no replacement for these functions because that functions returned CST nodes and those not exist anymore.

We should check of the 3 mentioned projects (mod_wsgi,
kdevelop-python, unbound) use the removed functions to suggest a similar replacement. I understood that there is no drop-in replacement.

unbound does not use to get the parsed Python code as CST, but uses PyParser_SimpleParseFile() just to display an error message to stderr. I understand that PyParser_ASTFromFileObject() + PyErr_Print() could be used.

But PyParser_ASTFromFileObject() is low-level, it requires to pass an arena object. Maybe the *intent* here is to call compile() and display the error message? Pseudo-code:

---

fp = fopen(filename, "r");
bytes = readall(fp);
PyObject *builtins = PyEval_GetBuiltins();
obj = PyObject_CallMethod(builtins, "compile", "O", bytes);
Py_DECREF(bytes);
if (!obj) {
    PyErr_Print();
}
else {
    Py_DECREF(obj);
}
fclose(fp);

This code is non-trivial :-( Should we provide a *new* C function doing that?

Input: filename
Output: code object

Or maybe I just missed an existing function :-)

This code is non-trivial :-( Should we provide a *new* C function doing that?

We could discuss adding a new C function, but IMHO that code is not especially horrible or unreadable. I agree it could be simpler, though.

Hi All,

It seems this patch removes some functions provided by the Stable ABI (PEP-384), most notably Py_CompileString. Was this the intention? If not, is there still a chance to reintroduce it before the release?

It seems this patch removes some functions provided by the Stable ABI (PEP-384), most notably Py_CompileString. Was this the intention? If not, is there still a chance to reintroduce it before the release?

The functions removal is intentional and was approved by the PEP-617.

But Py_CompileString() function was not removed, it's still in the master branch (future Python 3.10). Why do you think that it has been removed?

But Py_CompileString() function was not removed, it's still in the master branch (future Python 3.10). Why do you think that it has been removed?

Thank you. It looked that way because of the removed block of lines in the commit 1ed83ad (pythonrun.c). We were also getting a missing symbol error. We'll check again to be sure.

Attached is a sample program which works on 3.9 but fails linking with 3.10.0a2

The .so is missing the symbol:

igor@LAPTOP:~/py_limited_api_example$ nm /home/igor/lib/libpython3.9.so | grep Py_CompileString
0000000000212720 T Py_CompileString
000000000020fe30 T Py_CompileStringExFlags
0000000000212730 T Py_CompileStringFlags
000000000020fd40 T Py_CompileStringObject

igor@LAPTOP:~/py_limited_api_example$ nm /home/igor/lib/libpython3.10.so | grep Py_CompileString
0000000000201a40 T Py_CompileStringExFlags
0000000000201980 T Py_CompileStringObject

Please stop breaking the Stable ABI :/

Hm, I wonder if there's a typo here in pythonrun.c:

/* For use in Py_LIMITED_API */
#undef Py_CompileString
PyObject *
PyCompileString(const char *str, const char *filename, int start)
{
    return Py_CompileStringFlags(str, filename, start, NULL);
}

Shouldn't that function be named Py_CompileString (i.e. Py_ instead of Py)?

This seems to be old code, but there's normally a macro Py_CompileString() that translates to Py_CompileStringFlags() in pythonrun.h:

#ifdef Py_LIMITED_API
PyAPI_FUNC(PyObject *) Py_CompileString(const char *, const char *, int);
#else
#define Py_CompileString(str, p, s) Py_CompileStringExFlags(str, p, s, NULL, -1)
#define Py_CompileStringFlags(str, p, s, f) Py_CompileStringExFlags(str, p, s, f, -1)
.
.
.

Seems that commit 1ed83ad was deleting some functions that were not correctly covered by redirection macros. I have opened PR 23606 to restore those as they were.

New changeset 46bd5ed by Pablo Galindo in branch 'master':
bpo-40939: Restore some stable API functions incorrectly deleted (GH-23606)
46bd5ed

Thanks for the fix! I'd like to submit a test to avoid this and similar issues in future. Are there any guidelines for this? Sorry if this is a wrong place to ask.

I am currently working on test that checks that the stable API symbols are correctly exported. Unfortunately there is no official maintained list if those symbols, so is taking a while

I am currently working on test that checks that the stable API symbols are correctly exported.

Thank you very much! For added motivation, the 3.8.0 release was unusable thanks to bpo-37633 which was somewhat similar (also in pythonrun). For reference, I've attached a list of symbols we currently use (a few more we have to import dynamically since they're not in the stable ABI but we'd like to keep that list as short as possible).

Unfortunately there is no official maintained list if those symbols, so is taking a while

On Windows there is PC/python3dll.c.

On Windows there is PC/python3dll.c.

Even that is severely out of date, unfortunately. There are many functions that were removed in 3.8 and 3.7 that are still listed there.

Opened https://bugs.python.org/issue42545 for that

FYI the unbound project was fixed by calling Py_CompileString() on Python 3.9 and newer:
NLnetLabs/unbound@e0d426e

• The removed functions are still listed in the documentation; this is confusing.

• struct _node is not very usable in the C API, but what I saw in mod_wsgi was giving gave PyParser_* output to PyNode_Compile. I tried to write porting notes for this usage. It should also help with the "compile to check for syntax errors" case in unbound.

New changeset 29987f7 by Petr Viktorin in branch 'main':
bpo-40939: Remove documentation for PyParser_* & add porting notes (GH-26855)
29987f7

New changeset dc10264 by Miss Islington (bot) in branch '3.10':
bpo-40939: Remove documentation for PyParser_* & add porting notes (GH-26855) (GH-26898)
dc10264