This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Remove the old parser
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pablogsal Nosy List: Igor.Skochinsky, brandtbucher, gvanrossum, hroncok, lys.nikolaou, miss-islington, ned.deily, nnemkin, pablogsal, petr.viktorin, terry.reedy, vstinner
Priority: normal Keywords: patch

Created on 2020-06-10 11:25 by pablogsal, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
py_limited_api_example.zip Igor.Skochinsky, 2020-12-02 00:08 Example of using Py_CompileString with Py_LIMITED_API
py3.txt Igor.Skochinsky, 2020-12-02 16:47 Imports from python3.dll
Pull Requests
URL Status Linked Edit
PR 20768 merged pablogsal, 2020-06-10 11:25
PR 20800 merged lys.nikolaou, 2020-06-11 11:02
PR 20801 merged miss-islington, 2020-06-11 12:45
PR 20802 merged lys.nikolaou, 2020-06-11 13:13
PR 20814 merged pablogsal, 2020-06-11 16:43
PR 20815 merged pablogsal, 2020-06-11 16:46
PR 20822 merged pablogsal, 2020-06-12 00:25
PR 21005 merged lys.nikolaou, 2020-06-20 16:53
PR 21012 merged lys.nikolaou, 2020-06-20 18:34
PR 21016 merged lys.nikolaou, 2020-06-20 19:26
PR 21025 merged pablogsal, 2020-06-21 03:06
PR 21036 merged lys.nikolaou, 2020-06-21 23:46
PR 21152 merged ned.deily, 2020-06-25 14:23
PR 21624 merged gvanrossum, 2020-07-26 03:22
PR 19969 merged pablogsal, 2020-07-26 21:47
PR 21641 merged gvanrossum, 2020-07-27 18:31
PR 21642 merged lys.nikolaou, 2020-07-27 18:57
PR 22294 merged terry.reedy, 2020-09-18 01:07
PR 23321 merged lys.nikolaou, 2020-11-16 18:28
PR 23606 merged pablogsal, 2020-12-02 02:58
PR 26855 merged petr.viktorin, 2021-06-22 15:40
PR 26898 merged miss-islington, 2021-06-24 12:57
Messages (69)
msg371182 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-10 11:25
As stated in PEP 617, the old parser will be removed in Python 3.10
msg371264 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-11 12:45
New changeset 9727694f08cad4b019d2939224e3416312b1c0e1 by Lysandros Nikolaou in branch 'master':
bpo-40939: Generate keyword.py using the new parser (GH-20800)
https://github.com/python/cpython/commit/9727694f08cad4b019d2939224e3416312b1c0e1
msg371265 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 12:50
>>> __new_parser__
  File "<stdin>", line 1
    __new_parser__
    ^
SyntaxError: You found it!

"new", "ex" or "ng" are not really future proof names. Can we rename the keyword to "__peg_parser__"?
msg371266 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-11 12:59
> Can we rename the keyword to "__peg_parser__"?

I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after GH-20768 is there.
msg371267 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-11 13:01
> I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after GH-20768 is there.

I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".
msg371268 - (view) Author: miss-islington (miss-islington) Date: 2020-06-11 13:02
New changeset 961edf7979ca34d6fe104a1cce005aa8cac35821 by Miss Islington (bot) in branch '3.9':
bpo-40939: Generate keyword.py using the new parser (GH-20800)
https://github.com/python/cpython/commit/961edf7979ca34d6fe104a1cce005aa8cac35821
msg371270 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-11 13:08
> I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

Ok then! On it.
msg371271 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 13:17
I did'nt ask to remove the easter egg. I'm just asking to avoid the "new" name. In my experience, each time that a "new" thing happens, later we have to use "new extended", "new_v2" or worse name :-)

Oh, if the name changes, please change it in 3.9 as well.


Look at this amazing names of the 5 flavors of functions parsing a string:

PyParser_ParseString()
PyParser_ParseStringFlags()
PyParser_ParseStringFlagsFilename()
PyParser_ParseStringFlagsFilenameEx() <= public!
PyParser_ParseStringObject()

Same for parsing a file:

PyParser_ParseFile()
PyParser_ParseFileFlags()
PyParser_ParseFileFlagsEx() <= public!
PyParser_ParseFileObject()

Or PyRun functions:

PyRun_String()
PyRun_AnyFile()
PyRun_AnyFileEx() <= public!
PyRun_AnyFileFlags()
PyRun_SimpleString()
PyRun_SimpleFile()
PyRun_SimpleFileEx() <= public!
PyRun_InteractiveOne()
PyRun_InteractiveLoop()
PyRun_File()
PyRun_FileEx() <= public!
PyRun_FileFlags()

ceval.c:

PyEval_EvalCode()
PyEval_EvalCodeEx() <= public!
_PyEval_EvalCodeWithName()
_PyEval_EvalCode()

I cannot count the number of "Ex" functions that we have :-)

Py_Finalize() -> :Py_FinalizeEx() <= public!
PyErr_Print() -> PyErr_PrintEx() <= public!
PySys_SetArgv() -> PySys_SetArgvEx() <= public!
PyErr_WarnEx() <= public!

_PyBytes_FormatEx()
_PyDict_MergeEx()
_Py_DecodeLocaleEx(), _Py_EncodeLocaleEx()
struct PyMemAllocatorEx
_Py_DecodeUTF8Ex(), _Py_EncodeUTF8Ex()
etc.
msg371288 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-06-11 14:27
Honestly I see no reason to keep that easter egg. Can we remove it please?
msg371289 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 14:29
> New changeset 961edf7979ca34d6fe104a1cce005aa8cac35821 by Miss Islington (bot) in branch '3.9':
> bpo-40939: Generate keyword.py using the new parser (GH-20800)

This change broke this buildbot:

AMD64 Arch Linux VintageParser 3.9:
https://buildbot.python.org/all/#/builders/765/builds/67
1 test failed: test_keyword
msg371290 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-11 14:32
Ok, let's remove it. Lysandros can you modify PR 20802 to remove it?
msg371291 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-11 14:48
> Lysandros can you modify PR 20802 to remove it?

Yup!
msg371292 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-11 14:55
> This change broke this buildbot:

GH-20802 fixes this breakage.
msg371301 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-06-11 16:09
New changeset bcd7deed9118e365c1225de2a2e1a81bf988c6ab by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove PEG parser easter egg (__new_parser__) (#20802)
https://github.com/python/cpython/commit/bcd7deed9118e365c1225de2a2e1a81bf988c6ab
msg371307 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-11 16:30
New changeset 1ed83adb0e95305af858bd41af531e487f54fee7 by Pablo Galindo in branch 'master':
bpo-40939: Remove the old parser (GH-20768)
https://github.com/python/cpython/commit/1ed83adb0e95305af858bd41af531e487f54fee7
msg371308 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 16:36
A few remaining references to the good old times of the old parser:

Programs/_testembed.c:488:    putenv("PYTHONOLDPARSER=1");
Programs/_testembed.c:676:    putenv("PYTHONOLDPARSER=1");
Tools/scripts/run_tests.py:29:    if 'PYTHONOLDPARSER' not in os.environ:
msg371310 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-11 16:46
> A few remaining references to the good old times of the old parser:

Thanks Victor, opened https://github.com/python/cpython/pull/20815 to address those.
msg371311 - (view) Author: miss-islington (miss-islington) Date: 2020-06-11 17:08
New changeset 436b648910c27baf8164a6d46d746d36d8a93478 by Pablo Galindo in branch 'master':
bpo-40939: Remove some extra references to PYTHONOLDPARSER (GH-20815)
https://github.com/python/cpython/commit/436b648910c27baf8164a6d46d746d36d8a93478
msg371316 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-11 18:29
New changeset 3782497cc22e70b41e32ac09cb06d3948074d8a7 by Pablo Galindo in branch '3.9':
[3.9] bpo-40939: Fix test_keyword for the old parser (GH-20814)
https://github.com/python/cpython/commit/3782497cc22e70b41e32ac09cb06d3948074d8a7
msg371327 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-12 00:55
New changeset 756180b4bfa09bb77394a2b3754d331181d4f28c by Pablo Galindo in branch 'master':
bpo-40939: Clean and adapt the peg_generator directory after deleting the old parser (GH-20822)
https://github.com/python/cpython/commit/756180b4bfa09bb77394a2b3754d331181d4f28c
msg371929 - (view) Author: Nikita Nemkin (nnemkin) * Date: 2020-06-20 10:01
Shouldn't the following files be deleted too?

Include/bitset.h
Include/grammar.h
Include/graminit.h
Include/parsetok.h
Include/node.h
Python/graminit.c
Parser/node.c

Also declarations:
PyNode_Compile in Include/compile.h
PyParser_SimpleParse* in Include/pythonrun.h

And PyParser_ASTFrom* API need new implementations.
msg371940 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-20 15:45
I'm currently testing a commit that removes all these files on my fork, before I push it upstream. A question that I'm not 100% sure about is if we can already remove the symbol module. I guess it's okay since it got deprecated in 3.9 (bpo-40759) and the old parser is also out, but just to make sure.
msg371942 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-06-20 16:31
You can delete symbol.py -- it has no use now that the old parser is gone.

We should probably also update the regeneration targets in the Makefile. (At least review them.)
msg371953 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-20 18:07
New changeset 314858e2763e76e77029ea0b691d749c32939087 by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove the old parser (Part 2) (GH-21005)
https://github.com/python/cpython/commit/314858e2763e76e77029ea0b691d749c32939087
msg371977 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-21 01:16
New changeset d301d9473e9a9b78d6e6678e9fe5ef66d46084e1 by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40939: Deprecate the PyParser_SimpleParse* functions (GH-21012)
https://github.com/python/cpython/commit/d301d9473e9a9b78d6e6678e9fe5ef66d46084e1
msg372023 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-21 21:34
> bpo-40939: Remove the old parser (Part 2) (GH-21005)

This change removes PyNode_Compile() which is part of the public C API. Would you mind to also deprecate it, as you did for PyParser_xxx() functions?
msg372025 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-21 23:47
New changeset 564cd187677ae8d1488c4d8ae649aea34ebbde07 by Lysandros Nikolaou in branch 'master':
bpo-40939: Rename PyPegen* functions to PyParser* (GH-21016)
https://github.com/python/cpython/commit/564cd187677ae8d1488c4d8ae649aea34ebbde07
msg372026 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-21 23:49
Thanks Victor for the catch! I've opened a PR that deprecates PyNode_Compile in 3.9.
msg372027 - (view) Author: miss-islington (miss-islington) Date: 2020-06-22 00:07
New changeset 8ae5e8ec8147e6434454e66953c25848b848711a by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40939: Deprecate PyNode_Compile (GH-21036)
https://github.com/python/cpython/commit/8ae5e8ec8147e6434454e66953c25848b848711a
msg372370 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-25 14:46
New changeset 8d02f91dc6139a13b6efa9bd5a5b4bdd7ddcc29d by Ned Deily in branch 'master':
bpo-40939: run autoreconf to fix configure{,.ac} disparity (GH-21152)
https://github.com/python/cpython/commit/8d02f91dc6139a13b6efa9bd5a5b4bdd7ddcc29d
msg374282 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-07-25 21:25
There are some difficulties with removing Grammar/Grammar, since it is used to generate the full grammar in the reference docs (Doc/reference/grammar.rst). Producing a similar grammar from the PEG grammar is currently painful because our PEG grammar contains a number of "invalid_*" rules that just exist to be able to produce more useful error messages.

See https://github.com/we-like-parsers/cpython/issues/135.

The continued existence of Grammar/Grammar has confused at least one person, see issue41362.

Grepping for Grammar/Grammar in the docs, there are a few other occurrences:

library/token.rst:14:of the parse tree (terminal tokens).  Refer to the file :file:`Grammar/Grammar`
library/parser.rst:43::file:`Grammar/Grammar` in the standard Python distribution.  The parse trees
library/symbol.rst:15:names.  Refer to the file :file:`Grammar/Grammar` in the Python distribution for
reference/grammar.rst:7:.. literalinclude:: ../../Grammar/Grammar
msg374346 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-07-26 21:46
See also https://github.com/python/cpython/pull/19969 (Pablo's attempt at replacing the grammar in the reference docs with something derived from Grammar/python.gram).
msg374410 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-07-27 18:20
New changeset 72cabb2aa636272e608285f5a6ba83b62be9be4e by Pablo Galindo in branch 'master':
bpo-40939: Use the new grammar for the grammar specification documentation (GH-19969)
https://github.com/python/cpython/commit/72cabb2aa636272e608285f5a6ba83b62be9be4e
msg374413 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-07-27 18:38
The old parser is completely gone from the 3.10 branch. Closing.
msg374416 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-07-27 19:00
New changeset e6b2d93f0c3891827f609ecac1ced21e1626ed0a by Guido van Rossum in branch '3.9':
[3.9] bpo-40939: Use the new grammar for the grammar specification documentation (GH-19969) (#21641)
https://github.com/python/cpython/commit/e6b2d93f0c3891827f609ecac1ced21e1626ed0a
msg374417 - (view) Author: miss-islington (miss-islington) Date: 2020-07-27 19:53
New changeset b3fbff7289176ba1a322e6899c3d4a04880ed5a7 by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove even more references to the old parser (GH-21642)
https://github.com/python/cpython/commit/b3fbff7289176ba1a322e6899c3d4a04880ed5a7
msg377083 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-09-18 01:57
New changeset 5a8364780b7e881385f6fabcf072d599e80f51b8 by Terry Jan Reedy in branch 'master':
bpo-41808: Add What's New 3.9 entry missing from master (#22294)
https://github.com/python/cpython/commit/5a8364780b7e881385f6fabcf072d599e80f51b8
msg380826 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-12 15:13
I reopen the issue.

Would you mind to explicitly list function removed from the C API in What's New in Python 3.10?
https://docs.python.org/dev/whatsnew/3.10.html#id4

I'm talking about the commit 1ed83adb0e95305af858bd41af531e487f54fee7.

For example, the unbound project no longer builds with Python 3.10 because PyParser_SimpleParseFile() has been removed:
https://bugzilla.redhat.com/show_bug.cgi?id=1889726

There is no mention of PyParser_SimpleParseFile() removal in What's New in Python 3.10. There is only a mention that it's being deprecated in What's New in Python 3.9.
msg380828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-12 15:18
> There is no mention of PyParser_SimpleParseFile() removal in What's New in Python 3.10.

By the way, is there a replacement for this function? The unbound project uses it to display a SyntaxError when PyRun_SimpleFile() fails.

Petr Menšík asked:

"Could we instead modify PyRun_SimpleFile call to produce just one exception, then print it to stderr once and once into the log? (...) But it seems PyRun_SimpleFile does not throw Exception. Can you recommend variant or flags, which would make it raise an Exception, which log_py_err() would then to log file? After commenting out PyParser_SimpleParseFile it reports None, so it did not already raise an exception."

https://bugzilla.redhat.com/show_bug.cgi?id=1889726#c3

unbound used the removed function PyParser_SimpleParseFile() in pythonmod/pythonmod.c. Extract of unbound-1.12.0.tar.gz:

   if (PyRun_SimpleFile(script_py, pe->fname) < 0) {
      log_err("pythonmod: can't parse Python script %s", pe->fname);
      /* print the error to logs too, run it again */
      fseek(script_py, 0, SEEK_SET);
      /* we don't run the file, like this, because then side-effects
       *    s = PyRun_File(script_py, pe->fname, Py_file_input, 
       *        PyModule_GetDict(PyImport_AddModule("__main__")), pe->dict);
       * could happen (again). Instead we parse the file again to get
       * the error string in the logs, for when the daemon has stderr
       * removed.  SimpleFile run already printed to stderr, for then
       * this is called from unbound-checkconf or unbound -dd the user
       * has a nice formatted error.
      */
      /* ignore the NULL return of _node, it is NULL due to the parse failure
       * that we are expecting */
      (void)PyParser_SimpleParseFile(script_py, pe->fname, Py_file_input);
      log_py_err();
      PyGILState_Release(gil);
      fclose(script_py);
      return 0;
   }
msg380855 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-11-12 22:10
Honestly that code seems poorly thought out. If running it returns -1, an exception was presumably reported, but not necessarily SyntaxError -- so parsing it may not produce an error at all.

The functionality needed is in PyRun_InteractiveOneObjectEx(), but that is not public. :-(
msg380863 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-12 23:35
> By the way, is there a replacement for this function? The unbound project uses it to display a SyntaxError when PyRun_SimpleFile() fails.

There is no replacement for the function because that function returned CST nodes and those not exist anymore.
msg381095 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-11-16 12:51
Could the removal of the parser module be documented in https://docs.python.org/3.10/whatsnew/3.10.html please?
msg381103 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-16 15:03
Please also document the removal of the node.h header file. The removal of this file broke the build of the two following packages.


mod_wsgi: https://bugzilla.redhat.com/show_bug.cgi?id=1898158

In file included from src/server/mod_wsgi.c:22:
src/server/wsgi_python.h:44:10: fatal error: node.h: No such file or directory
   44 | #include "node.h"
      |          ^~~~~~~~
compilation terminated.

kdevelop-python: https://bugzilla.redhat.com/show_bug.cgi?id=1898116


In file included from /builddir/build/BUILD/kdev-python-5.6.0/parser/astbuilder.cpp:31:
/builddir/build/BUILD/kdev-python-5.6.0/parser/python_header.h:33:10: fatal error: node.h: No such file or directory
   33 | #include "node.h"
      |          ^~~~~~~~
compilation terminated.
msg381112 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-16 15:23
I will submit a PR today
msg381136 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-11-16 18:13
Pablo, have you already started on this? I didn't see your comment earlier and I've got a PR ready.
msg381137 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-16 18:21
> Pablo, have you already started on this? I didn't see your comment earlier and I've got a PR ready.

Yeah, but don't worry: submit your PR and I will review it :)
msg381140 - (view) Author: miss-islington (miss-islington) Date: 2020-11-16 18:46
New changeset c26d5916d68c47a20dd941f9e89afdaf85b2711e by Lysandros Nikolaou in branch 'master':
bpo-40939: Document removal of the old parser in 3.10 whatsnew (GH-23321)
https://github.com/python/cpython/commit/c26d5916d68c47a20dd941f9e89afdaf85b2711e
msg381172 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-11-16 22:40
Victor, Miro, both removal of the parser module and of all the C API functions are now documented in the 3.10 whatsnew document. Do you feel that this is enough for us to close this issue again?
msg381176 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-11-16 22:57
Thanks. I feel like the What's new document should teach people what to do when they are hit by the removals. The removals are documented, but the developers who are affected have no clue what to do. What do you think?

(Sorry I wasn't able to provide this feedback before the PR was merged.)
msg381182 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-16 23:26
> The removals are documented, but the developers who are affected have no clue what to do. What do you think?

Here is difficult to recommend a canonical Path because as I mentioned, there is no replacement for these functions because that functions returned CST nodes and those not exist anymore.
msg381227 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-17 12:20
We should check of the 3 mentioned projects (mod_wsgi, 
kdevelop-python, unbound) use the removed functions to suggest a similar replacement. I understood that there is no drop-in replacement.

unbound does not use to get the parsed Python code as CST, but uses PyParser_SimpleParseFile() just to display an error message to stderr. I understand that PyParser_ASTFromFileObject() + PyErr_Print() could be used.

But PyParser_ASTFromFileObject() is low-level, it requires to pass an arena object. Maybe the *intent* here is to call compile() and display the error message? Pseudo-code:

---
fp = fopen(filename, "r");
bytes = readall(fp);
PyObject *builtins = PyEval_GetBuiltins();
obj = PyObject_CallMethod(builtins, "compile", "O", bytes);
Py_DECREF(bytes);
if (!obj) {
    PyErr_Print();
}
else {
    Py_DECREF(obj);
}
fclose(fp);
---

This code is non-trivial :-( Should we provide a *new* C function doing that?

  Input: filename
  Output: code object

Or maybe I just missed an existing function :-)
msg381232 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-17 13:41
> This code is non-trivial :-( Should we provide a *new* C function doing that?

We could discuss adding a new C function, but IMHO that code is not especially horrible or unreadable. I agree it could be simpler, though.
msg382129 - (view) Author: Igor Skochinsky (Igor.Skochinsky) Date: 2020-11-30 09:26
Hi All,

 It seems this patch removes some functions provided by the Stable ABI (PEP 384), most notably Py_CompileString. Was this the intention? If not, is there still a chance to reintroduce it before the release?
msg382130 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-30 09:43
> It seems this patch removes some functions provided by the Stable ABI (PEP 384), most notably Py_CompileString. Was this the intention? If not, is there still a chance to reintroduce it before the release?

The functions removal is intentional and was approved by the PEP 617.

But Py_CompileString() function was not removed, it's still in the master branch (future Python 3.10). Why do you think that it has been removed?
msg382134 - (view) Author: Igor Skochinsky (Igor.Skochinsky) Date: 2020-11-30 10:02
> But Py_CompileString() function was not removed, it's still in the master branch (future Python 3.10). Why do you think that it has been removed?

Thank you. It looked that way because of the removed block of lines in the commit 1ed83adb0e95305af858bd41af531e487f54fee7 (pythonrun.c). We were also getting a missing symbol error. We'll check again to be sure.
msg382285 - (view) Author: Igor Skochinsky (Igor.Skochinsky) Date: 2020-12-02 00:08
Attached is a sample program which works on 3.9 but fails linking with 3.10.0a2 

The .so is missing the symbol:

igor@LAPTOP:~/py_limited_api_example$ nm /home/igor/lib/libpython3.9.so | grep Py_CompileString
0000000000212720 T Py_CompileString
000000000020fe30 T Py_CompileStringExFlags
0000000000212730 T Py_CompileStringFlags
000000000020fd40 T Py_CompileStringObject

igor@LAPTOP:~/py_limited_api_example$ nm /home/igor/lib/libpython3.10.so | grep Py_CompileString
0000000000201a40 T Py_CompileStringExFlags
0000000000201980 T Py_CompileStringObject


Please stop breaking the Stable ABI :/
msg382287 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-02 01:07
Hm, I wonder if there's a typo here in pythonrun.c:

/* For use in Py_LIMITED_API */
#undef Py_CompileString
PyObject *
PyCompileString(const char *str, const char *filename, int start)
{
    return Py_CompileStringFlags(str, filename, start, NULL);
}

Shouldn't that function be named Py_CompileString (i.e. Py_ instead of Py)?

This seems to be old code, but there's normally a macro Py_CompileString() that translates to Py_CompileStringFlags() in pythonrun.h:

#ifdef Py_LIMITED_API
PyAPI_FUNC(PyObject *) Py_CompileString(const char *, const char *, int);
#else
#define Py_CompileString(str, p, s) Py_CompileStringExFlags(str, p, s, NULL, -1)
#define Py_CompileStringFlags(str, p, s, f) Py_CompileStringExFlags(str, p, s, f, -1)
.
.
.
msg382289 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-12-02 02:59
Seems that commit 1ed83adb0e95305af858bd41af531e487f54fee7 was deleting some functions that were not correctly covered by redirection macros. I have opened PR 23606 to restore those as they were.
msg382293 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-12-02 05:16
New changeset 46bd5ed94cf3d5e03f45eecf9afea1659980c8bf by Pablo Galindo in branch 'master':
bpo-40939: Restore some stable API functions incorrectly deleted (GH-23606)
https://github.com/python/cpython/commit/46bd5ed94cf3d5e03f45eecf9afea1659980c8bf
msg382299 - (view) Author: Igor Skochinsky (Igor.Skochinsky) Date: 2020-12-02 09:43
Thanks for the fix! I'd like to submit a test to avoid this and similar  issues in future. Are there any guidelines for this? Sorry if this is a wrong place to ask.
msg382323 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-12-02 16:28
I am currently working on test that checks that the stable API symbols are correctly exported. Unfortunately there is no official maintained list if those symbols, so is taking a while
msg382324 - (view) Author: Igor Skochinsky (Igor.Skochinsky) Date: 2020-12-02 16:47
> I am currently working on test that checks that the stable API symbols are correctly exported.

Thank you very much! For added motivation, the 3.8.0 release was unusable thanks to issue37633 which was somewhat similar (also in pythonrun). For reference, I've attached a list of symbols we currently use (a few more we have to import dynamically since they're not in the stable ABI but we'd like to keep that list as short as possible).
msg382325 - (view) Author: Nikita Nemkin (nnemkin) * Date: 2020-12-02 16:55
> Unfortunately there is no official maintained list if those symbols, so is taking a while

On Windows there is PC/python3dll.c.
msg382329 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-12-02 17:55
> On Windows there is PC/python3dll.c.

Even that is severely out of date, unfortunately. There are many functions that were removed in 3.8 and 3.7 that are still listed there.
msg382337 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-12-02 20:52
Opened https://bugs.python.org/issue42545 for that
msg387017 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-15 15:23
FYI the unbound project was fixed by calling Py_CompileString() on Python 3.9 and newer:
https://github.com/NLnetLabs/unbound/commit/e0d426ebb10653a78bf5c4053198f6ac19bfcd3e
msg396348 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-22 15:44
- The removed functions are still listed in the documentation; this is confusing.

- `struct _node` is not very usable in the C API, but what I saw in mod_wsgi was giving gave PyParser_* output to PyNode_Compile. I tried to write porting notes for this usage. It should also help with the "compile to check for syntax errors" case in unbound.
msg396488 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-24 12:57
New changeset 29987f72650b7cccee4df216c8297e8484a44e6a by Petr Viktorin in branch 'main':
bpo-40939: Remove documentation for `PyParser_*` & add porting notes (GH-26855)
https://github.com/python/cpython/commit/29987f72650b7cccee4df216c8297e8484a44e6a
msg396622 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-28 09:36
New changeset dc10264eb880ed63fcf42c17057f3f5d879a0a0c by Miss Islington (bot) in branch '3.10':
bpo-40939: Remove documentation for `PyParser_*` & add porting notes (GH-26855) (GH-26898)
https://github.com/python/cpython/commit/dc10264eb880ed63fcf42c17057f3f5d879a0a0c
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85111
2021-08-13 16:00:49terry.reedylinkissue41234 superseder
2021-06-28 09:36:20petr.viktorinsetmessages: + msg396622
2021-06-24 12:57:37miss-islingtonsetpull_requests: + pull_request25474
2021-06-24 12:57:35petr.viktorinsetmessages: + msg396488
2021-06-22 15:44:55petr.viktorinsetnosy: gvanrossum, terry.reedy, vstinner, ned.deily, petr.viktorin, nnemkin, Igor.Skochinsky, hroncok, lys.nikolaou, pablogsal, miss-islington, brandtbucher
messages: + msg396348
2021-06-22 15:40:39petr.viktorinsetnosy: + petr.viktorin

pull_requests: + pull_request25434
2021-02-15 15:23:14vstinnersetmessages: + msg387017
2020-12-02 20:52:28pablogsalsetmessages: + msg382337
2020-12-02 17:55:15pablogsalsetmessages: + msg382329
2020-12-02 16:55:26nnemkinsetmessages: + msg382325
2020-12-02 16:47:10Igor.Skochinskysetfiles: + py3.txt

messages: + msg382324
2020-12-02 16:28:32pablogsalsetmessages: + msg382323
2020-12-02 09:43:00Igor.Skochinskysetmessages: + msg382299
2020-12-02 05:17:21pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-12-02 05:16:39pablogsalsetmessages: + msg382293
2020-12-02 02:59:49pablogsalsetmessages: + msg382289
2020-12-02 02:58:28pablogsalsetpull_requests: + pull_request22474
2020-12-02 01:07:27gvanrossumsetmessages: + msg382287
2020-12-02 00:08:50Igor.Skochinskysetfiles: + py_limited_api_example.zip

messages: + msg382285
2020-11-30 10:02:25Igor.Skochinskysetmessages: + msg382134
2020-11-30 09:43:03vstinnersetmessages: + msg382130
2020-11-30 09:26:37Igor.Skochinskysetnosy: + Igor.Skochinsky
messages: + msg382129
2020-11-17 13:41:51pablogsalsetmessages: + msg381232
2020-11-17 12:20:30vstinnersetmessages: + msg381227
2020-11-16 23:26:40pablogsalsetmessages: + msg381182
2020-11-16 22:57:27hroncoksetmessages: + msg381176
2020-11-16 22:40:13lys.nikolaousetmessages: + msg381172
2020-11-16 18:46:46miss-islingtonsetmessages: + msg381140
2020-11-16 18:28:02lys.nikolaousetstage: resolved -> patch review
pull_requests: + pull_request22212
2020-11-16 18:21:52pablogsalsetmessages: + msg381137
2020-11-16 18:13:41lys.nikolaousetmessages: + msg381136
2020-11-16 15:23:11pablogsalsetmessages: + msg381112
2020-11-16 15:03:44vstinnersetmessages: + msg381103
2020-11-16 12:51:58hroncoksetnosy: + hroncok
messages: + msg381095
2020-11-12 23:35:38pablogsalsetmessages: + msg380863
2020-11-12 22:10:44gvanrossumsetmessages: + msg380855
2020-11-12 15:18:32vstinnersetmessages: + msg380828
2020-11-12 15:13:02vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg380826
2020-09-18 01:57:12terry.reedysetmessages: + msg377083
2020-09-18 01:07:28terry.reedysetnosy: + terry.reedy

pull_requests: + pull_request21347
2020-07-27 19:53:06miss-islingtonsetmessages: + msg374417
2020-07-27 19:00:50gvanrossumsetmessages: + msg374416
2020-07-27 18:57:57lys.nikolaousetpull_requests: + pull_request20783
2020-07-27 18:38:59gvanrossumsetstatus: open -> closed
resolution: fixed
messages: + msg374413

stage: patch review -> resolved
2020-07-27 18:31:35gvanrossumsetpull_requests: + pull_request20782
2020-07-27 18:20:44gvanrossumsetmessages: + msg374410
2020-07-26 21:47:49pablogsalsetpull_requests: + pull_request20774
2020-07-26 21:46:40gvanrossumsetmessages: + msg374346
2020-07-26 03:22:31gvanrossumsetpull_requests: + pull_request20765
2020-07-25 21:25:46gvanrossumsetmessages: + msg374282
2020-06-25 14:46:53ned.deilysetmessages: + msg372370
2020-06-25 14:23:10ned.deilysetnosy: + ned.deily
pull_requests: + pull_request20311
2020-06-22 00:07:49miss-islingtonsetmessages: + msg372027
2020-06-21 23:49:04lys.nikolaousetmessages: + msg372026
2020-06-21 23:47:50pablogsalsetmessages: + msg372025
2020-06-21 23:46:43lys.nikolaousetpull_requests: + pull_request20207
2020-06-21 21:34:14vstinnersetmessages: + msg372023
2020-06-21 03:06:33pablogsalsetpull_requests: + pull_request20197
2020-06-21 01:16:01pablogsalsetmessages: + msg371977
2020-06-20 19:26:43lys.nikolaousetpull_requests: + pull_request20190
2020-06-20 18:34:56lys.nikolaousetpull_requests: + pull_request20186
2020-06-20 18:07:29pablogsalsetmessages: + msg371953
2020-06-20 16:53:14lys.nikolaousetpull_requests: + pull_request20179
2020-06-20 16:31:55gvanrossumsetmessages: + msg371942
2020-06-20 15:45:54lys.nikolaousetmessages: + msg371940
2020-06-20 10:01:44nnemkinsetnosy: + nnemkin
messages: + msg371929
2020-06-12 00:55:41pablogsalsetmessages: + msg371327
2020-06-12 00:25:28pablogsalsetpull_requests: + pull_request20018
2020-06-11 18:29:20pablogsalsetmessages: + msg371316
2020-06-11 17:18:01brandtbuchersetnosy: + brandtbucher
2020-06-11 17:08:13miss-islingtonsetmessages: + msg371311
2020-06-11 16:46:54pablogsalsetmessages: + msg371310
2020-06-11 16:46:24pablogsalsetpull_requests: + pull_request20013
2020-06-11 16:43:27pablogsalsetpull_requests: + pull_request20012
2020-06-11 16:36:39vstinnersetmessages: + msg371308
2020-06-11 16:30:50pablogsalsetmessages: + msg371307
2020-06-11 16:09:24gvanrossumsetmessages: + msg371301
2020-06-11 14:55:52lys.nikolaousetmessages: + msg371292
2020-06-11 14:48:46lys.nikolaousetmessages: + msg371291
2020-06-11 14:32:28pablogsalsetmessages: + msg371290
2020-06-11 14:29:04vstinnersetmessages: + msg371289
2020-06-11 14:27:44gvanrossumsetmessages: + msg371288
2020-06-11 13:17:38vstinnersetmessages: + msg371271
2020-06-11 13:13:24lys.nikolaousetpull_requests: + pull_request19998
2020-06-11 13:08:00lys.nikolaousetmessages: + msg371270
2020-06-11 13:02:40miss-islingtonsetmessages: + msg371268
2020-06-11 13:01:07pablogsalsetmessages: + msg371267
2020-06-11 12:59:54lys.nikolaousetmessages: + msg371266
2020-06-11 12:50:41vstinnersetnosy: + vstinner
messages: + msg371265
2020-06-11 12:45:33miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19997
2020-06-11 12:45:22pablogsalsetmessages: + msg371264
2020-06-11 11:02:44lys.nikolaousetpull_requests: + pull_request19996
2020-06-10 11:25:59pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request19972
2020-06-10 11:25:28pablogsalsetnosy: + gvanrossum, lys.nikolaou
2020-06-10 11:25:17pablogsalcreate