Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C API] Functions not exported with PyAPI_FUNC() #89479

Closed
vstinner opened this issue Sep 29, 2021 · 18 comments
Closed

[C API] Functions not exported with PyAPI_FUNC() #89479

vstinner opened this issue Sep 29, 2021 · 18 comments
Labels
3.11 only security fixes topic-C-API

Comments

@vstinner
Copy link
Member

BPO 45316
Nosy @vstinner, @markshannon, @ericsnowcurrently, @serhiy-storchaka, @corona10, @shihai1991
PRs
  • bpo-45316: Move private PyCode C API to internal C API #31576
  • bpo-45316: Move private PyDict functions to internal C API #31577
  • bpo-45316: Move private functions to internal C API #31579
  • bpo-45316: Move _PyArg_Fini() to internal C API #31580
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-09-29.08:18:04.824>
    labels = ['expert-C-API', '3.11']
    title = '[C API] Functions not exported with PyAPI_FUNC()'
    updated_at = <Date 2022-02-25.16:49:14.761>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2022-02-25.16:49:14.761>
    actor = 'eric.snow'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['C API']
    creation = <Date 2021-09-29.08:18:04.824>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45316
    keywords = ['patch']
    message_count = 10.0
    messages = ['402827', '402831', '402844', '403641', '413995', '413996', '414001', '414007', '414008', '414019']
    nosy_count = 6.0
    nosy_names = ['vstinner', 'Mark.Shannon', 'eric.snow', 'serhiy.storchaka', 'corona10', 'shihai1991']
    pr_nums = ['31576', '31577', '31579', '31580']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue45316'
    versions = ['Python 3.11']

    @vstinner
    Copy link
    Member Author

    The Python C API contains multiple functions which are not exported with PyAPI_FUNC() and so are not usable.

    Public functions:
    ---

    void PyLineTable_InitAddressRange(const char *linetable, Py_ssize_t length, int firstlineno, PyCodeAddressRange *range);
    int PyLineTable_NextAddressRange(PyCodeAddressRange *range);
    int PyLineTable_PreviousAddressRange(PyCodeAddressRange *range);

    => Either make this functions private ("_Py" prefix) and move them to the internal C API, or add PyAPI_FUNC().

    Private functions:
    ---

    int _PyCode_InitAddressRange(PyCodeObject* co, PyCodeAddressRange *bounds);
    int _PyCode_InitEndAddressRange(PyCodeObject* co, PyCodeAddressRange* bounds);
    
    PyDictKeysObject *_PyDict_NewKeysForClass(void);
    Py_ssize_t _PyDict_KeysSize(PyDictKeysObject *keys);
    uint32_t _PyDictKeys_GetVersionForCurrentState(PyDictKeysObject *dictkeys);
    Py_ssize_t _PyDictKeys_StringLookup(PyDictKeysObject* dictkeys, PyObject *key);
    
    PyObject *_PyDict_Pop_KnownHash(PyObject *, PyObject *, Py_hash_t, PyObject *);
    PyObject *_PyDict_FromKeys(PyObject *, PyObject *, PyObject *);
    int _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, PyObject *name, PyObject *value);
    PyObject *_PyDict_LoadGlobal(PyDictObject *, PyDictObject *, PyObject *);
    Py_ssize_t _PyDict_GetItemHint(PyDictObject *, PyObject *, Py_ssize_t, PyObject **);
    
    PyFrameObject* _PyFrame_New_NoTrack(struct _interpreter_frame *, int);
    int PySignal_SetWakeupFd(int fd);
    uint32_t _PyFunction_GetVersionForCurrentState(PyFunctionObject *func);
    PyObject *_PyGen_yf(PyGenObject *);
    PyObject *_PyCoro_GetAwaitableIter(PyObject *o);
    PyObject *_PyAsyncGenValueWrapperNew(PyObject *);
    void _PyArg_Fini(void);
    int _Py_CheckPython3(void);

    => I suggest to move all these declarations to the internal C API.

    Moreover, Include/moduleobject.h contains:
    ---

    #ifdef Py_BUILD_CORE
    extern int _PyModule_IsExtension(PyObject *obj);
    #endif

    IMO this declaration should be moved to the internal C API.

    See also bpo-45201 about PySignal_SetWakeupFd().

    @vstinner vstinner added 3.11 only security fixes topic-C-API labels Sep 29, 2021
    @serhiy-storchaka
    Copy link
    Member

    Agree.

    We should also review all recently added functions with PyAPI_FUNC(). If they are not intended to be public API, PyAPI_FUNC() should be removed and declarations moved.

    @markshannon
    Copy link
    Member

    Regarding these three functions:

    void PyLineTable_InitAddressRange(const char *linetable, Py_ssize_t length, int firstlineno, PyCodeAddressRange *range);
    int PyLineTable_NextAddressRange(PyCodeAddressRange *range);
    int PyLineTable_PreviousAddressRange(PyCodeAddressRange *range);

    They are explicitly not part of the C API (limited or otherwise): https://www.python.org/dev/peps/pep-0626/#out-of-process-debuggers-and-profilers

    @vstinner
    Copy link
    Member Author

    See also bpo-45431: [C API] Rename CFrame or hide it to only export names starting with Py.

    @vstinner
    Copy link
    Member Author

    New changeset 4a0c7a1 by Victor Stinner in branch 'main':
    bpo-45316: Move private PyCode C API to internal C API (GH-31576)
    4a0c7a1

    @vstinner
    Copy link
    Member Author

    New changeset 8ddbdd9 by Victor Stinner in branch 'main':
    bpo-45316: Move private PyDict functions to internal C API (GH-31577)
    8ddbdd9

    @vstinner
    Copy link
    Member Author

    New changeset 8f2a337 by Victor Stinner in branch 'main':
    bpo-45316: Move private functions to internal C API (GH-31579)
    8f2a337

    @vstinner
    Copy link
    Member Author

    New changeset f780d96 by Victor Stinner in branch 'main':
    bpo-45316: Move _PyArg_Fini() to internal C API (GH-31580)
    f780d96

    @vstinner
    Copy link
    Member Author

    Update: only two remaining functions are not exported by the public C API:

    int PySignal_SetWakeupFd(int fd);
    int _Py_CheckPython3(void);

    @ericsnowcurrently
    Copy link
    Member

    Thanks for working on this, Victor.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @penguin-wwy
    Copy link
    Contributor

    penguin-wwy commented Apr 28, 2022

    @vstinner hello, I find #31576 cause the _opcode module to fail to compile (config --enable-pystats)

    renaming "_opcode" since importing it failed: dlopen(/build/lib.macosx-12.1-arm64-3.11-pydebug/_opcode.cpython-311d-darwin.so, 0x0002): symbol not found in flat namespace '__Py_GetSpecializationStats'
    

    Maybe this declaration should be written to another file

    @vstinner
    Copy link
    Member Author

    Please open a new issue for your build error. It's unrelated to this issue.

    @vstinner
    Copy link
    Member Author

    I close this issue. I don't know how to handle PySignal_SetWakeupFd() and _Py_CheckPython3().

    @vstinner
    Copy link
    Member Author

    Please open a new issue for your build error. It's unrelated to this issue.

    Oh wait, I misunderstood the issue. You're right, it's a regression caused by commit 4a0c7a1. I wrote PR #92011 to fix it.

    I didn't notice that this function is actually used by a C extension built as a shared library, but only for a special build.

    @vstinner vstinner reopened this Apr 28, 2022
    @penguin-wwy
    Copy link
    Contributor

    Thanks

    vstinner added a commit that referenced this issue Apr 28, 2022
    When Python is built with "./configure --enable-pystats" (if the
    Py_STATS macro is defined), the _Py_GetSpecializationStats() function
    must be exported, since it's used by the _opcode extension which is
    built as a shared library.
    @vstinner
    Copy link
    Member Author

    I wrote PR #92011 to fix it.

    Merged. I close again the issue.

    @nathan454
    Copy link

    #89479 (comment)

    How are you supposed to use these functions than?
    If they're private functions, and their symbol isn't exported, how can I link to these functions?
    For now I copy paste them, but what if they change?

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 6, 2022

    How are you supposed to use these functions than? If they're private functions, and their symbol isn't exported, how can I link to these functions?

    They are not public on purpose. If you consider that they should be public, please open a feature request (a new issue). This issue is now closed, it's a bad place to discuss it.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 only security fixes topic-C-API
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants