Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python modules not linking to libpython causes issues for RTLD_LOCAL system-wide #80934

Open
reimar mannequin opened this issue Apr 29, 2019 · 7 comments
Open

Python modules not linking to libpython causes issues for RTLD_LOCAL system-wide #80934

reimar mannequin opened this issue Apr 29, 2019 · 7 comments
Labels
3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@reimar
Copy link
Mannequin

reimar mannequin commented Apr 29, 2019

BPO 36753
Nosy @doko42, @vstinner, @ericvw, @pablogsal, @furiel, @MrAnno
Files
  • pytest.tar.gz
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2019-04-29.16:16:00.782>
labels = ['3.8', 'type-bug', 'library']
title = 'Python modules not linking to libpython causes issues for RTLD_LOCAL system-wide'
updated_at = <Date 2020-04-06.21:57:11.332>
user = 'https://bugs.python.org/reimar'

    bugs.python.org fields:

    activity = <Date 2020-04-06.21:57:11.332>
actor = 'Joshua Merchant'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-04-29.16:16:00.782>
creator = 'reimar'
dependencies = []
files = ['48291']
hgrepos = []
issue_num = 36753
keywords = []
message_count = 7.0
messages = ['341094', '341096', '341105', '349232', '349255', '349736', '349739']
nosy_count = 8.0
nosy_names = ['doko', 'vstinner', 'ericvw', 'pablogsal', 'reimar', 'furiel', 'Joshua Merchant', 'MrAnno']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36753'
versions = ['Python 3.8']
    @reimar
    Copy link
    Mannequin Author

    reimar mannequin commented Apr 29, 2019

    Most affected platforms: RedHat and Debian, but with the changes from bpo-21536 probably all Linux distributions will be affected.

    bpo-34814 and bpo-21536 and https://bugzilla.redhat.com/show_bug.cgi?id=1585201 make statements along the lines of "In short, RTLD_LOCAL is not supported."
    This might have been considered a reasonable stance because of the specific example opening libpython directly.
    However Python modules not linking to libpython also breaks things when libpython is loaded in the most indirect ways via dlopen.
    E.g. dlopen("libA.so", RTLD_LOCAL | RTLD_NOW)
    libA might have linked against libB, libB against libC and libC might optionally link against libpython.

    As a developer generally cannot really know if some library might ever pull in a most indirect reference to libpython, not supporting RTLD_LOCAL in Python essentially means RTLD_LOCAL can NEVER EVER be used safely.

    A test-case that will fail the import command when modules have not been linked against libpython is attached (demonstrating only one layer of indirection, but much more complex cases are of course possible).
    You will need to adjust the (include, lib) paths in test.sh for your Python version, it was written to demonstrate the issue against RedHat's modifications of Python 2.7 (to my knowledge, RedHat and Debian has been affected by this issue much longer than mainline Python).

    While dlmopen is an alternative with similar behaviour to RTLD_LOCAL on recent Linux versions for this case, it is not portable.

    @reimar reimar mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Apr 29, 2019
    @doko42
    Copy link
    Member

    doko42 commented Apr 29, 2019

    why is this an issue on Debian? Debian is already not linking with libpython.

    @reimar
    Copy link
    Mannequin Author

    reimar mannequin commented Apr 29, 2019

    The modules not linking against libpython CAUSES this issue, thus Debian being affected even for old Python versions before bpo-21536.

    @MrAnno
    Copy link
    Mannequin

    MrAnno mannequin commented Aug 8, 2019

    I'd like to mention a real-world example for this issue: syslog-ng with its
    plugin system.

    Plugins are loaded using dlopen() after the initial plugin discovery. RTLD_LOCAL
    would be a reasonable choice, since symbols in plugins should not pollute the
    global namespace (so plugins using different versions of OpenSSL, Python, etc.
    would be possible).

    Since we have Python-based plugins, we had to use RTLD_GLOBAL instead (with RTLD_LAZY to
    reduce the possibility of conflicts) as a workaround.

    Note: The latest Python 3 version seems to be working with RTLD_LOCAL as everything
    under lib-dynload/ linked against libpython.

    @pablogsal
    Copy link
    Member

    In the What's New section of 3.8 (https://docs.python.org/3.8/whatsnew/3.8.html#changes-in-the-c-api) it indicates that:

    On Unix, C extensions are no longer linked to libpython except on Android and Cygwin. When Python is embedded, libpython must not be loaded with RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it was already not possible to load C extensions which were not linked to libpython, like C extensions of the standard library built by the *shared* section of Modules/Setup. (Contributed by Victor Stinner in bpo-21536.)

    So if you are embedding python by dlopen'ing libpython.so you should use RTLD_GLOBAL to make sure everyone is using the same symbols. You have more information on the rationale and decisions in bpo-21536.

    Is your use-case not cover by the arguments in bpo-21536? What is the problem of embedding by dlopening libpython using RTLD_GLOBAL?

    @pablogsal pablogsal added the 3.8 only security fixes label Aug 8, 2019
    @reimar
    Copy link
    Mannequin Author

    reimar mannequin commented Aug 14, 2019

    I'm not sure how I can explain it much better, I even wrote example code after all, but I'll try...

    So if you are embedding python by dlopen'ing libpython.so

    Neither me nor Laszlo are using/embedding or otherwise involving Python (directly/intentionally at least).
    We just want to load some .so file.
    That .so itself might then use libpython. Or use a library that uses libpython. Or uses a library that uses a library .... that uses libpython.
    And how could we know whether SOME library down that dependency chain uses libpython or not?
    The result is that now EVERY SINGLE LIBRARY IN THE WHOLE SYSTEM needs to be loaded with RTLD_GLOBAL.
    Because a library 50 dependencies down that uses python might break otherwise and there's not really any way to know.

    Just to try be very clear: We are not users of libpython, we do not write or use any python code ourselves (except through indirect dependencies) and we are still hit by this issue.

    @reimar
    Copy link
    Mannequin Author

    reimar mannequin commented Aug 14, 2019

    I guess one way this could be "solved" would be by libpython doing a dlopen on itself with RTLD_GLOBAL on Python initialization.
    I wouldn't bet that that wouldn't end up with some very interesting unintended consequences as well though.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @doko42 @pablogsal