Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

segfault with gunicorn and a library made with cython bindings #87057

Closed
amirouche mannequin opened this issue Jan 11, 2021 · 7 comments
Closed

segfault with gunicorn and a library made with cython bindings #87057

amirouche mannequin opened this issue Jan 11, 2021 · 7 comments
Labels
3.8 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@amirouche
Copy link
Mannequin

amirouche mannequin commented Jan 11, 2021

BPO 42891
Nosy @vstinner, @amirouche

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-11.18:07:31.672>
created_at = <Date 2021-01-11.14:48:27.268>
labels = ['3.8', 'type-crash']
title = 'segfault with gunicorn and a library made with cython bindings'
updated_at = <Date 2021-01-11.18:07:31.672>
user = 'https://github.com/amirouche'

bugs.python.org fields:

activity = <Date 2021-01-11.18:07:31.672>
actor = 'amirouche'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-11.18:07:31.672>
closer = 'amirouche'
components = []
creation = <Date 2021-01-11.14:48:27.268>
creator = 'amirouche'
dependencies = []
files = []
hgrepos = []
issue_num = 42891
keywords = []
message_count = 7.0
messages = ['384836', '384838', '384839', '384840', '384843', '384845', '384852']
nosy_count = 2.0
nosy_names = ['vstinner', 'amirouche']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue42891'
versions = ['Python 3.8']
@amirouche
Copy link
Mannequin Author

amirouche mannequin commented Jan 11, 2021

Here is a simple way to reproduce:

from lsm import LSM


db = LSM('db.sqlite')

def app(environ, start_response):
    """Simplest possible application object"""

    for (index, (key, value)) in enumerate(db[b'\x00':b'\xFF']):
        pass

    start_response(b'200', {})
    return b''

db.close()

In my real program, if I add 'global db' in the function app, it does not segfault.

program: https://git.sr.ht/~amirouche/segfault
trace: https://wyz.fr/0I-MO

@amirouche amirouche mannequin added 3.8 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump labels Jan 11, 2021
@vstinner
Copy link
Member

I put the code in a "bla.p" script and I ran it with Python 3.8 on Fedora 33:

python3.8 -m venv env
env/bin/python -m pip install lsm-db
env/bin/python bla.py

It doesn't crash. Please provide a script reproducing the issue.

Note: lsm-db cannot be installed in Python 3.9 (errors about the removed PyTypeObject.tp_print member in C).

@amirouche
Copy link
Mannequin Author

amirouche mannequin commented Jan 11, 2021

You need to run the program with the following:

python -X dev -c "from gunicorn.app.wsgiapp import run; run()" --workers=1 foobar:app

where foobar.py is the code from the previous message.

The crash happen when, the function app is executed, hence you need to call in another console:

curl http://localhost:8000

note: lsm package can be installed with pip install lsm-db
note2: lsm db can not be installed with py3.9

@vstinner
Copy link
Member

I get a crash without any code, just by running gunicorn. It looks like a bug in gunicorn.

---

$ python3.8 -m venv env
$ env/bin/python -m pip install gunicorn
(..)
Successfully installed gunicorn-20.0.4

$ ./env/bin/python -X dev -c 'from gunicorn.app.wsgiapp import run; run()' foobar:app 
...)
[2021-01-11 16:14:11 +0100] [32107] [ERROR] Exception in worker process
Traceback (most recent call last):
  (...)
  File "/home/vstinner/z/env/lib64/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/vstinner/z/env/lib64/python3.8/site-packages/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  (...)
ModuleNotFoundError: No module named 'foobar'

[2021-01-11 16:14:11 +0100] [32107] [INFO] Worker exiting (pid: 32107)
sys:1: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8000)>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Debug memory block at address p=0x55cfa92154a0: API '1'
3271705103877500672 bytes originally requested
The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
at p-7: 0x00 *** OUCH
at p-6: 0x00 *** OUCH
at p-5: 0x00 *** OUCH
at p-4: 0x00 *** OUCH
at p-3: 0x00 *** OUCH
at p-2: 0x00 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 8 pad bytes at tail=0x2d67c444d794c3a0 are [2021-01-11 16:14:11 +0100] [32113] [INFO] Booting worker with pid: 32113

(...)
---

You should report the issue to gunicorn: https://github.com/benoitc/gunicorn/issues

@amirouche
Copy link
Mannequin Author

amirouche mannequin commented Jan 11, 2021

ModuleNotFoundError: No module named 'foobar'

That is not a segfault. The problem I am reporting is a segfault.

It can be reproduced with uvicorn as follow:

from lsm import LSM


db = LSM('db.sqlite')


async def app(scope, receive, send):
    assert scope['type'] == 'http'
    global db
    for (index, (key, value)) in enumerate(db[b'\x00':b'\xFF']):
        pass
await send({
    'type': 'http.response.start',
    'status': 200,
    'headers': [
        [b'content-type', b'text/plain'],
    ],
})
await send({
    'type': 'http.response.body',
    'body': b'Hello, world!',
})

db.close()

Run the above program with: python -X dev -m uvicorn bobo:app

Then in another console: curl http://localhost:8000/

Here is the output:

$ python -X dev -m uvicorn bobo:app
INFO:     Started server process [3580316]
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
Fatal Python error: Segmentation fault

Current thread 0x00007fbb6e49b740 (most recent call first):
File "./bobo.py", line 10 in app
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45 in __call__
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 394 in run_asgi
File "/usr/lib/python3.8/asyncio/events.py", line 81 in _run
File "/usr/lib/python3.8/asyncio/base_events.py", line 1851 in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 570 in run_forever
File "/usr/lib/python3.8/asyncio/base_events.py", line 603 in run_until_complete
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/server.py", line 48 in run
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/main.py", line 386 in run
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/main.py", line 362 in main
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 610 in invoke
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 1066 in invoke
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 782 in main
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 829 in __call__
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/main.py", line 4 in <module>
File "/usr/lib/python3.8/runpy.py", line 87 in _run_code
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
Segmentation fault (core dumped)

@amirouche amirouche mannequin reopened this Jan 11, 2021
@amirouche amirouche mannequin reopened this Jan 11, 2021
@amirouche
Copy link
Mannequin Author

amirouche mannequin commented Jan 11, 2021

I tried to reduce the program by replacing gunicorn / uvicorn dependency with threading.Thread and multiprocess.Pool without success.

@amirouche
Copy link
Mannequin Author

amirouche mannequin commented Jan 11, 2021

The problem is prolly in lsm-db.

ref: coleifer/python-lsm-db#20

Sorry for the noise.

@amirouche amirouche mannequin closed this as completed Jan 11, 2021
@amirouche amirouche mannequin closed this as completed Jan 11, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.8 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vstinner