This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: PyMarshal_ReadLastObjectFromFile
Type: Stage:
Components: Interpreter Core Versions: Python 2.6
Status: closed Resolution: out of date
Dependencies: Superseder: python script segment fault at PyMarshal_ReadLastObjectFromFile in import_submodule
View: 7332
Assigned To: Nosy List: christian.heimes, mark.dickinson, paulfelix, pitrou, terry.reedy
Priority: normal Keywords: patch

Created on 2003-07-12 20:42 by paulfelix, last changed 2022-04-10 16:09 by admin. This issue is now closed.

File name Uploaded Description Edit
marshal.c.patch paulfelix, 2003-07-12 20:42 The patch file
marshal.c.patch2 paulfelix, 2003-07-14 16:11 The patch file (alternative approach)
Messages (6)
msg44265 - (view) Author: Paul Felix (paulfelix) Date: 2003-07-12 20:42
This is a new implementation of the performance boost
in PyMarshal_ReadLastObjectFromFile where the pyc file
data is loaded into memory before parsing.

The original logic is as follows:
- Get the file size.
- If the file size is 16K or less, allocate a 16k char
buffer on the call stack and load the file data into
the buffer.
- Otherwise, if file size is 256k or less, malloc a
temp char buffer from the heap and load the file data
into the buffer.
- Otherwise, don't use a buffer.

Allocating 16k on the call stack can cause a stack
overflow when doing nested imports in a secondary
thread.  The default thread stack size on some systems
is quite small (64k on HP-UX!).  Also,
allocating/freeing memory on the heap for this purpose
may not be warranted.

This patch proposes a new implentation that allocates a
static 256k char buffer and uses a thread lock to keep
it thread-safe.  The lock/unlock function logic was
taken from import.c.

Note: performance tests show an improvment on some
platforms when the static char buffer is defined at the
file level instead of inside the function.
msg44266 - (view) Author: Paul Felix (paulfelix) Date: 2003-07-14 16:11
Logged In: YES 

Adding 256k to the core footprint is probably not acceptable
either. How about 16k? Patch 2 is a compromise that uses the
original logic, but allocates the 16k buffer from static memory.
msg59188 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-01-04 00:18
Is the patch still interesting for Python 2.6?
msg59249 - (view) Author: Paul Felix (paulfelix) Date: 2008-01-04 19:14
Maybe not. With the addition of thread.stack_size([size]), it's fairly
easy to get around the problem by bumping up the stack size. It would be
nice if PyMarshal_ReadLastObjectFromFile didn't allocate a small file
buffer on the stack (think of the impact when *recursively* importing
many .pyc files).
msg104543 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-04-29 17:38
Can this be closed as either out-of-date or rejected?
msg104545 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-04-29 17:50
I think this is superseded by issue 7332, which is now fixed in trunk (but not in 2.6).
Date User Action Args
2022-04-10 16:09:57adminsetgithub: 38844
2010-04-29 17:50:49mark.dickinsonsetstatus: open -> closed

nosy: + pitrou, mark.dickinson
messages: + msg104545

superseder: python script segment fault at PyMarshal_ReadLastObjectFromFile in import_submodule
resolution: remind -> out of date
2010-04-29 17:38:01terry.reedysetstatus: pending -> open
nosy: + terry.reedy
messages: + msg104543

2008-01-04 19:14:03paulfelixsetmessages: + msg59249
2008-01-04 00:18:31christian.heimessetstatus: open -> pending
resolution: remind
messages: + msg59188
nosy: + christian.heimes
versions: + Python 2.6, - Python 2.3
2003-07-12 20:42:08paulfelixcreate