New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
simple bug in mmap size check #57357
Comments
The condition contradicts the exception text:
if (offset >= st.st_size) {
PyErr_SetString(PyExc_ValueError,
"mmap offset is greater than file size");
return NULL;
}
The condition should be changed to (offset > st.st_size), similar to the later condition which is correct:
} else if (offset + (size_t)map_size > st.st_size) {
PyErr_SetString(PyExc_ValueError,
"mmap length is greater than file size");
return NULL;
} The patch is attached. |
Well, you have to understand what this code does: it tries to prevent non-meaningful offsets. If the offset is equal to the file size, mapping from that offset would not map anything in the file (and the system call may actually fail). |
First of all, it doesn't fail (at least on Linux), I tested it before posting. And the latest, it's not like I'm just stalking around and reading Python sources for fun. It's a real and pretty valid case, I hit it while upgrading our production code to python 2.7.
I'm using NumPy (linear algebra module) that uses Python's core mmap module under the hood.
In NumPy, it's pretty valid to have arrays of size 0.
I have a file with a fixed-size header that holds size of the array and some other meta-data. I mmap this file as a NumPy array using the offset equal to header size. Of course, if there is no data in the array then the file will be just header, and the offset will be equal to the size of the file - here is where this bug hits us as I can't load this file with Python 2.7.2 anymore (while I was able to with Python 2.5).
This patch fixes this and everything works as expected, giving an array with zero dimensions ('shape' in terms of NumPy):
>>> x.shape
(0,)
>>> x.size
0 Please kindly consider applying the patch. |
I don't think it makes sense to accept mmap'ing empty contents when at offset "n" but not at offset "n + 1". Either we remove the check entirely and let people deal with the consequences, or we keep the check as-is. |
Well, "n+1" is clearly outside the file, wile "n" is within and therefore valid. Also, if your position is to forbid zero-size mmapping completely, then the checks inside "if (map_size == 0) {" don't make any sense, especially as they may or may fail. Removing the check completely is a viable option too, it was already requested for special files: I believe users should have an ability to enjoy whatever their OS provides, and "deal with the consequences" as you said. |
tried on newer Linux - crashes with my patch. |
Why?
Hmmm.
That's exactly why we perform such checks. Here's what POSIX says: Since we usually want to avoid implementation-defined behavior (and crashes), I think it's better to stick with the current checks (note that issue bpo-12556 concerned a really corner case: /proc entries supporting mmap but reporting a zero-length).
Alright, closing then. |
Regarding the contradiction, it's probably still worth changing the exception message to "mmap offset is greater than _or equal to_ file size", to match the condition. Just 'greater than' means the '>' check, not the '>=' check from the code, mathematically. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: