classification
Title: mmap: add empty file check prior to offset check
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jcea Nosy List: Steven.Willis, jcea, neologix, python-dev, skrah
Priority: normal Keywords: patch

Created on 2012-08-15 17:46 by Steven.Willis, last changed 2012-09-10 21:35 by jcea. This issue is now closed.

Files
File name Uploaded Description Edit
issue15676.patch Steven.Willis, 2012-09-07 02:29 Better mmap error message for empty file plus patch for python2.7 mmap empty file check review
issue15676-3.1.patch Steven.Willis, 2012-09-09 14:52 patch for python3.1 mmap empty file check review
issue15676-3.2.patch Steven.Willis, 2012-09-09 14:59 patch for python3.2 mmap empty file check review
issue15676-default.patch Steven.Willis, 2012-09-09 15:06 patch for default branch mmap empty file check review
issue15676-default.patch Steven.Willis, 2012-09-09 20:16 Updated patch with better test review
Messages (19)
msg168314 - (view) Author: Steven Willis (Steven.Willis) Date: 2012-08-15 17:46
There are a number of issues dealing with the offset and length checks in offset, such as issue12556.

I'm running into this issue as well, but with a normal file that happens to be empty. I'm trying to access it with:

mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_READ)

So the length and offset should be calculated automatically. The man page for mmap says:

"SUSv3 specifies that mmap() should fail if length is 0.  However, in kernels before 2.6.12, mmap() succeeded in this case: no mapping was created and the call returned  addr.   Since  kernel  2.6.12, mmap() fails with the error EINVAL for this case."

So alright, mmapping an empty file is now allowed. But, could the check for an empty file be done prior to the check for the offset exceeding the size of the file? It would be much clearer in the cases where an empty (regular or otherwise) file was mmapped if the error message were something like "empty files cannot be mapped" insted of "offset is greater than file size" since I didn't even set the offset.
msg168323 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-08-15 20:08
I do agree. Solaris also returns an error if len=0.

Could you please, provide patches for 2.7, 3.2 and 3.3?
msg169967 - (view) Author: Steven Willis (Steven.Willis) Date: 2012-09-07 02:29
Here's a patch for 2.7. I don't know if it cleanly applies to the rest.
msg170106 - (view) Author: Steven Willis (Steven.Willis) Date: 2012-09-09 14:52
Here's a patch for 3.1
msg170107 - (view) Author: Steven Willis (Steven.Willis) Date: 2012-09-09 14:59
Here's a patch for 3.2
msg170108 - (view) Author: Steven Willis (Steven.Willis) Date: 2012-09-09 15:06
Here's a branch against the default branch in mercurial. I couldn't find a branch for 3.3 or 3.4.
msg170111 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-09-09 15:39
Thanks for the patches.
Note that you don't have to provide a patch for each branch, it's usually the committer's job.

The patch looks good, but the test could be rewritten with assertRaisesRegex():
http://docs.python.org/dev/library/unittest.html#unittest.TestCase.assertRaisesRegex
msg170133 - (view) Author: Steven Willis (Steven.Willis) Date: 2012-09-09 20:16
Sorry, I thought that's what jcea was asking for. Here's an updated patch for the default branch in mercurial that uses assertRaisesRegex in the test.
msg170140 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-09-09 22:15
Having separate patches is useful if the patch can not be applied cleanly to different python branches. This is specially true with 2.7/3.x.

Checking the fix and committing it to 2.7, 3.2 and 3.3 (probably available in 3.3.1, not 3.3.0, already in release candidate).
msg170145 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-09 23:24
New changeset d85f80b31b54 by Jesus Cea in branch '3.2':
Closes #15676: mmap: add empty file check prior to offset check
http://hg.python.org/cpython/rev/d85f80b31b54

New changeset f962ec8e47a1 by Jesus Cea in branch 'default':
Closes #15676: mmap: add empty file check prior to offset check
http://hg.python.org/cpython/rev/f962ec8e47a1

New changeset 27837a33790d by Jesus Cea in branch '2.7':
Closes #15676: mmap: add empty file check prior to offset check
http://hg.python.org/cpython/rev/27837a33790d
msg170162 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-09-10 06:56
You forgot to add an entry in Misc/ACKS.
msg170212 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-10 18:20
New changeset 9ed2e8307e60 by Jesus Cea in branch '2.7':
#15676: Proper attribution in Misc/ACKS
http://hg.python.org/cpython/rev/9ed2e8307e60

New changeset 4f21f7532038 by Jesus Cea in branch '3.2':
#15676: Proper attribution in Misc/ACKS
http://hg.python.org/cpython/rev/4f21f7532038

New changeset 4fdc6c501e63 by Jesus Cea in branch 'default':
MERGE: #15676: Proper attribution in Misc/ACKS
http://hg.python.org/cpython/rev/4fdc6c501e63
msg170228 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-09-10 20:12
For some reason all Windows buildbots are failing since f962ec8e47a1:

======================================================================
ERROR: test_entire_file (test.test_mmap.MmapTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_mmap.py", line 19, in setUp
    os.unlink(TESTFN)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '@test_2636_tmp'

======================================================================
ERROR: test_error (test.test_mmap.MmapTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_mmap.py", line 19, in setUp
    os.unlink(TESTFN)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '@test_2636_tmp'

======================================================================


[...]
msg170232 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-09-10 20:33
This gets rid of the permission error:

diff -r f962ec8e47a1 Lib/test/test_mmap.py                                                                
--- a/Lib/test/test_mmap.py     Mon Sep 10 01:23:05 2012 +0200                                            
+++ b/Lib/test/test_mmap.py     Mon Sep 10 22:22:38 2012 +0200                                            
@@ -491,11 +491,11 @@                                                                                     
     def test_empty_file (self):                                                                          
         f = open (TESTFN, 'w+b')                                                                         
         f.close()                                                                                        
-        f = open(TESTFN, "rb")                                                                           
-        self.assertRaisesRegex(ValueError,                                                               
-                               "cannot mmap an empty file",                                              
-                               mmap.mmap, f.fileno(), 0, access=mmap.ACCESS_READ)                        
-        f.close()                                                                                        
+        with open(TESTFN, "rb") as f:                                                                    
+            self.assertRaisesRegex(ValueError,                                                           
+                                   "cannot mmap an empty file",                                          
+                                   mmap.mmap, f.fileno(), 0,                                             
+                                   access=mmap.ACCESS_READ)                                              
                                                                                                          
     def test_offset (self):                                                                              
         f = open (TESTFN, 'w+b')                                                                         
                                                          




But the test still fails:



== CPython 3.3.0rc2+ (default, Sep 10 2012, 22:01:26) [MSC v.1600 64 bit (AMD64)]                         
==   Windows-7-6.1.7601-SP1 little-endian                                                                 
==   C:\Users\stefan\pydev\cpython\build\test_python_2908                                                 
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_use
r_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1)     
[1/1] test_mmap                                                                                           
test_empty_file (test.test_mmap.MmapTests) ... FAIL                                                       
                                                                                                          
======================================================================                                    
FAIL: test_empty_file (test.test_mmap.MmapTests)                                                          
----------------------------------------------------------------------                                    
ValueError: mmap offset is greater than file size                                                         
                                                                                                          
During handling of the above exception, another exception occurred:                                       
                                                                                                          
Traceback (most recent call last):                                                                        
  File "C:\Users\stefan\pydev\cpython\lib\test\test_mmap.py", line 498, in test_empty_file                
    access=mmap.ACCESS_READ)                                                                              
AssertionError: "cannot mmap an empty file" does not match "mmap offset is greater than file size"
msg170234 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-10 20:46
New changeset 25d477647a2d by Jesus Cea in branch '2.7':
#15676: mmap: add empty file check prior to offset check <- Previous patch was incomplete
http://hg.python.org/cpython/rev/25d477647a2d
msg170235 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-10 20:50
New changeset 88a88c32661e by Jesus Cea in branch '3.2':
#15676: mmap: add empty file check prior to offset check <- Previous patch was incomplete
http://hg.python.org/cpython/rev/88a88c32661e

New changeset 0ac94ae29abe by Jesus Cea in branch 'default':
#15676: mmap: add empty file check prior to offset check <- Previous patch was incomplete
http://hg.python.org/cpython/rev/0ac94ae29abe
msg170236 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-09-10 20:55
I think Py_DECREF(m_obj) is missing (at least in 3.3, I didn't
look at the other versions).
msg170237 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-10 20:58
New changeset 39efccf7a167 by Jesus Cea in branch '2.7':
#15676: mmap: add empty file check prior to offset check <- Previous patch was incomplete (fix 2)
http://hg.python.org/cpython/rev/39efccf7a167

New changeset 56a2e862561c by Jesus Cea in branch '3.2':
#15676: mmap: add empty file check prior to offset check <- Previous patch was incomplete (fix 2)
http://hg.python.org/cpython/rev/56a2e862561c

New changeset 306b2ecb1a3e by Jesus Cea in branch 'default':
MERGE: #15676: mmap: add empty file check prior to offset check <- Previous patch was incomplete (fix 2)
http://hg.python.org/cpython/rev/306b2ecb1a3e
msg170241 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-09-10 21:35
Thanks for the heads-up, Stefan.
History
Date User Action Args
2012-09-10 21:35:40jceasetstatus: open -> closed
assignee: jcea
resolution: fixed
messages: + msg170241
2012-09-10 20:58:50python-devsetmessages: + msg170237
2012-09-10 20:55:08skrahsetmessages: + msg170236
2012-09-10 20:50:44python-devsetmessages: + msg170235
2012-09-10 20:46:00python-devsetmessages: + msg170234
2012-09-10 20:33:31skrahsetmessages: + msg170232
2012-09-10 20:13:08skrahlinkissue15909 superseder
2012-09-10 20:12:10skrahsetstatus: closed -> open

nosy: + skrah
messages: + msg170228

resolution: fixed -> (no value)
2012-09-10 18:20:18python-devsetmessages: + msg170212
2012-09-10 06:56:42neologixsetmessages: + msg170162
2012-09-09 23:24:44python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg170145

resolution: fixed
stage: resolved
2012-09-09 22:15:44jceasetmessages: + msg170140
2012-09-09 20:16:44Steven.Willissetfiles: + issue15676-default.patch

messages: + msg170133
2012-09-09 15:39:37neologixsetnosy: + neologix
messages: + msg170111
2012-09-09 15:08:08Steven.Willissetversions: + Python 3.1
2012-09-09 15:06:05Steven.Willissetfiles: + issue15676-default.patch

messages: + msg170108
2012-09-09 15:00:00Steven.Willissetfiles: + issue15676-3.2.patch

messages: + msg170107
2012-09-09 14:52:15Steven.Willissetfiles: + issue15676-3.1.patch

messages: + msg170106
2012-09-07 02:29:19Steven.Willissetfiles: + issue15676.patch
keywords: + patch
messages: + msg169967
2012-08-31 07:31:29berker.peksagsetversions: - Python 3.1, Python 3.4
2012-08-15 20:08:15jceasetmessages: + msg168323
2012-08-15 20:06:08jceasetnosy: + jcea
2012-08-15 17:46:33Steven.Williscreate