classification
Title: test_mmap failing on AIX
Type: behavior Stage: resolved
Components: Extension Modules, Tests Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, ajaksu2, loewis, mdr0, nnorwitz, pitrou, r.david.murray, sable, tim.peters, wheelrl
Priority: normal Keywords: patch

Created on 2003-01-31 17:44 by nnorwitz, last changed 2010-12-11 02:13 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
patch_flush_mmap.diff sable, 2010-09-21 16:36
patch_mmap_flush_updated.diff sable, 2010-09-22 09:04
Messages (21)
msg14327 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2003-01-31 17:44
test_mmap is failing on a flush while trying to do:
  Copy-on-write memory map data not written correctly

The problem is that the mmap is opened with 
ACCESS_COPY.  This translates to MAP_PRIVATE.  
On AIX, the msync man page says:  "When the 
MS_SYNC and MAP_PRIVATE flags both are used, the 
msync subroutine returns an errno value of EINVAL."

I'm not sure what the correct fix should be.
msg14328 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-03-04 07:00
Logged In: YES 
user_id=21627

I think the test is somewhat bogus: It tries to check that
modification to an ACCESS_COPY doesn't modify the underlying
file, but assumes that .flush becomes a no-op, even though
an exception is more reasonable (IMO; errors should never
pass silently).

So I see two options: Declare that .flush() always raises an
exception (and modify implementations that don't produce an
exception accordingly), or declare that aspect to be
system-dependent, and modify the test (and the
documentation) to expect and ignore an exception.

Assigning to Tim, as he incorporated that feature into mmap.
msg14329 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2003-04-28 19:55
Logged In: YES 
user_id=31435

Sorry, I've had nothing to do with mmap beyond fixing bugs.  
The "access" feature was due to Jay Miller, although I believe 
I checked in his patch.

Martin, I don't understand why you think it's reasonable for 
flush to complain here:  the mmap is open for writing, so 
what's surprising about expecting to be able to flush after a 
write?  Simply that there's no associated file, due to copy-on-
write?  Then user code would have to be acutely aware of how 
an mmap'ed object was opened, just to avoid nuisance 
complaints when they flush after writing.

So that's a third alternative:  alter the implementation to make 
mmap.flush() do nothing when an mmap object was opened 
as copy-on-write.
msg14330 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-05-03 10:09
Logged In: YES 
user_id=21627

The documentation for flush says

"Flushes changes made to the in-memory copy of a file back
to disk."

But it doesn't do that, and we all agree it shouldn't do
that. So I would claim that it is an error to use .flush on
an mmap object that was opened in ACCESS_COPY. 

This is like trying to write to a file that was opened for
reading only: one *could* declare that the write just does
nothing, but it helps the developer more if you get an
exception, because the code is likely wrong (i.e. not
following the likely intentions of the author).
msg14331 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2003-05-06 20:00
Logged In: YES 
user_id=31435

Hmm.  I suspect the flush docs() are too strong (does flush 
really promise to materialize bytes *on disk*?  it doesn't for 
other Python file objects, you also need os.fsync() for that).

Your point is well taken, though, and whatever flush() does 
normally do, it's not going to do it for a copy-on-write mmap.  
So fine by me if we declare that attempting to flush() a copy-
on-write mmap raises an exception.
msg14332 - (view) Author: Richard Wheeler (wheelrl) Date: 2004-01-07 21:57
Logged In: YES 
user_id=249546

I am getting the same error on AIX 5.2 with Python 2.3.3 
release.  What can I do to get around the test error and 
verify the mmap is working?
msg14333 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2004-01-07 22:06
Logged In: YES 
user_id=33168

run the test with the -v flag:  ./python
./Lib/test/regrtest.py -v test_mmap

I think only everything should be fine up to copy-on-write
tests.  Here's what a good run looks like:

[neal@epoch c3]$ ./python ./Lib/test/regrtest.py -v test_mmap
test_mmap
<type 'mmap.mmap'>
  Position of foo: 1.0 pages
  Length of file: 2.0 pages
  Contents of byte 0: '\x00'
  Contents of first 3 bytes: '\x00\x00\x00'
                                                           
                    
  Modifying file's content...
  Contents of byte 0: '3'
  Contents of first 3 bytes: '3\x00\x00'
  Contents of second page: '\x00foobar\x00'
  Regex match on mmap (page start, length of match): 1.0 6
  Seek to zeroth byte
  Seek to 42nd byte
  Seek to last byte
  Try to seek to negative position...
  Try to seek beyond end of mmap...
  Try to seek to negative position...
  Attempting resize()
  Creating 10 byte test data file.
  Opening mmap with access=ACCESS_READ
  Ensuring that readonly mmap can't be slice assigned.
  Ensuring that readonly mmap can't be item assigned.
  Ensuring that readonly mmap can't be write() to.
  Ensuring that readonly mmap can't be write_byte() to.
  Ensuring that readonly mmap can't be resized.
  Opening mmap with size too big
  Opening mmap with access=ACCESS_WRITE
  Modifying write-through memory map.
  Opening mmap with access=ACCESS_COPY
  Modifying copy-on-write memory map.
  Ensuring copy-on-write maps cannot be resized.
  Ensuring invalid access parameter raises exception.
 Test passed
1 test OK.
msg14334 - (view) Author: Mark D. Roth (mdr0) Date: 2004-03-10 17:14
Logged In: YES 
user_id=994239

I'm running into this problem under both AIX 4.3.3 and 5.1.
 Is this something that's going to affect python if I put it
into production, or is it "safe" to ignore it?
msg14335 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2004-03-11 18:59
Logged In: YES 
user_id=33168

Mark, the 3 bugs you commented on (this one, 713169, and
678264) are all test errors AFAIK.  It should be safe to
ignore them.  They are in extension modules that are not
used by many people.
msg81716 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-02-12 03:28
Is this concern still valid?
msg114215 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-18 12:49
Closed as no response to msg81716.
msg117054 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-21 14:47
I would like to reopen this issue as it is still occurring in py3k on AIX 5.3 and 6.1:

Re-running test test_mmap in verbose mode
test_access_parameter (test.test_mmap.MmapTests) ... ERROR
test_anonymous (test.test_mmap.MmapTests) ... ok
test_bad_file_desc (test.test_mmap.MmapTests) ... ok
test_basic (test.test_mmap.MmapTests) ... ok
test_context_manager (test.test_mmap.MmapTests) ... ok
test_context_manager_exception (test.test_mmap.MmapTests) ... ok
test_double_close (test.test_mmap.MmapTests) ... ok
test_entire_file (test.test_mmap.MmapTests) ... ok
test_error (test.test_mmap.MmapTests) ... ok
test_extended_getslice (test.test_mmap.MmapTests) ... ok
test_extended_set_del_slice (test.test_mmap.MmapTests) ... ok
test_find_end (test.test_mmap.MmapTests) ... ok
test_io_methods (test.test_mmap.MmapTests) ... ok
test_move (test.test_mmap.MmapTests) ... ok
test_offset (test.test_mmap.MmapTests) ... ok
test_prot_readonly (test.test_mmap.MmapTests) ... ok
test_rfind (test.test_mmap.MmapTests) ... ok
test_subclass (test.test_mmap.MmapTests) ... ok
test_tougher_find (test.test_mmap.MmapTests) ... ok

======================================================================
ERROR: test_access_parameter (test.test_mmap.MmapTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/san_u02/home/recette/buildbot/buildbot-aix5/py3k-aix5-xlc/build/Lib/test/test_mmap.py", line 219, in test_access_parameter
    m.flush()
mmap.error: [Errno 22] Invalid argument

----------------------------------------------------------------------
Ran 19 tests in 0.216s

FAILED (errors=1)


Should flush be modified to do nothing in this case or should the unit test be updated?
thanks
msg117070 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-09-21 15:28
> Should flush be modified to do nothing in this case or should the unit test be updated?

Tim is suggesting that flush should indeed become a noop. Since nobody
else speaking in favor of it being an error, I guess this is the way to
go: flush, on an ACCESS_COPY file, does nothing, and the test is fine
as it stands.
msg117073 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-21 15:43
Interestingly, the matter was discussed on another issue, #2643. I also agree that ideally flush() should become a no-op (only in 3.2, since it would break compatibility). But then we should also expose a separate sync() method with the current behaviour.
msg117075 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-09-21 15:53
> Interestingly, the matter was discussed on another issue, #2643. I
> also agree that ideally flush() should become a no-op (only in 3.2,
> since it would break compatibility). But then we should also expose a
> separate sync() method with the current behaviour.

I think you misunderstand. I'm not proposing that flush should become
a noop entirely - only for ACCESS_COPY mappings.
msg117081 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-21 16:36
Would that patch be OK? It solves the test_mmap on AIX.
msg117084 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-09-21 17:16
Looks fine to me.
msg117129 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-22 09:04
After Antoine commit concerning issue2643, here is a new patch (just removing the changes in close).

Could you commit it?
msg118993 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-18 01:15
Committed to py3k in r85678.  If I'm reading this string correctly, I believe this can (and should be) be backported.  Am I correct?
msg119002 - (view) Author: Sébastien Sablé (sable) Date: 2010-10-18 10:25
I also believe this patch should be backported.

If the issue2643 is not backported, then the patch to apply should be "patch_flush_mmap.diff" instead of "patch_mmap_flush_updated.diff"
msg123764 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-12-11 02:13
Committed the patch_flush_mmap patch to 3.1 in r87163 and 2.7 in r87164.
History
Date User Action Args
2010-12-11 02:13:28r.david.murraysetstatus: open -> closed

messages: + msg123764
2010-10-18 12:19:55georg.brandlsettitle: test_mmap failling on AIX -> test_mmap failing on AIX
2010-10-18 10:25:31sablesetmessages: + msg119002
2010-10-18 01:15:28r.david.murraysetnosy: + r.david.murray
messages: + msg118993

resolution: fixed
stage: resolved
2010-09-23 14:24:47eric.araujosetfiles: - unnamed
2010-09-23 14:24:44eric.araujosetfiles: - unnamed
2010-09-23 14:24:42eric.araujosetfiles: - unnamed
2010-09-23 14:24:39eric.araujosetfiles: - unnamed
2010-09-23 14:24:34eric.araujosetfiles: - unnamed
2010-09-22 09:04:38sablesetfiles: + patch_mmap_flush_updated.diff

messages: + msg117129
2010-09-21 22:21:28loewissetmessages: - msg117110
2010-09-21 22:21:23loewissetmessages: - msg117106
2010-09-21 22:21:13loewissetmessages: - msg117107
2010-09-21 22:21:05loewissetmessages: - msg117108
2010-09-21 22:20:55loewissetmessages: - msg117109
2010-09-21 21:58:36BreamoreBoysetfiles: + unnamed

messages: + msg117110
2010-09-21 21:58:24BreamoreBoysetfiles: + unnamed

messages: + msg117109
2010-09-21 21:58:13BreamoreBoysetfiles: + unnamed

messages: + msg117108
2010-09-21 21:58:02BreamoreBoysetfiles: + unnamed

messages: + msg117107
2010-09-21 21:57:52BreamoreBoysetfiles: + unnamed

messages: + msg117106
2010-09-21 17:16:39loewissetmessages: + msg117084
2010-09-21 16:36:46sablesetfiles: + patch_flush_mmap.diff
keywords: + patch
messages: + msg117081
2010-09-21 15:53:26loewissetmessages: + msg117075
2010-09-21 15:43:18pitrousetnosy: + pitrou
messages: + msg117073
2010-09-21 15:28:05loewissetmessages: + msg117070
2010-09-21 14:53:03r.david.murraysetstatus: closed -> open
resolution: out of date -> (no value)
2010-09-21 14:47:34sablesettype: crash -> behavior
messages: + msg117054
versions: + Python 2.7, Python 3.2
2010-09-14 13:07:19sablesetnosy: + sable
2010-08-18 12:49:04BreamoreBoysetstatus: open -> closed

nosy: + BreamoreBoy
messages: + msg114215

resolution: out of date
2009-02-12 03:28:25ajaksu2setnosy: + ajaksu2
type: crash
messages: + msg81716
components: + Tests
2003-01-31 17:44:42nnorwitzcreate