Title: open fails randomly on AIX
Type: Stage: resolved
Components: Versions: Python 2.7
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: kadler, wiggin15
Priority: normal Keywords: patch

Created on 2015-08-18 11:24 by wiggin15, last changed 2020-11-14 02:02 by methane. This issue is now closed.

File name Uploaded Description Edit
aix_fopen_fix.diff wiggin15, 2015-08-18 11:24 review
Messages (2)
msg248769 - (view) Author: Arnon Yaari (wiggin15) * Date: 2015-08-18 11:24
We are using Python 2.7.8 on AIX 7.1 TL 3.
On rare occasions, calls to open files with mode "w" fail with: IOError: File exists.

      File "/root/jenkins/workspace/powertools/eggs/infi.credentials_store-0.1-py2.7.egg/infi/credentials_store/", line 175, in set_credentials
        with open(self.get_file_path(), 'w') as f:
    IOError: [Errno 17] File exists: '/root/.infinidat/infinihost'

This happens randomly on multiple systems and different files (including e.g. "/dev/null").
This is very strange because "File exists" should only be raised if we open the file with the 'x' flag (os.O_EXCL) and we don't.
After debugging, we discovered that this happens due to a bug in AIX. On AIX, fopen checks for mode[2] (where 'mode' is the second argument passed to it - in our case, "w") regardless of its length. fopen checks this byte against 'x' to decide if O_EXCL should be used. In the case of passing "w" as 'mode', mode[2] will contain undefined data -- and in rare cases it will contain the byte 'x'.
This was reported to IBM and APARs exist for this issue: (see related APARs for various AIX versions in this page)

Python can and should work around this because the code in fileobject.c is also partly to blame:
Python 2.7.x contains the following lines:

    /* probably need to replace 'U' by 'rb' */
    newmode = PyMem_MALLOC(strlen(mode) + 3);

For the 'mode' parameter, Python allocates 3 additional bytes that are not always used, and these bytes contain uninitialized data. The byte 'x' that causes the problem comes from this allocation. Python should set the bytes to zero to avoid this issue.
I'm attaching a patch. Note that this applies only to the 2.x branch.
msg380950 - (view) Author: Kevin (kadler) * Date: 2020-11-14 01:04
Given that the AIX bug has long been fixed and Python 2.7 is EOL we can probably close this bug.
Date User Action Args
2020-11-14 02:02:09methanesetstatus: open -> closed
resolution: out of date
stage: resolved
2020-11-14 01:04:44kadlersetnosy: + kadler
messages: + msg380950
2015-08-18 11:24:52wiggin15create