classification
Title: os.popen causes illegal seek on AIX in Python 3.1rc
Type: Stage:
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: amaury.forgeotdarc, drukker, nestor, pitrou, srid
Priority: critical Keywords: patch

Created on 2009-06-08 02:20 by nestor, last changed 2009-09-22 13:13 by nestor. This issue is now closed.

Files
File name Uploaded Description Edit
seek-aix.patch pitrou, 2009-06-20 18:24
fileio-seekable.patch drukker, 2009-09-21 18:43
Messages (14)
msg89065 - (view) Author: nestor (nestor) Date: 2009-06-08 02:20
Python 2.6.2 (r262:71600, Jun  4 2009, 16:07:26) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.popen('cat','w')

<open file 'cat', mode 'w' at 0x1101ab4f8>


Python 3.0.1 (r301:69556, Jun  4 2009, 16:07:22) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.popen('cat','w')

<os._wrap_close object at 0x1103e5748> 

Python 3.1rc1 (r31rc1:73054, Jun  1 2009, 10:49:24) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.popen('cat','w')

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Python-3.1rc1/Lib/os.py", line 641, in popen
    return _wrap_close(io.TextIOWrapper(proc.stdin), proc)
IOError: [Errno 29] Illegal seek 

This in turn causes help not to work:

Python 3.1rc1 (r31rc1:73054, Jun  1 2009, 10:49:24) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> help(open)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Python-3.1rc1/Lib/site.py", line 429, in __call__
    return pydoc.help(*args, **kwds)
  File "/Python-3.1rc1/Lib/pydoc.py", line 1709, in __call__
    self.help(request)
  File "/Python-3.1rc1/Lib/pydoc.py", line 1756, in help
    else: doc(request, 'Help on %s:')
  File "/Python-3.1rc1/Lib/pydoc.py", line 1505, in doc
    pager(render_doc(thing, title, forceload))
  File "/Python-3.1rc1/Lib/pydoc.py", line 1320, in pager
    pager(text)
  File "/Python-3.1rc1/Lib/pydoc.py", line 1340, in <lambda>
    return lambda text: pipepager(text, 'less')
  File "/Python-3.1rc1/Lib/pydoc.py", line 1359, in pipepager
    pipe = os.popen(cmd, 'w')
  File "/Python-3.1rc1/Lib/os.py", line 641, in popen
    return _wrap_close(io.TextIOWrapper(proc.stdin), proc)
IOError: [Errno 29] Illegal seek
msg89106 - (view) Author: nestor (nestor) Date: 2009-06-08 20:09
This quick and dirty fix in pydoc.py makes so it no longer aborts help.

(less behaves somewhat strange for some commands but that is better than
no help at all)

def pipepager(text, cmd):
    """Page through text by feeding it to another program."""
    import subprocess
    pipe=subprocess.Popen(cmd,stdin=subprocess.PIPE).stdin
    #pipe = os.popen(cmd, 'w')
    try:
        pipe.write(bytes(text,sys.getdefaultencoding()))
        #pipe.write(text)
        pipe.close()
    except IOError:
        pass # Ignore broken pipes caused by quitting the pager program.
msg89546 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-06-20 17:45
Ouch, this is quite annoying. I will try to fix this before the final
release.
msg89547 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-06-20 18:24
Here is my current interpretation:

subprocess uses os.pipe() to create the file handles used for
communication. These handles normally always raise an error ([Errno 29]
Illegal seek) when trying to seek() on them, which the IO lib interprets
as meaning the stream is not seekable, which it then handles fine.

However, if the first seek() succeeds, the IO lib thinks the stream is
seekable and treats any subsequent seek() failure as an error which it
reports to the user. It may be what you are witnessing.

Can you try applying the following patch?
msg89548 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-06-20 18:31
Just before, could you try to type the following commands:

>>> r, w = os.pipe()
>>> os.lseek(r, 0, 1)
msg89687 - (view) Author: nestor (nestor) Date: 2009-06-24 23:06
That fails consistently:
Python 2.6.2 (r262:71600, Jun  4 2009, 16:07:26) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> r,w=os.pipe()
>>> os.lseek(r,0,1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek
>>>
Python 3.0.1 (r301:69556, Jun  4 2009, 16:07:22) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> r,w=os.pipe()
>>> os.lseek(r,0,1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek
>>>
Python 3.1rc2 (r31rc2:73411, Jun 15 2009, 10:56:49) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> r,w=os.pipe()
>>> os.lseek(r,0,1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek
>>>
msg89726 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-06-26 11:13
Ah, then I don't know what happens...
msg90762 - (view) Author: nestor (nestor) Date: 2009-07-21 15:40
Maybe this has something to do with it?

Python 3.1 (r31:73572, Jul  9 2009, 16:28:28) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> open("/dev/tty","a").seekable()
True
>>>
msg90873 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-07-24 08:20
Perhaps, although I don't know where /dev/tty would be involved in
os.popen().

For reference, under Linux I get:
>>> open("/dev/tty","a")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 29] Illegal seek
msg90882 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-07-24 13:17
/dev/tty is probably similar to stdout, at least when python is started
without redirection.

What is the output of
>>> sys.stdout.seekable()
when run from an interactive interpreter?
Does it depend on the active console? (xterm, ssh...)
msg91007 - (view) Author: nestor (nestor) Date: 2009-07-28 13:13
Python 3.1 (r31:73572, Jul  9 2009, 16:28:28) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import sys
>>> sys.stdout.seekable()
True
>>>

I can access the machine only through ssh.
msg92951 - (view) Author: Derk Drukker (drukker) Date: 2009-09-21 18:43
The problem is that the fileio struct in Modules/_io/fileio.c defines
the 2-bit seekable field as int.

From the C99 standard, ยง6.7.2: for bit-fields, it is
implementation-defined whether the specifier int designates the same
type as signed int or the same type as unsigned int.

Contrary to gcc, both xlc and suncc default to unsigned.

Adding 'signed' solves the problem (and also issue #6348).

Patch attached.
msg92964 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-09-21 21:43
Thanks for the patch! Committed in r75007, r75008, r75009, r75010.
If there's any problem, please reopen.
msg92990 - (view) Author: nestor (nestor) Date: 2009-09-22 13:13
Fantastic. Applied the patch and it solved the problem with xlc 8.0 on
AIX 5.3.
History
Date User Action Args
2009-09-22 13:13:46nestorsetmessages: + msg92990
2009-09-21 21:43:38pitrousetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg92964

versions: + Python 2.6, Python 2.7, Python 3.2
2009-09-21 21:18:24pitrousetresolution: accepted
2009-09-21 21:15:28sridsetnosy: + srid
2009-09-21 21:10:58pitroulinkissue6348 superseder
2009-09-21 18:43:33drukkersetfiles: + fileio-seekable.patch
nosy: + drukker
messages: + msg92951

2009-07-28 13:13:12nestorsetmessages: + msg91007
2009-07-24 13:17:30amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg90882
2009-07-24 08:20:08pitrousetmessages: + msg90873
2009-07-21 15:40:09nestorsetmessages: + msg90762
2009-06-26 11:13:04pitrousetmessages: + msg89726
2009-06-24 23:06:37nestorsetmessages: + msg89687
2009-06-20 18:31:32pitrousetmessages: + msg89548
2009-06-20 18:24:51pitrousetfiles: + seek-aix.patch
keywords: + patch
messages: + msg89547
2009-06-20 17:45:03pitrousetpriority: critical

nosy: + pitrou
messages: + msg89546

assignee: pitrou
2009-06-08 20:09:32nestorsetmessages: + msg89106
2009-06-08 02:20:02nestorcreate