An interactive prompt should offer working completion out-of-the-box,
without requiring every Python user on earth to create a $PYTHONSTARTUP
with '''import readline; readline.parse_and_bind("tab: complete")'''.

Note that it should work not only when Python is run without arguments,
but also when running with the -i option.
And ideally, it should only install the completion function when
dropping to the interpreter, not from the start, so that code that runs
before dropping to the interpreter is not affected.  But this is really
not important.

Safety: if 'rlcompleter' or 'readline' are not availiable (e.g. broken
python install), the user must still get a working interpreter, just
without completion.

Portability: there won't be completion on windows (as long as pyreadline
or similar is not part of Python).  But that shouldn't delay unix support!

What would be the right place for this to happen? I first thought of
site.py, but -S would then turn off readline support. Does it have to be
done somewhere in C land?

But perhaps tab isn’t the right key to bind. I think inputrc could set
it to something different, perhaps shell rc files too. Is there an API
to get this setting, does readline.so already do the right thing, or
does it have to be added?

Moreover, in my PYTHONSTARTUP I not only bind tab to complete, but also
set a history file. Should this be automatic too? UNIX rules would
suggest a standard ~/.pyhistory file, or we could follow newer
freedesktop.org specifications [1].

So, I don’t know the history and choices behind readline.so, but from a
user point of view, being fed up with lack of tab completion finally had
me put up a PYTHONSTARTUP file with these settings, and then some useful
functions, and so on. This lack of completion was ultimately a good
thing for me. Still, completion could be made to work out of the box,
and the man page could link to the doc explaining how to customize this
behavior.

[1] XDG Base Directory Specification:
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

Cheers, Merwok

He, adding completion is also something I find myself adding on every
box on which I use Python...

for what it is worth, I am +1 on having completion and history file work by default.  The sqlite3 command line does this, for example.  I think it is what unix user expect nowadays, and I think it is reasonable.

Looking at my home directory, it would appear that the de-facto standard history file name would be .python_history.  This is based on .mysql_history, .psql_history, and .sqlite_history, none of which I configured explicitly to support history saving.

Okay, that’s one question answered.  Still to solve: How to bind the right key? (“But perhaps tab isn’t the right key to bind. I think inputrc could set it to something different, perhaps shell rc files too. Is there an API to get this setting, does readline.so already do the right thing, or does it have to be added?”)

I think TAB is the key expected by most people, so let's make it the default.
As for the location, site.py is an adequate one IMO.

Keep in mind that the Python readline module may be linked to either GNU readline or the BSD editline (libedit) library and they have different command strings.  Note the warning here:

http://docs.python.org/dev/py3k/library/readline.html

Here's a snippet of what I have today in my startup file:

    import rlcompleter
    if 'libedit' in readline.__doc__:
        readline.parse_and_bind("bind ^I rl_complete")
    else:
        readline.parse_and_bind("tab: complete")

See Issue10666 for more details and some possible changes.

Nosying Martin: any Windows installer concerns?

There is no readline support on Windows at all, so I don't think the Windows installer can be affected. I'm uncertain what the proposed change to Python is at this point, though.

Ned, does readline.read_init_file() with libedit?

readline.read_init_file() does work with libedit.  The directives read have to be in libedit format.

Please ignore the generated patches, left here for Martin’s benefit, and review the manually attached fix-5845.diff.

I have used read_init_file instead of hard-coding tab (it’s anyway the default for readline).

I may have repeated the same info in too much doc files, but I preferred to have more than one place where people could learn about the new behavior.  Advice welcomed.

If the basic approach looks good, especially if site still feels the right place for this, I will add tests.

Updated patch (not using Mercurial, looks like I haven’t enough bandwidth to push all those changesets).

> Updated patch (not using Mercurial, looks like I haven’t enough
> bandwidth to push all those changesets).

I think there's a bug in the way you are detecting interactive mode:

$ ./python -c "import sys; print(sys.stdin.isatty())"
True

Interestingly, there's already the following code in Modules/main.c:

    if ((Py_InspectFlag || (command == NULL && filename == NULL && module == NULL)) &&
        isatty(fileno(stdin))) {
        PyObject *v;
        v = PyImport_ImportModule("readline");
        if (v == NULL)
            PyErr_Clear();
        else
            Py_DECREF(v);
    }


...meaning readline already gets imported automatically when desired. And indeed:

$ ./python -S
Python 3.3.0a0 (default:50f1922bc1d5, Aug 18 2011, 00:09:47) 
[GCC 4.5.2] on linux2
>>> import sys
>>> sys.modules['readline']
<module 'readline' from '/home/antoine/cpython/default/build/lib.linux-x86_64-3.3-pydebug/readline.cpython-33dm.so'>


So perhaps we could simply change this code to import another, private module (e.g. "_setupinteractive.py") which would setup readline and rlcompleter?

We can’t just decide to enable completion by checking “'readline' in sys.modules” in site, because it always returns False.  site is imported in Python/pythonrun.c, and I guess that Modules/main.c is run afterwards.  More importantly, I think the import of readline in Modules/main.c is a CPython implementation detail, and I’d prefer to have code in site that is directly useful for other VMs.  (I’d go as far as proposing to remove the import readline from main.c, when my patch makes it obsolete.)

I re-read the docs for sys.argv and the -m switch and decided to translate to C checks (“command == NULL && filename == NULL && module == NULL”) to “if not sys.argv[0]”.  Unfortunately, sys.argv is not available when site is imported, so this is a dead end :(

So, I could try your _setupinteractive module idea, even though I think it’s a bit unclean, or drink the cool aid and set up completion in main.c.

Easily detecting interactive mode is of general interest for customization.

1. What if C also set sys.flags.interactive in "python" mode,
   or exposed sys.flags.implicit_interactive (but with better name)?

2. It's more useful to have a hook called when entering interactive mode,
   rather than a flag that's set from the beginning:

   $ python -i -c 'import sys; print sys.flags.interactive'
   1
   >>> 

   For this, importing _setupinteractive is a step forward;
   calling e.g. sys.__interactivehook__ sounds even better.
   (site.py would set it by default to a function that enables
   rlcompleter, user can always override...)

BTW, drawback of doing any such setup in site.py: "python -S" would 
be unfriendly!

> Easily detecting interactive mode is of general interest for
> customization.
Thanks for the feedback.  We could open a feature request for that, and/or ask python-ideas.

> What if C also set sys.flags.interactive in "python" mode, or exposed
> sys.flags.implicit_interactive (but with better name)?
An attribute in the sys module sounds like a good idea.  It would not be an attribute of sys.flags though, as those are strictly command-line arguments.

> It's more useful to have a hook called when entering interactive mode, rather than a flag
> that's set from the beginning:
We already have such a hook: $PYTHONSTARTUP

> calling e.g. sys.__interactivehook__ sounds even better.
That’s another interesting idea.

> BTW, drawback of doing any such setup in site.py: "python -S" would be unfriendly!
People using -S don’t want the customizations done in site, so I don’t think there’s a problem here.

> > It's more useful to have a hook called when entering interactive mode, rather than a flag
> > that's set from the beginning:
> We already have such a hook: $PYTHONSTARTUP

$PYTHONSTARTUP doesn't work with -i

[sorry, html mail was bad idea]

On Tue, Sep 6, 2011 at 17:54, Antoine Pitrou <report@bugs.python.org> wrote:

Éric Araujo <merwok@netwok.org> added the comment:
> > It's more useful to have a hook called when entering interactive mode,
> > rather than a flag
> > that's set from the beginning:
> We already have such a hook: $PYTHONSTARTUP

Good point!
It covers the user's desire customization very well (esp. if it worked with -i).
sys.__interactivehook__ has the benefit of being cleanly settable from python code.
But it might well be a YAGNI idea.

> $PYTHONSTARTUP doesn't work with -i
 
Perhaps it should?
I can't think of a thing that makes sense in $PYTHONSTARTUP that I wouldn't want with -i.
(and if there is one, one can add a test for sys.flags.interactive, or run with env PYTHONSTARTUP='')
 
Point to watch out for: errors in $PYTHONSTARTUP.
One of the uses of "python -i script.py" is doing pdb.pm() on an exception thrown by the script;
ideally a broken $PYTHONSTARTUP would not overr

> > BTW, drawback of doing any such setup in site.py: "python -S" 
> > would be unfriendly!
> People using -S don’t want the customizations done in site, so I 
> don’t think there’s a problem here.

python -S doesn't disable readline.  What makes completions more of a "customization" than editing?
The fact that it'd be implemented in site.py?
Yes, obviously, if it's implemented in site.py, -S should disable it.  My point was that it doesn't have to be implemented there.  You could drink the cool aid instead :-)

> sys.__interactivehook__ has the benefit of being cleanly settable from python code.
> But it might well be a YAGNI idea.
I’ll ask python-dev about that.  For the moment, I prefer the idea of a new sys.interactive attribute (boolean).

>> $PYTHONSTARTUP doesn't work with -i
> Perhaps it should?
-i means two things: interactive and inspect.  The second meaning is used when you want python to execute a script or module just as usual but without exiting afterward, to let you inspect the state of the program.  Supporting PYTHONSTARTUP would change these use cases and be an unacceptable change.

 
>> BTW, drawback of doing any such setup in site.py: "python -S" 
>> would be unfriendly!
> People using -S don’t want the customizations done in site, so I 
> don’t think there’s a problem here.

> python -S doesn't disable readline.
> What makes completions more of a "customization" than editing?
> The fact that it'd be implemented in site.py?
If you go back to the first messages of this report, you’ll see that I wasn’t sure what a good location would be, and decided site because it did not look out of place.  There isn’t a stronger rationale than that.  One could argue that even now, site mixes disparate functionality: one program could wish to use -S to disable sys.path munging but run into bugs because the builtin exit is not defined anymore for example.

> Yes, obviously, if it's implemented in site.py, -S should disable it.
We agree.

> My point was that it doesn't have to be implemented there.
I really want to explore all options before renouncing to implement it in site.  I’d like the implementation to be maintainable and shareable with other Python VMs.

FTR, I tried checking sys.ps1 instead of argv but it’s the same problem.

In the spirit of pushing this forward, here is an updated patch using the sys.__interactivehook__ approach. I didn't add any tests since it doesn't seem very easy to write any.

If nobody objects, I would like to commit this soon.

New changeset d5ef330bac50 by Antoine Pitrou in branch 'default':
Issue #5845: Enable tab-completion in the interactive interpreter by default, thanks to a new sys.__interactivehook__.
http://hg.python.org/cpython/rev/d5ef330bac50

End-users will hopefully rejoice :)

On OS X 10.6, I now get the following message at interpreter startup:


iwasawa:cpython mdickinson$ ./python.exe
Python 3.4.0a0 (default:d5ef330bac50, May  6 2013, 13:05:57) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Failed calling sys.__interactivehook__
Traceback (most recent call last):
  File "/Users/mdickinson/Python/cpython/Lib/site.py", line 481, in register_readline
    readline.read_init_file()
OSError: [Errno -1] Unknown error: -1

More information: readline.read_init_file() produced the same traceback before this commit, so all that's changed is that we're now calling this at interpreter startup.  It looks like I'm using the system libedit:

iwasawa:cpython mdickinson$ otool -L build/lib.macosx-10.6-x86_64-3.4-pydebug/readline.so 
build/lib.macosx-10.6-x86_64-3.4-pydebug/readline.so:
	/usr/lib/libedit.2.dylib (compatibility version 2.0.0, current version 2.11.0)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)
iwasawa:cpython mdickinson$

So I'm failing to find any documentation for libedit, but it looks as though this error occurs if rl_read_init_file fails to find an .editrc file in the appropriate place.  If I create an empty .editrc file in my home directory, the error disappears.  (Having an .inputrc file doesn't seem to make a difference either way.)

Perhaps the solution is to ignore an OSError from readline.read_init_file in register_readline---i.e., add a try / except OSError there.  The other option would be to ignore a nonzero errno from rl_read_init_file, but that doesn't sound so good: I'd expect someone deliberately calling readline.read_init_file to get an exception if the initialization file isn't found.

> So I'm failing to find any documentation for libedit, but it looks as
> though this error occurs if rl_read_init_file fails to find an
> .editrc file in the appropriate place.  If I create an empty .editrc
> file in my home directory, the error disappears.  (Having an
> .inputrc file doesn't seem to make a difference either way.)
> 
> Perhaps the solution is to ignore an OSError from
> readline.read_init_file in register_readline---i.e., add a try /
> except OSError there.

This sounds fine to me. Can you propose a patch? I'm unlikely to ever have
an OS X machine.

The attached fixes the issue for me.  I'm not sure whether the try / except should only be done on Apple, though.  What's the behaviour on Linux if there's no .inputrc file?

> The attached fixes the issue for me.  I'm not sure whether the try /
> except should only be done on Apple, though.  What's the behaviour
> on Linux if there's no .inputrc file?

Everything works fine under Linux (as usual :-)).
There's no need to restrict the try / except to Apple platforms, though.

I'm not able to test the patch at the moment, but since it essentially just uses the recipe in the docs, I expect it will have the same side-effect. Namely, it prevents you using the tab key to indent in the interactive interpreter.

Now I don't know if I'm missing something painfully obvious, but having to bang out space-space-space-space for every indent is surely not going to be a win for usability ;-) Even if I am missing something, surely so will a lot of other users.

I use a readline completer that (in my opinion) does the Right Thing: at the start of the line, hitting TAB inserts a tab, otherwise it does completion. It also sets a history file, adds a couple of key bindings that I sometimes find useful, and wraps it all up in a class. (Attached.) If you think there is any value in this, I'm happy to update it to Python 3.3 and polish it up as a patch to the readline module. Or someone can just mine it for ideas.

> Now I don't know if I'm missing something painfully obvious, but
> having to bang out space-space-space-space for every indent is
> surely not going to be a win for usability ;-) Even if I am missing
> something, surely so will a lot of other users.

What *looks* painfully obvious to me is that Python doesn't force you
to use 4 spaces (or tabs) for indents :-)

> I use a readline completer that (in my opinion) does the Right Thing:
> at the start of the line, hitting TAB inserts a tab, otherwise it
> does completion. It also sets a history file, adds a couple of key
> bindings that I sometimes find useful, and wraps it all up in a
> class.

The "indent at beginning of line" thing sounds useful. The rest seems
to conflate the concept of a completer with the readline module itself.
Even though it's called rlcompleter, the rlcompleter module is
actually generic and could be used with other input schemes, so I don't
think it's the right place to set history options or key bindings.

In any case, you should post your patch as part of a separate issue, IMO.

New changeset 82e92da929eb by Mark Dickinson in branch 'default':
Issue #5845: avoid an exception at startup on OS X if no .editrc file exists.
http://hg.python.org/cpython/rev/82e92da929eb

Applied the OS X fix;  reclosing.

The tab-doesn't-indent still needs to be fixed before 3.4 is released.

> The tab-doesn't-indent still needs to be fixed before 3.4 is
> released.

I don't really understand why this would be a release blocker?
The interpreter prompt is a convenience, we don't guarantee
compatibility like with stdlib APIs.

It is a release blocker because it is a major usability regression.

Just to be clear how important I consider this, I would advocate for backing out this patch rather than releasing 3.4 with a broken tab key at the interactive prompt.  But I'd rather have the patch *and* a working tab key.

> It is a release blocker because it is a major usability regression.

Really? You can press the space key to indent, it works as well
as the tab key...

Well, post it to python-dev and see what reaction you get :)

I could be wrong, but I don't think I am (obviously).

I expect that a lot of users use the tab key to indent in the repl (as well as in editors, smart enough editors can convert the tab presses to spaces)

> Well, post it to python-dev and see what reaction you get :)

I'm not interested in python-dev reactions here. For this
kind of issue, we'll have hundreds of messages for and against
the change without any useful content.

The commit adds an often-requested feature (to the point that many
people were already taking the pain of activating it manually).
But, yes, we could add Steven's refinement so that all bases are
covered.

> I expect that a lot of users use the tab key to indent in the repl
> (as well as in editors, smart enough editors can convert the tab
> presses to spaces)

The interpreter prompt is not a text editor at all. You can misuse
it as one, but that's a loss of time and energy IMO.

Really, everyone I've ever showed tab-completion to has always
been enthusiastic about it ("why isn't this enabled by default?").
None has ever raised the concern of being able to use the tab key
to indent code under the prompt.

(probably because noone is crazy enough to type long chunks of
code under the interactive prompt anyway)

I am in AP's camp on the tab issue, but I think we can preserve "tab inserts tab" behavior at the continuation prompt.  I don't like "indent at beginning of line."  I have rlcompleter enabled in Python 2.6 and i get the following when I press tab:

Python 2.6.6 (r266:84292, Feb 22 2013, 00:00:18)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> <tab>
Display all 173 possibilities? (y or n) y
ArithmeticError(            abs(                        input(
AssertionError(             all(                        int(
AttributeError(             and                         intern(
BaseException(              any(                        is
..

I find this rather useful.

At the ... prompt, however, a tab (or better four spaces) is arguably the right completion at the beginning of the line.

Take it with a large grain of Internet salt, but Ezio pointed me to the following reactions:
http://www.reddit.com/r/Python/comments/1dq7sh/python_repl_finally_gets_tab_completion_by_default/

Also made it to the front-page of Hacker News[1], with better quality comments than the reddit thread.

[1] https://news.ycombinator.com/item?id=5658062