I use Python in my CS1 and CS2 curriculum and I have a question.
As I've been using the Python 3.0 alphas one of the things that I am
bothered by is that I cannot see the sequence produced by range
without introducing students to the list() function.

I typically introduce range on day 1 of class and show students what
it produces without making a big deal out of the fact that it creates
a list.  They all accept this and things work out nicely when I
introduce lists for real in a week or two.

My question is why couldn't the __str__ method for the range object be
more friendly and show a representation of the sequence?  I understand
why __repr__ should return range(0,10) for an object created using
range(10) but couldn't print(range(10)) produce [0, 1, 2, ... 9]
The ... could even be used if the sequence were excessively long.

I have attached a patch, which follows the suggestion from Guido on how 
to format the string so it is not confused with a list.

This is my first attempt at patching any part of the C code for Python.  
Please let me know what should be changed and If I've missed something.

In particular I wonder whether I should be nesting any calls to PyNumber 
functions or whether temporary variables should be used to avoid leaking 
memory?

In addition I get the following warning on the line where I install the 
range_str function in the PyRange_Type array.

Objects/rangeobject.c:357: warning: initialization from incompatible 
pointer type


Brad

Thanks for the patch! You can avoid the compiler warning by casting to
reprfunc like this:

(reprfunc)range_str /* tp_str */

Also, some of the whitespace needs reformating. Please put a space
between the parameters passed a function, and remove the ones in the
conditionals.

You should use PyObject_RichCompareBool.

All the calls to functions like PyLong_FromLong and PyNumber_* can
possibly return NULL on error, so you must check for that. Sometimes
it's useful to have a goto for failures at the end of your function in
which you Py_XDECREF everything.

Could you take the space from the "<" and ">" at the end of the
representation?

> I use Python in my CS1 and CS2 curriculum and I have a question.

At this point, I would have almost stopped reading - the tracker
is not a place for questions. I read on and then noticed that you
have a patch also, not just a question :-)

> This is my first attempt at patching any part of the C code for Python.  
> Please let me know what should be changed and If I've missed something.

The formatting of the C code is wrong in on place (there must not be a
space after the opening or before the closing parenthesis); I would also
drop the space after the < and before the > in the result.

Please try to avoid replicating code as much as possible. Rather than
this complex muliply/add code, use range_item(r, 1), range_item(r, 2),
range_item(r, -2), range_item(r, -1). Use range_length to find out how
many things are in the range, and then again range_item for the 0..7th
element.

Don't use PyUnicode_FromFormat for a single %R; use PyObject_Repr
instead.

The code leaks many object references. Almost any function you call
produces a new reference, and you have to decref it after you don't
need the value anymore. E.g. write

  repr = PyObject_Repr(nth);
  appended = PyUnicode_Concat(result, repr);
  Py_DECREF(repr); // not needed anymore, so release it
  Py_DECREF(result); // about to assign to result, so release old value
  result = appended;

Consider exceptions. About any function returning PyObject* can fail
with an exception, if for not other reason than being out-of-memory.
Don't forget to release partial results in case an exception, making
your code like

  repr = PyObject_Repr(nth);
  if (!repr)
    goto fail;
  appended = PyUnicode_Concat(result, repr);
  if (!appended)
    goto fail;
  Py_DECREF(repr);
  Py_DECREF(result);
  result = appended;
...
fail:
  Py_XDECREF(repr);
  Py_XDECREF(appended);
  ...
  return NULL;

As we don't know in the failure case easily whether appended was already
assigned to, we use XDRECREF; don't forget to initialize appended with
NULL.

As both the success case and the failure case end with a long list
of DECREFs, it is common to phrase that as

  PyObject *result = NULL;
  ...
  result = what_shall_i_really_return;
  what_shall_i_really_return = NULL;
fail:
  lots_of_xdecrefs;
  return result;

In your case, it might be useful to create an array of up to 7
PyObject*, zero-initialize that, and put the partial results in
there; then XDECREF that in a loop at the end.

HTH,
Martin

Thanks for all of the help.  Using range_item certainly simplified my 
job.  Although I more than made up for it with all of the decref and 
error checking calls.

One side effect of using range_item is that str(range(...)) is now 
subject to the same errors as indexing into really big range:
for example: 
x = range(1000000000000000000)
x[1]
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C ssize_t

In a world where ints and long ints have been unified the existence of 
this error seems like a bug.

I think the right thing would be to fix range_item and range_length so 
they do not use Py_ssize_t for the length of the range.  But I may be in 
over my head on that guess.

I also discovered that range_item does not work with negative indices.

I have been poking around the tests and added some simple tests to the 
test_range.py class as well.  I assume I should submit a patch for that 
as well?

I have attached a new patch for rangeobject.c

Brad

> In a world where ints and long ints have been unified the existence of 
> this error seems like a bug.
> 
> I think the right thing would be to fix range_item and range_length so 
> they do not use Py_ssize_t for the length of the range.  But I may be in 
> over my head on that guess.

The question is: what C type to use instead? Whatever native type you
use, it will always have a limitation. Assuming that range() elements
are typically used as indices into collections, having them Py_ssize_t
is fine because no standard collection (string, tuple, list) can have
more than 2**(8*sizeof(Py_ssize_t)) elements, on any computer, as
that would exceed the address space of that computer.

In any case, I think it is fine that str() has this limitation when
you can't actually access the values of the range object, anyway.
If the limit where to be removed, somebody would have to rewrite

> I have been poking around the tests and added some simple tests to the 
> test_range.py class as well.  I assume I should submit a patch for that 
> as well?

Indeed. The code looks fine to me now, please to provide a patch also.

Here is a new patch file.  This one contains the modifications to 
rangeobject.c as well as test_range.py

I think this is everything.  If there is something else I need to do 
please let me know.  I looked to see if there was any documentation I 
could or should change for this but did not find any.

Thanks again for your help.

-1

I don't think <0, 1, ..., 9> is much clearer than range(0, 10).  The 
only problem students may have is that 10 not in range(0, 10), but this 
can be learned very quickly.  The <..> repr breaks x == eval(repr(x)) 
invariant which is actually important in some of my code. Furthermore, 
the proposed <..> representation suggests that the range object contains 
the integers, which is not true.  While eval/repr round-tripping may not 
be important for the majority of applications, it is a nice feature for 
any interactive work including education.  Finally, I think introducing 
students to list(range(..))/tuple(range(..)) idioms early is a good 
thing. The sooner they learn that list/tuple are real containers while 
ranges, dict key/values etc. are not the better.

The patch does not change the behavior of repr.  It modifies the 
behavior of str.

I agree that learning list/tuple sooner is better, but students who have 
never written a line of code before can only absorb so much information, 
this little patch allows them to print(range(100,0,-1)) and get a much 
better intuition about what is happening.

I did not realize that the proposed patch only affects str and not repr.  
Some of may previous arguments against it do not hold in this case, but 
I am still -1.

If you introduce range before list, it will be hard to explain why lists 
are printed in valid python syntax while ranges are not.

What do you do with ranges in your first classes?  Maybe you can 
introduce it as a part of the looping construct and make students think 
of for i in range(10) as an equivalent of C's for(i = 0; i < 10; i++)?

Also

>>> [i for i in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

can be demonstrated early without explaining the mechanics of list 
comprehensions.  I would expect most students being able to guess what 
for i in range(10): does and half of the students being able to guess 
what [i for i in range(10)] means.

It looks like our messages have crossed in flight.

> this little patch allows them to print(range(100,0,-1))
> and get a much better intuition about what is happening.

This is a really bad example.  Why would you want to introduce the print 
function in the first class?  You can explain much of the language using 
python as an advanced calculator without introducing any IO.  
Furthermore, given that print changes from being a statement to being a 
function in 3.0, it will be a much less novice friendly construct for 
years to come.  Also range(100, 0, -1) is not the most shining example 
of python's clarity. :-)

I think the best idea for educators would be to limit introductory 
material to 2.x - 3.x portable constructs.  This means avoid using 
range(..) outside of looping constructs, don't use print before you have  
to (and then use a subset of 3.x syntax.)

BTW, you really only need to use print function when you start writing 
scripts and that is better avoided in the first class when you have 
interactive environment as you do with python.

As a side remark, you may find the following trick useful in the first 
class:

>>> range(10)
range(0, 10)
>>> list(_)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Our use of range in the first few classes is exactly for iteration 
purposes, but I do not like students to have to have too many mysteries.  
So I always have liked to show that range(10) simply produces a sequence 
of integers.  In Python 3.0 range returns a mysterious iteration object.  
No thanks.  My proposal was to provide a more user friendly 
implementation of the str method for this new range object that would 
allow the user to see the sequence.  I like Python because it is so easy 
to start up a shell and poke around and see what things are.

I have no problem, introducing list(range(10)) in week 3 when I start 
talking about lists, and I like list comprehensions of that purpose too.

Again, what I do not like is that things that used to be very easy for 
students to get a conceptual handle on are now more difficult in 3.0.
- range is one example the dict_keys and dict_values objects are another 
example.  dict_keys et. al. are much easier to deal with since I've 
already covered lists and the list() function by the time I get there.

BTW, I think  we must have very different teaching styles as I would 
never introduce something as mysterious as list(_) on the first day of 
class.  I'd be happy to continue our discussion of teaching philosophy 
but I doubt that this is the right forum.

Brad

On Fri, Apr 11, 2008 at 12:06 PM, Brad Miller <report@bugs.python.org> wrote:

>  Our use of range in the first few classes is exactly for iteration
>  purposes, but I do not like students to have to have too many mysteries.
>  So I always have liked to show that range(10) simply produces a sequence

.. but it does not in 3.x and your patch does not change that.
Instead, range(10) produces range(0,10) and I don't see anything
mysterious here.

>  of integers.  In Python 3.0 range returns a mysterious iteration object.
>  No thanks.  My proposal was to provide a more user friendly
>  implementation of the str method for this new range object that would
>  allow the user to see the sequence.  I like Python because it is so easy
>  to start up a shell and poke around and see what things are.
>

Maybe you should try it with some of your students first.  Note that
you can customize the way results are printed on the console with
sys.displayhook.   I fail to see how <0, 1, ... 9> is less mysterious
than range(0,10), particularly combined with the fact that print
representation is different from console display.  I think <0, 1, ...
9> will confuse more students than it will help.

>  I have no problem, introducing list(range(10)) in week 3 when I start
>  talking about lists, and I like list comprehensions of that purpose too.
>

Why would you want to talk about loops before talking about lists?
Python is not C.  In many cases for x in some_list or for i,x in
enumerate(some_list) is a better construct than for i in range(..).

>  Again, what I do not like is that things that used to be very easy for
>  students to get a conceptual handle on are now more difficult in 3.0.

With your proposed <..> display students will be getting a wrong
concept, thinking that <1, 2, 3> is yet another container in addition
to (1,2,3), [1,2,3] and {1,2,3}.

>  - range is one example the dict_keys and dict_values objects are another
>  example.  dict_keys et. al. are much easier to deal with since I've
>  already covered lists and the list() function by the time I get there.
>

IMHO list function is simpler than the print function, particularly in
cases when it does not print what you enter.

>  BTW, I think  we must have very different teaching styles as I would
>  never introduce something as mysterious as list(_) on the first day of
>  class.  I'd be happy to continue our discussion of teaching philosophy
>  but I doubt that this is the right forum.
>

Your students are probably older than mine.  I don't teach, but I
introduced my son to python when he was 10.  He is 12 now and is quite
comfortable using python as a calculator (computing 2**1000 in python
what quite an experience!)  Explaining that _ stands for the value
that just printed was easier than introducing variables (as in
x=range(10); list(x)).

Clearly neither Alexander nor I are going to get the other to come 
around to our point of view.  Thats fine, I think we can disagree here, 
and I can adapt and change my class either way.

My question is how does this get resolved.  When I posted this idea to 
python-dev Guido suggested an approach. Nobody else expressed an opinion 
so after waiting a few days I took Guido's suggestion and implemented a 
patch.  Now there are objections.

Other than some mild frustration at having invested a fair amount of 
time in producing my first python patch, I am also in the middle of 
editing a textbook that will come out this fall.  my intention is to 
have this book be 3.0 compatible.  So it would be nice for me to have 
some clarity on the direction this will go.

Thanks,

Brad

On Fri, Apr 11, 2008 at 2:45 PM, Brad Miller <report@bugs.python.org> wrote:
..
>  My question is how does this get resolved.  When I posted this idea to
>  python-dev Guido suggested an approach. Nobody else expressed an opinion
>  so after waiting a few days I took Guido's suggestion and implemented a
>  patch.  Now there are objections.
>

My -1 vote is not a veto by any means.  It us ultimately up to Guido
to accept or reject your patch.  I am actually willing to change my
vote to -0.  I cannot recall ever using either str(..) or print on a
range object, so this change will not affect me while I still believe
it will not solve your problem either.

>  Other than some mild frustration at having invested a fair amount of
>  time in producing my first python patch

This happens a lot.  Oftentimes proposals are not even considered
without a working patch.  I hope writing the patch and receiving
Martin's comments on your code was a valuable experience in itself
even if your patch does not get accepted.

> I am also in the middle of
>  editing a textbook that will come out this fall.  my intention is to
>  have this book be 3.0 compatible.  So it would be nice for me to have
>  some clarity on the direction this will go.

I would suggest considering a custom displayhook approach. You can
write a custom displayhook that will print range(..), {}.keys(),
{}.values() etc in a student-friendly way.   I believe a module
installing such display hook can be included in the standard library.
In addition to iterable's display problem such a hook can limit the
amount of output when displaying long lists, insert smart line breaks
etc.  For a textbook, you can instruct the readers to download your
code and import a custom module before trying the examples.  This is
likely to make your textbook more future-proof because you will be
able to update your displayhook code as python changes.

> I would suggest considering a custom displayhook approach. You can
> write a custom displayhook that will print range(..), {}.keys(),
> {}.values() etc in a student-friendly way.   I believe a module
> installing such display hook can be included in the standard library.
> In addition to iterable's display problem such a hook can limit the
> amount of output when displaying long lists, insert smart line breaks
> etc.  For a textbook, you can instruct the readers to download your
> code and import a custom module before trying the examples.  This is
> likely to make your textbook more future-proof because you will be
> able to update your displayhook code as python changes.

That is an interesting idea!  I just hacked together a quick prototype.  
I'll discuss this with my co-author.  Its a fine line between using 
'real'
Python and providing an appropriate level of help for the beginner.  We
have tried to minimize the number of additional modules and 
dependencies, so
if there was a hook like this as part of the standard library that would 
be
great.

Brad

> I don't think <0, 1, ..., 9> is much clearer than range(0, 10).  The 
> only problem students may have is that 10 not in range(0, 10), but this 
> can be learned very quickly.  The <..> repr breaks x == eval(repr(x)) 
> invariant which is actually important in some of my code.

I think you misunderstand the patch. It doesn't affect repr() at all.

> Now there are objections.
> 
> Other than some mild frustration at having invested a fair amount of 
> time in producing my first python patch, I am also in the middle of 
> editing a textbook that will come out this fall.

Don't be frustrated. This is a regular thing to happen; sometimes,
opposition won't form until the code is committed and released in
an alpha release.

> So it would be nice for me to have 
> some clarity on the direction this will go.

We should wait a few more days for people to speak up.

Myself, I'm -0 on the patch. The '<' syntax is awkward.

FWIW, I would like to see a newsgroup or python-dev discussion for a 
more general solution to the problem for helpful repr's for iterators.  
In 3.0, lots of things return iterators, not just range().  

Before applying one ad-hoc patch, it would be helpful to think through 
how the issue should be handled across the whole language.  For 
example, perhaps pprint() should display general iterators with the 
<...> notation.  

For teaching and experimentation purposes, maybe the right answer is 
the make the command-line a bit smarter.  It already knows how to 
suppress None return values.  Perhaps, it should be taught how to 
display range objects and dict.items() and such.

Based on the discussion on python-dev  I've worked up a patch and tests 
for repr of dict_items, dict_keys, and dict_values.  I've also modified 
the patch and test for str of the range object.  (If there was a way to 
get the str(range(10)) to automatically display in the interactive shell 
that would be ideal.

I assume one combined patch for all of this is OK since its been part of 
the same discussion.

Brad

Some review of dv_range.patch:

- repr(d.keys()) depends on the internal ordering of items, their hash
values, the insertion order... the test seems fragile.
Or you may rely on the fact that ints are their own hash values, so a
small dict containing small keys will appear ordered. I suggest
something like {6: 1.2, 1: 'A', 7: 42, 4: None}

- empty dicts or empty ranges will display:
   <range: >
   <dict_items: >
Isn't there a better repr? At least this should appear in the tests.

On Apr 17, 2008, at 4:26 PM, Amaury Forgeot d'Arc wrote:

>
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> Some review of dv_range.patch:
>
> - repr(d.keys()) depends on the internal ordering of items, their hash
> values, the insertion order... the test seems fragile.
> Or you may rely on the fact that ints are their own hash values, so a
> small dict containing small keys will appear ordered. I suggest
> something like {6: 1.2, 1: 'A', 7: 42, 4: None}

I wondered about that, but my assumption was that the hash function  
for strings and ints was equally unlikely to change.  I can go with  
all ints if that is really a safer assumption.

>
>
> - empty dicts or empty ranges will display:
>   <range: >
>   <dict_items: >
> Isn't there a better repr? At least this should appear in the tests.

I'm not sure how to represent that better.
<range: None>
<dict_items: None>       -- These seem misleading

<range: empty>
<dict_items: empty>       -- I'm not sure that is better than  
<dict_items: > and empty may not mean anything to non-english speakers

I'll be happy to add the test for the empty case after once I know  
what it should look like.

>
>
> ----------
> nosy: +amaury.forgeotdarc
>
> __________________________________
> Tracker <report@bugs.python.org>
> <http://bugs.python.org/issue2610>
> __________________________________

Is there any chance this patch will make it into the latest alpha??

Brad

> Is there any chance this patch will make it into the latest alpha??

I don't think there is any such chance.

On May 7, 2008, at 1:55 PM, Martin v. Löwis wrote:

Thanks for the very clear answer.

This being my first attempt at contributing to the Python core, I am  
looking for some sort of clarity on the future of this patch.  It  
feels like some sort of consensus was reached and now the patch has  
been lingering without additional comment for several weeks.   I have  
some sense of urgency in knowing the future as the final opportunity  
to include these changes in my introduction to CS book is looming on  
the horizon.

Thanks,

Brad

> __________________________________
> Tracker <report@bugs.python.org>
> <http://bugs.python.org/issue2610>
> __________________________________

After reviewing this again, I'm skeptical that this is a good idea. It
doesn't achieve its original purpose (anymore), as it only changes
tp_str for range objects (although it does change tp_repr for dict views
- why this inconsistency?).

So I'm -0 on the patch, meaning that I won't commit it, but won't object
to anybody else committing it, either.

Technically, in dictview_repr, the first call to PyUnicode_Concat isn't
checked for exceptions.

I'm also -1 on the patch and think it is the wrong solution.  

Efforts should probably be directed at hooking the output of the 
interactive interpreter.  We already have code that suppresses None 
return values and assigns results to "_".  That same code can also add 
custom interactive reporting for a few, known iterators.

On Tue, Jun 17, 2008 at 12:23 AM, Martin v. Löwis
<report@bugs.python.org> wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
> After reviewing this again, I'm skeptical that this is a good idea. It
> doesn't achieve its original purpose (anymore), as it only changes
> tp_str for range objects (although it does change tp_repr for dict views
> - why this inconsistency?).
>
The reason for the inconsistency is that there was a strong argument
that the tp_repr for range already returned something useful that
people could take advantage of in their code.  The same was not the
case for the dict views.

I do not understand why you think that having the interpreter display
<dict_keys: 2, 3, 4, ...>
when x.keys() is called is not an improvement over
<dict_keys object at 0xe72b0>

Maybe it is just because I spend a lot more time in the interactive
interpreter that I see this as a big improvement.

So if, as Raymond Hettinger suggests, the interpreter is the right
place to make this change I'd still be happy to provide a patch if
someone could give me a pointer for where to start looking.

Brad

> So I'm -0 on the patch, meaning that I won't commit it, but won't object
> to anybody else committing it, either.
>
> Technically, in dictview_repr, the first call to PyUnicode_Concat isn't
> checked for exceptions.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue2610>
> _______________________________________
>

> I do not understand why you think that having the interpreter display
> <dict_keys: 2, 3, 4, ...>
> when x.keys() is called is not an improvement over
> <dict_keys object at 0xe72b0>

I didn't say that the change in dict_keys is not an improvement,
but that the patch doesn't achieve its objective anymore, which
was to display range() nicely at the interactive prompt.

> Maybe it is just because I spend a lot more time in the interactive
> interpreter that I see this as a big improvement.

So do I. But I typically display the dictionary itself when looking
at it in an interactive prompt; I would not normally look at .keys(),
except when I want to know what all the keys in the dictionary are,
in which case the new repr won't help, either (and I will have to
apply list() in 3.0, or, rather, sorted())

I would like to see something along the general lines of bmiller's patch
for dict views in a 3.x release... there are probably other iterators
that could do with chattier reprs also. (Range, on the other hand, is
fine as it is.)

It's not just at the interactive prompt that good reprs are useful, it's
also in debugging: debug-prints in scripts in general, web application
error pages that dump variable information, error detail logging to a
database for inspection long after the ability to inspect the value has
passed, and so on.

I think it's better to put the feature in repr than require all these
things - everything that might want to display values helpfully - to
implement detection and prettification for specific iterators.

Sure, you can apply list() if you know in advance you're going to need
to, but for beginners and debuggers getting the information out there
without having to ask for it is a real win. I certainly remember how
pleasantly surprised I was when learning Python 1.something and finding
it just told me what I wanted to know without having to ask for it - in
contrast to previous tedious languages that only ever said 'object' or
'[Array]'...

I can't really think of any advantage to having repr keep that
information hidden.

It's going to have to be deferred to 3.1.

Just to restart the discussion on the original issue since I see that 
the latest 3.1 has solved the problem with dict_keys, dict_values, etc 
al objects.  Many thanks!

A suggestion was made by Alexander to create a custom displayhook that 
could be included in the standard library.  From the research I've done 
it looks like a solution for range would be something like the 
following:

import sys
def eduhook(o):
    if o is None:
        return
    if isinstance(o,range):
        print(list(o))
    else:
        print(repr(o))
        
sys.displayhook = eduhook

Now if 5233/5234 were approved I could tell my students to setup an 
environment variable PYTHONSTARTUP that points to a file that imports a 
module containing the above code.  (or I could just tell them to import 
said module each time.)

The above hook appears to work fine.  Is there anything obvious I'm 
missing?  If this is along the right track then I could extend it to 
support other custom display ideas that my fellow educators have in 
mind.

Thanks,

Brad

Closing since this is now implemented.