Using Enum to illustrate:

    --> class Grade(enum.Enum):
    ...   A = 4
    ...   B = 3
    ...   C = 2
    ...   D = 1
    ...   F = 0
    ...   def __index__(self):
    ...     return self._value_

    --> ['miserable'][Grade.F]
    'miserable'

    --> '%x' % Grade.F
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: %x format: a number is required, not Grade

    --> hex(Grade.F)
    '0x0'

I suggest that hex() and oct() have the same check that %x and %o do so that non-numbers are not representable as hex and octal.  

While we're at it, we should do the same for bin().  Are there any others?

I'll create a patch once we have a decision on which way to solve this issue.

Calls:
* hex()/oct() => PyNumber_ToBase() => PyNumber_Index().
* PyUnicode_Format() => mainformatlong() => PyNumber_Long()

I never understood the difference between "long" (__int__ method) and "index" (__index__ method). Is the difference on the behaviour of floating point numbers?

It seems to me that by giving it an __index__ method, you're saying it can be used as an integer. It's not surprising to me that hex(), oct(), and bin() would work with a Grade.F object. If anything, I'd say that more places should use __index__ than currently do.

Victor Stinner commented:
-------------------------
> I never understood the difference between "long" (__int__ method)
> and "index" (__index__ method). Is the difference on the behaviour
> of floating point numbers?

__index__ was originally added so that non-int integers, such as NumPy's int16, int32, etc, integer types could be used as indices and slices.

Now it means "if your type can produce a lossless integer, use __index__", which is why float and similar types don't define it.

The current meaning is unfortunate in that it is possible to want a type that can be used as an index or slice but that is still not a number, and in fact won't be used as a number in any scenario _except_ bin(), hex(), and oct().

It seems to me that by having those three functions check that the argument is a number, and bailing if it is not, is a decent way to ensure consistency.

One question I do have, since I don't have NumPy installed, is what happens with:

  --> "NumPy's int's work here? %x" % uint16(7)

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42) 
>>> import numpy
>>> hex(numpy.uint16(257))
'0x101'
>>> "%x" % numpy.uint16(257)
'101'
>>> x=numpy.uint16(257)
>>> x.__int__()
257
>>> x.__index__()
257

Ethan Furman <report@bugs.python.org> wrote:
> The current meaning is unfortunate in that it is possible to want a type that
> can be used as an index or slice but that is still not a number, and in fact
> won't be used as a number in any scenario _except_ bin(), hex(), and oct().

memoryview, struct and probably also array.array accept __index__.

Did I mention __index__ is an unfortunate name for the current trend for this method?

Stefan Krah commented:
----------------------
> memoryview, struct and probably also array.array accept __index__.

When you say "accept __index__" do you mean for use as indices, or for use as values in the data structure itself?

> Did I mention __index__ is an unfortunate name for the current trend for this method?

Yes, but it's probably too late to change that now.  Also, a fully precise
name would be something like:

  __to_int_exact_iff_object_has_integer_nature__ :)

> When you say "accept __index__" do you mean for use as indices, or for use as
> values in the data structure itself?

The latter, see Lib/test/test_buffer.py:2489 .

Hmmm...

Well, much as I hate to say it, it's sounding like the correct solution here is to have %o and %x work when __index__ is available, instead of the other way around.  :(

.format is not an issue because one must specify one's own if inheriting from object.

So the complete list of spcecifiers then is d, i, o, u, U, and c [1], and they should work if __index__ works.

Are we in agreement?


[1] http://docs.python.org/dev/library/stdtypes.html#printf-style-string-formatting

> Are we in agreement?

Start maybe on writing unit tests :-)

IMO all int-like objects should behave the same. I don't see any good
reason why hex(value) would succeed whereas "%x" % value fails. Both
should succeed (or both should fail).

> > Did I mention __index__ is an unfortunate name for the current trend for
> > this method?
> Yes, but it's probably too late to change that now.  Also, a fully precise
> name would be something like:
> 
>   __to_int_exact_iff_object_has_integer_nature__ :)

Perhaps in future (may be in 4.0) __index__ should be renamed to __int__ and 
__int__ to __trunc__.

Yes, I think adding __index__ to d, i, o, u, U, and c is the correct thing to do here.

Not so fast.  Currently, even in Python 3, '%d' % 3.14 returns '3'.  "Fixing" this will likely break a huge amount of code.

Also (the tracker email interface swallowed this):

> it is possible to want a type that can be used as an index or slice but that is still not a number

I'm sorry, but this requirement is absurd. An index *is* a number. You
have to make up your mind. (I know, in the context of the example that
started this, this is funny, but I still stand by it.)

---

Finally, the correct name should perhaps have been __integer__ but I don't see enough reason to change it now.

If you were going to make this change, I'd think you'd have to look for __index__ and then __int__. But I'll admit I haven't thought through all of the ramifications. It would be interesting to see what tests would break.

Eric V. Smith commented:
------------------------
> If you were going to make this change, I'd think you'd have to look
> for __index__ and then __int__.

Does the order matter?  Are there any types (and should there be) that would have both and return different answers for each?

If not, pick an order, try one and, if that one fails, try the other.

I'm with Guido: it doesn't really make sense to allow __index__ but not __int__ on a type. So trying __index__ in str.format() sounds like a distraction.

Antoine Pitrou opined:
----------------------
> I'm with Guido: it doesn't really make sense to allow __index__ but not __int__ on
> a type. So trying __index__ in str.format() sounds like a distraction.

  --> hex(3.14)  # calls __index__
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: 'float' object cannot be interpreted as an integer

  --> '%x' % 3.14  # calls __int__
  '3'

One of those behaviours is wrong.  Which?

>   --> '%x' % 3.14  # calls __int__
>   '3'
> 
> One of those behaviours is wrong.  Which?

For '%x' and '%o', it probably doesn't make sense to convert the float
to an int.
(however, it does make sense for other formatters, such as '%d')

Ethan Furman previously stated:
-------------------------------
> So the complete list of spcecifiers then is d, i, o, u, U, and c [1], and they
> should work if __index__ works.

Okay, so 'd' then should be considered a conversion operation, whilst the others should only work if the object is actually an integer type (which is implied by specifying __index__).

In other words

  - if %d or %u is specified, try __int__, then __index__
    (according to the docs, u is obsolete and identical to d)

  - if %i, %o, %x, %X, or %c is specified, try only __index__

Agreed?

> In other words
> 
>   - if %d or %u is specified, try __int__, then __index__
>     (according to the docs, u is obsolete and identical to d)

Again, I don't think trying __index__ is useful.

>   - if %i, %o, %x, %X, or %c is specified, try only __index__

I think everything yielding a decimal output should work with floats
(i.e. %i too).

Antoine, if I understand you correctly, you are saying that any type that defines __index__ is an integer, and should therefore also define __int__, in which case Python can just use __int__ and not worry about __index__?

Here's the problem with that:

  --> '%x' % 3.14
  '3'

While I am beginning to agree that an integer type needs to implement both __int__ and __index__, it still remains true that Python needs to call __index__ if what it needs is already a real, true int, and not just something that can be truncated or otherwise converted into an int -- such as float.

> Antoine, if I understand you correctly, you are saying that any type
> that defines __index__ is an integer, and should therefore also define
> __int__, in which case Python can just use __int__ and not worry about
> __index__?

... is an integer-like, yes.

> While I am beginning to agree that an integer type needs to implement
> both __int__ and __index__, it still remains true that Python needs to
> call __index__ if what it needs is already a real, true int, and not
> just something that can be truncated or otherwise converted into an
> int -- such as float.

Of course.

Antoine,

Does that mean you are reducing your previous statement of

> So trying __index__ in str.format() sounds like a distraction.

to "using __index__ for %d, %i, and %u is not correct, but is correct for %c, %o, %x, and %X" ?

> Antoine,
> 
> Does that mean you are reducing your previous statement of
> 
> > So trying __index__ in str.format() sounds like a distraction.
> 
> to "using __index__ for %d, %i, and %u is not correct, but is correct
> for %c, %o, %x, and %X" ?

Ah, yes, sorry for the confusion :)

In addition, PyLong_AsLong() calls __int__, while PyLong_AsUnsignedLong() doesn't call __int__.

Thank you, Victor and Serhiy, for your pointers into the code.

I'm hoping we have general agreement about %c, %o, %x, and %X and having them use __index__ only (using __int__ would open the door to float conversions).

I still have a question about %i, though.  The docs say %u is exactly the same as %d and is therefore deprecated.  The docs do not say thay %i is the same as %d, but the descriptions are the same.

Are %i and %d the same, or is there some difference?

AFAIK %i and %d are the same.

Yes, looking through Objects/unicodeobject.c, 'u', 'i', and 'd' are treated the same everywhere.

It seems to me that anything that is an 'integer' that can be turned into an int without loss of information (has .__index__) is logically a 'number' that can be turned into an int possibly with loss of information (has .__int__).  So perhaps one of the following should be true:

1. The doc for .__index__ specifies that def __index__ 'must' be followed by __int__ = __index__ to make a coherent class. (So Ethan's Grade as written above would not qualify.)

2. The type constructor does this for us by adding __int__ as an alias for __index__ when the latter is present.

3. Every core usage of __int__ looks for __index__ also. Int() does not do this now, but '%d' does, so int(Grade.F) fails but int('%d' % Grade.f) works.

The exact details would depend on whether we want to allow (or at least bless) classes with __int__ and __index__ returning different ints.

The docs for bin/oct/hex(x) are clear. "Convert an integer number to a binary/octal/hexadecimal string. The result is a valid Python expression. If x is not a Python int object, it has to define an __index__() method that returns an integer." This should not change.

If the domain of %x is going to be a subset of of the domain of %d, it seems to me that the exclusion should be of non-integers (such as floats) rather than of non-int integers. Given things as they are, I would simply expand the domain of %x, etc, to that of %d without bothering to go through a deprecation process.

Thank you everyone for increasing my understanding.  :)

Terry J Reedy wrote:
--------------------
> [snip everything I now agree with, which is most of Terry's comment]

> 3. Every core usage of __int__ looks for __index__ also. Int() does not
> do this now, but '%d' does [...]
> 
> The exact details would depend on whether we want to allow (or at least
> bless) classes with __int__ and __index__ returning different ints.

I think we should state that __int__ and __index__ are expected to return the same value (if both are defined) otherwise one is on one's own (otherwise known as: if it breaks, you own all the pieces ;) .

> Given things as they are, I would simply expand the domain of %x, etc,
> to that of %d without bothering to go through a deprecation process.

This is not correct.  `hex(3.14)` raises a TypeError, and so should `'%x' % 3.14`

While Terry's option 2 would have to wait for 3.5, is there any reason why fixing the %-formating characters to use __index__ instead of __int__ can't go into 3.4?  That portion seems to be clearly a bug and not an enhancement (the enhancement portions of this ticket can be separated out into another issue).

Enhancement portion split off into issue20092.

Attached patch includes doc change, tests, and code changes.

If I'm overstepping my bounds, I'm sure somebody will slap me with a fish. ;)

Ran full test suite; some errors came up in test_format from the following test lines:

    testformat("%#x", 1.0, "0x1")
    testformat("%x", float(big), "123456_______________", 6)
    testformat("%o", float(big), "123456__________________________", 6)
    testformat("%x", float(0x42), "42")
    testformat("%o", float(0o42), "42")

Removed as float() is not supposed to be valid input.

Also fixed two memory leaks in unicodeobject from my changes, and a float->oct bug in tarfile.

Better doc enhancement thanks to R. David Murray (thanks!).

While I think this is an improvement, I don't think we can make this change in behavior at this stage in 3.4. Won't it have to go in 3.5?

I can see why we wouldn't want to make this change in a point release, but this is a feature release we're talking about and this seems like buggy behavior:

--> hex(3.14)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'float' object cannot be interpreted as an integer

--> '%x' % 3.14
'3'

I agree the behavior is bad and should be fixed. But it's a new feature that will break existing code, and 3.4 is in beta, and therefore feature freeze. Unfortunately, I think this has to go into 3.5, unless you can get Larry to grant you an exception.

We don't usually *break* existing working code even if the output is wrong.  If you want to make '%x' % 3.14, etc, raise an error, I think you we should go through a deprecation period first.  The deprecation messages should, of course, be added now.

In other words, I would probably have voted against commit even if we weren't in beta yet.  It's a bit borderline, since the output is *wrong*, but given that we *are* in beta I think we'd better go the deprecation route.

I can live with the deprecation route.  I'll create a patch today or tomorrow for that.

I wouldn't call this a new feature--it's definitely a bug fix.  So the "feature freeze" rule does not automatically apply.  I definitely wouldn't permit this once we reach release candidates, but we aren't there yet.

I get the impression that it will break code, and it does seem kind of late.  So I'm not saying yes or no yet.  Let me think about it some more.

I'm willing to risk it in 3.4.  Can you check it in in the next twelve hours?

(Sorry for the short notice, it slipped my mind until just now.)

New changeset 2f81f0e331f6 by Ethan Furman in branch 'default':
Issue19995: %o, %x, %X now only accept ints
http://hg.python.org/cpython/rev/2f81f0e331f6

I was travelling yesterday and wasn't sure about the time stamp.  Looks like I missed the 12-hour cutoff.  Let me know if I should backout the commit.

Ethan, I thought we were going the deprecation route?
This *will* break code.

(also, there are typos: "shuld", "psuedo")

Antoine: I made the call to bite the bullet and fix it.  If that's a terrible idea we can change it before RC1.  But from my (admittedly dim) understanding of the issue, we're going to have to fix this sooner or later, and sooner is probably better.

If by permitting this I've made a terrible mistake, could you do me a favor and summarize the issue and the alternatives so I can understand it?

And, yes, Ethan's checkin missed the cutoff for beta 2.

I think the code-breakage issue is that although this is a bug now, it did not use to be a bug.  (Or maybe it was always a bug, but nobody noticed -- I don't know when hex() and oct() were introduced.)

Basically, having %o and %x work with floats was the intended behavior, as there were tests to make sure it worked.

But hex() and oct() were designed to only work with integer types, resulting in two ways to get an octal or hexadecimal string, one of which worked with non-ints, and one which didn't.

tarfile.py, for example, was using %o to convert a stat member (a timestamp?) from int/float to octal.

It is agreed that working with non-ints is no longer the intended behavior, but there is undoubtedly code that uses it -- especially in cases where the return value could be either an int or a float.

Given that even the stdlib used it, there is no question in my mind that a deprecation period is required.  We don't just arbitrarily break peoples code on purpose, we give them at least a release in which to fix it first.

Keep in mind that with new-style division, it's easy to get a float even if all your inputs are ints.

I'll switch it to a deprecation warning instead sometime this week.

The changes to datamodel.rst and tarfile.py should stay, I think.

test_format and test_unicode will both verify current* behavior and check for the deprecation warning, and unicodeobject.c will generate the warning but keep current behavior.

Once 3.5 is tagged I'll put the updates to these three files there.


* By 'current' I mean accepting non-ints.

Merely from reading the docs, including the hex() and {}.format sections, I consider the current
  '%x' % 1.5 == '1'
a bug in that I would expect either a Type or Value Error
or a hex fraction(!) ('1,8') matching the core of the output of float.hex(1.5). But given the history, I agree that a deprecation notice is needed first.

In 3.3 section 6.1.3.1. Format Specification Mini-Language, integer and float conversions are in separate tables, so no confusion is possible. In 3.3 section 4.7.2. printf-style String Formatting there is just one table with all conversion chars. I think the table would be clearer if it were somehow split into three sections: perhaps 3 otherwise blank rows containing  'integer', 'float', and 'other' in the Conversion column. (I don't know if this works for .rst tables.)

I think I would change the table for 3.4 with a note 8 that 'o', 'x', and 'X' currently accept floats by calling int() on the value, but that this implicit behavior is deprecated, so one should be explicit. The DeprecationWarning should also give the same remedy.

Could somebody explain this?
===============================================================================
ethan@media:~/source/python/issue19995_depr$ ./python -W error
Python 3.4.0b1 (default:2f81f0e331f6+, Jan  9 2014, 20:30:18) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> '%x' % 3.14
oxX
no __index__
depracation is fatal
oxX
no __index__
depracation is fatal
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
DeprecationWarning: automatic int conversions have been deprecated
>>> 
===============================================================================
ethan@media:~/source/python/issue19995_depr$ ./python -W message
Invalid -W option ignored: invalid action: 'message'
Python 3.4.0b1 (default:2f81f0e331f6+, Jan  9 2014, 20:30:18) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> '%x' % 3.14
oxX
no __index__
depracation not fatal, attempting __int__ conversion
conversion with __int__ successful
'3'
>>> 
===============================================================================
ethan@media:~/source/python/issue19995_depr$ ./python -W once
Python 3.4.0b1 (default:2f81f0e331f6+, Jan  9 2014, 20:30:18) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> '%x' % 3.14
oxX
no __index__
sys:1: DeprecationWarning: automatic int conversions have been deprecated
depracation not fatal, attempting __int__ conversion
conversion with __int__ successful
'3'
>>> 
===============================================================================

As you can see, with the -W error option it seems to go through the routine twice; does that mean the the '1' in 'line 1' is being specified as a float?

For anyone paying really close attention, I've already switched the assertEquals to assertEqual.  ;)

New changeset 22e55bd5583c by Ethan Furman in branch 'default':
Issue19995: issue deprecation warning for non-integer values to %c, %o, %x, %X
http://hg.python.org/cpython/rev/22e55bd5583c

Typo in the C code: depracation :-)

New changeset cc8b21988efb by Ethan Furman in branch 'default':
Issue19995: fixed typo; switched from test.support.check_warnings to assertWarns
http://hg.python.org/cpython/rev/cc8b21988efb

Depracation warning is in place for 3.4.

When 3.5 is tagged I'll switch it an error.

New changeset 775fb736b4b8 by Serhiy Storchaka in branch 'default':
Catch deprecation warnings emitted when non-integers are formatted with %c, %o
http://hg.python.org/cpython/rev/775fb736b4b8

New changeset 9120196b3114 by Ethan Furman in branch 'default':
Issue19995: passing a non-int to %o, %c, %x, or %X now raises an exception
http://hg.python.org/cpython/rev/9120196b3114

It looks like there's typo: Psuedo => Pseudo.  (Unless the first one
is an alternate spelling I'm not aware of.)

No, apparently I am incapable of spelling pseudo correctly.  I'll fix that along with the better error message:

  %x format: an integer is required, not float

(variable, obviously ;)

New changeset e266525c9294 by Ethan Furman in branch 'default':
Issue19995: more informative error message; spelling corrections; use operator.mod instead of __mod__
http://hg.python.org/cpython/rev/e266525c9294

Final status:

  3.4 -> DeprecationWarning

  3.5 -> TypeError

Found my first 3.5 breakage which I think is due to this.

>>> from uuid import uuid4
>>> '%.32x' % uuid4()

Fix:

>>> '%.32x' % uuid4().int