classification
Title: datetime: timedelta(minutes = i) silently fails with numpy.int32 input
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder: Py_TPFLAGS_LONG_SUBCLASS is not documented
View: 9307
Assigned To: teoliphant Nosy List: LambertDW, amaury.forgeotdarc, barronh, belopolsky, grubert, mark.dickinson, serhiy.storchaka, teoliphant, terry.reedy, zaytsev
Priority: normal Keywords:

Created on 2009-03-11 15:50 by barronh, last changed 2013-07-16 16:06 by zaytsev. This issue is now closed.

Messages (17)
msg83470 - (view) Author: Barron Henderson (barronh) Date: 2009-03-11 15:50
Initializing a timedelta object with numpy.int32 arguments give mixed
results; it fails for days keyword, gives bad results for minutes, and
give correct results for seconds/microseconds.  Failure confirmed on
Linux i686 (Py 2.5.2; numpy 1.2.1) and OS X 10.5.6 (Py 2.5.1;
1.2.1).Test case below:

from datetime import timedelta
from numpy import int32
from numpy import int32
from datetime import timedelta

assert timedelta(seconds = 36) == timedelta(seconds = int32(36))
print 'pass 36 sec'
assert timedelta(microseconds = 36) == timedelta(microseconds = int32(36))
print 'pass 36 usec'
assert timedelta(minutes = 35) == timedelta(minutes = int32(35))
print 'pass 35 min'
assert timedelta(minutes = 36) == timedelta(minutes = int32(36))
print 'pass 36 min' # returns bad value
assert timedelta(days = 36) == timedelta(days = int32(36))
print 'pass 36 days' # fails SystemError: Objects/longobject.c:223
msg83471 - (view) Author: David W. Lambert (LambertDW) Date: 2009-03-11 16:06
With older versions of each timedelta rejects the data type.  Maybe
that's a good resolution?

$ /usr/local/bin/python2.4 
Python 2.4.2 (#2, Jul  7 2006, 10:20:47) 
[GCC 3.4.5 20051201 (Red Hat 3.4.5-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'0.9.8'
>>> from numpy import int32
>>> from datetime import timedelta
>>> assert timedelta(seconds = 36) == timedelta(seconds = int32(36))
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: unsupported type for timedelta seconds component: int32scalar
>>>
msg88022 - (view) Author: engelbert gruber (grubert) * Date: 2009-05-18 09:25
testing with 2.6 and numpy 1.3 already ::

   timedelta(seconds = int32(36)) 
   TypeError: unsupported type for timedelta seconds component: 
numpy.int32

But not if I apply a change to numpy to fix issue2263
then ``timedelta(days = int32(36))`` and ``timedelta(minutes = 
int32(36))`` result in wrong timedeltas.
msg109352 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-05 20:15
This is a bit strange.  timedelta is accepting int subclass instances:


>>> class I(int):
...    pass
... 
>>> timedelta(days = I(36))
datetime.timedelta(36)


but not numpy.unt_:


>>> timedelta(days = int_(36))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported type for timedelta days component: numpy.int64

which is also a subclass of int:

>>> isinstance(int_(36), int)
True

This is probably a numpy quirk rather than python issue, but is probably worth another look.
msg110788 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-19 17:55
As I suspected, datetime not recognizing numpy.int_ as a valid input is a numpy issue.  Unlike regular int subclasses, numpy.int_ does not have Py_TPFLAGS_INT_SUBCLASS flag set:


>>> numpy.int_.__flags__ & (1<<23)
0

>>> class foo(int): pass
... 
>>> foo.__flags__ & (1<<23)
8388608

What python can improve in this area is documentation.  See issue9307.
msg110833 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-07-19 23:08
Py_TPFLAGS_INT_SUBCLASS is an implementation detail, and extension modules should not have to be aware of it.
Does Numpy correctly call PyType_Ready()?
msg110865 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-20 05:08
> Does Numpy correctly call PyType_Ready()?

As far as I can tell, it does:


#define DUAL_INHERIT(child, parent1, parent2)                           \
    Py##child##ArrType_Type.tp_base = &Py##parent2##ArrType_Type;       \
    Py##child##ArrType_Type.tp_bases =                                  \
        Py_BuildValue("(OO)", &Py##parent2##ArrType_Type,               \
                      &Py##parent1##_Type);                             \
    if (PyType_Ready(&Py##child##ArrType_Type) < 0) {                   \
        PyErr_Print();                                                  \
        PyErr_Format(PyExc_SystemError,                                 \
                     "could not initialize Py%sArrType_Type",           \
                     #child);                                           \
        return -1;                                                      \
    }                                                                   \
    Py##child##ArrType_Type.tp_hash = Py##parent1##_Type.tp_hash;
    ...
    DUAL_INHERIT(Int, Int, SignedInteger);

Could it be the multiple inheritance that causes the problem here?
msg110870 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-07-20 06:27
multiple inheritance should not be a problem: there can be only one "dominant base", which is 'int' in this case.
someone with a debugger should step into this call to PyType_Ready() and see why it does not set the flag correctly (at the end of inherit_special())
msg112912 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-04 22:26
Inconsistent headers ;-)
Should this be closed as invalid and superceded,
or open without resolution?
I cannot tell from discussion; one of you please choose.
msg112913 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-08-04 22:33
This is clearly an invalid issue for python, but I wanted to keep it open until I or someone else gets a chance to report it to numpy project.  I was hoping to close this with a link to numpy tracker.
msg112941 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-05 01:53
A link can be added at any time. Closing does nothing except remove an issue from searches of open issues. Please close this or allow me to. There is a cost to keeping things like this open -- the puzzlement of anyone reading it and the time cost to triage maintainers like me ;-)
msg112992 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-08-05 16:05
Terry,

Realistically, though, if you close this issue it is virtually guaranteed that not further action will be taken.  I have not been involved with numpy for a couple of years now, so it would take me some time to find their tracker and restore my credentials or open a new account.  If someone more familiar with the current numpy development comes across this issue, I would appreciate if he or she would make a report.

If you think keeping this issue open in this state is too much burden on triagers, feel free to close it.  IMO, however, triagers should ignore assigned issue unless they are more than say 6 months old with no activity.
msg123004 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-01 18:46
Travis, feel free to close this without further action or use the information here to open a numpy issue.
msg173061 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-16 16:55
It is more than 6 months old with no activity.
msg173103 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-10-16 21:55
From the macro posted earlier:
  Py##child##ArrType_Type.tp_base = &Py##parent2##ArrType_Type;

tp_base is *not* PyInt_Type, so I was wrong when I said that the dominant base was int.
This is wrong IMO: tp_base should be int, for the type to be correctly initialized.

Compare:
>>> from numpy import *
>>> print int_.__bases__, int_.__base__
(<type 'numpy.signedinteger'>, <type 'int'>) <type 'numpy.signedinteger'>
>>> class C(signedinteger, int):
...     pass
... 
>>> print C.__bases__, C.__base__
(<type 'numpy.signedinteger'>, <type 'int'>) <type 'int'>
msg174061 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-28 17:42
I created http://projects.scipy.org/numpy/ticket/2235 to track this.  Closing this issue.
msg193180 - (view) Author: Yury V. Zaytsev (zaytsev) Date: 2013-07-16 16:06
NumPy no longer uses the Trac for issues, the tracking has been moved to GitHub. I hope that the problem will be finally solved in NumPy by https://github.com/numpy/numpy/pull/3526 .
History
Date User Action Args
2013-07-16 16:06:20zaytsevsetnosy: + zaytsev
messages: + msg193180
2012-10-28 17:42:25mark.dickinsonsetstatus: open -> closed

messages: + msg174061
2012-10-16 21:55:10amaury.forgeotdarcsetmessages: + msg173103
2012-10-16 17:15:00mark.dickinsonsetnosy: + mark.dickinson
2012-10-16 16:55:46serhiy.storchakasetstatus: pending -> open
nosy: + serhiy.storchaka
messages: + msg173061

2010-12-01 18:46:11belopolskysetstatus: open -> pending
assignee: belopolsky -> teoliphant
messages: + msg123004
2010-08-05 16:05:44belopolskysetnosy: + teoliphant
messages: + msg112992
2010-08-05 01:53:42terry.reedysetmessages: + msg112941
2010-08-04 22:33:25belopolskysetmessages: + msg112913
2010-08-04 22:26:15terry.reedysetnosy: + terry.reedy

messages: + msg112912
versions: - Python 2.6, Python 2.5
2010-07-20 06:27:11amaury.forgeotdarcsetmessages: + msg110870
2010-07-20 05:08:39belopolskysetmessages: + msg110865
2010-07-19 23:08:11amaury.forgeotdarcsetstatus: pending -> open
nosy: + amaury.forgeotdarc
messages: + msg110833

2010-07-19 17:55:29belopolskysetstatus: open -> pending

type: behavior
versions: + Python 2.7
messages: + msg110788
superseder: Py_TPFLAGS_LONG_SUBCLASS is not documented
resolution: not a bug
stage: resolved
2010-07-05 20:15:54belopolskysetmessages: + msg109352
2010-05-26 00:57:37belopolskysetassignee: belopolsky

nosy: + belopolsky
2009-05-18 09:25:27grubertsetnosy: + grubert

messages: + msg88022
versions: + Python 2.6
2009-03-11 16:06:13LambertDWsetnosy: + LambertDW
messages: + msg83471
2009-03-11 15:51:24barronhsetcomponents: + Library (Lib)
versions: + Python 2.5
2009-03-11 15:50:45barronhcreate