The datetime module does not have a C API which means
that modules written in C cannot take advantage of the
datetime module without incurring significant overhead.
These patches supply this lack and are based on the C
API exported by the mxDateTime module and the cStringIO
module and enhanced as suggested by Guido.

Logged In: YES 
user_id=31435

Marc-Andre, can you review this?  I think you have more 
experience building C APIs for modules than anyone else 
(while I don't have any).

There's certainly no objection to giving datetime a C API, and 
Guido already said he doesn't want to rely on that Martin 
changed datetime to be built as part of the core (in 2.3 on 
Windows, datetime.pyd was a distinct DLL; in CVS, datetime 
is compiled into the core DLL now).

Anthony, you probably need to add doc and test patches.  
_testcapimodule.c holds tests of things you can only get at 
from C.

Logged In: YES 
user_id=619560

I have no objection to providing patches for doc and test. A
quick look at _testcapimodule.c doesn't provide any obvious
ideas as to how to patch it. I looked for cStringIO which
has a C API already and found nothing in there at all. The
documentation also simply says "see the source for the
information you require". Any suggestions as to how to proceed?

Logged In: YES 
user_id=38388

This is a good start. However, you should add more APIs for
extracting date/time information to the C API. mxDateTime.h
from egenix-mx-base can provide a good basis for this. 

If you want to make the datetime module usable for database
interfacing at C level, you'll also have to add interfaces for
Unix ticks conversion, since these are often used by database
C level interfaces.

Logged In: YES 
user_id=619560

I didn't think any additional API would be necessary for
extracting date time information since the following macros
are available:

PyDateTime_GET_YEAR(o)
PyDateTime_GET_MONTH(o)
PyDateTime_GET_DAY(o)
PyDateTime_DATE_GET_HOUR(o)
PyDateTime_DATE_GET_MINUTE(o)
PyDateTime_DATE_GET_SECOND(o)
PyDateTime_DATE_GET_MICROSECOND(o)
PyDateTime_TIME_GET_HOUR(o)
PyDateTime_TIME_GET_MINUTE(o)
PyDateTime_TIME_GET_SECOND(o)
PyDateTime_TIME_GET_MICROSECOND(o)

Were you thinking of something else that is needed?

As for interfacing at the C level for converting from Unix
ticks, the following method is already available and could
simply be used as a drop in replacement for DateFromTicks()
and TimestampFromTicks() from the DB API document.

DateFromTicks() --> datetime.date.fromtimestamp()
TimestampFromTicks() --> datetime.datetime.fromtimestamp()

TimeFromTicks() is not already exposed but discussions with
Tim Peters already suggested that would not happen since the
concept is rather strange. You'll have to discuss that with
him if you disagree! :-)

Any comments?

Logged In: YES 
user_id=38388

Thanks for the clarification. I had forgotten about the
macros -- I'd suggest that you document them in the
datetimeapi.h include file to not cause the same confusion
on the developer side (the trick to achieve adoption is to
make things easy on the developer).

As for TimeFromTicks(): 
This should be rather straight forward to implement: you
take the time part of the ticks timestamp and create a time
object from it. This is the way the DB API constructor works.

Could you add C APIs for the three DB API ticks interface
methods ?

Other things that are missing:
* documentation of the way the CAPI object can be used in
datetimeapi.h
* documentation of the various CAPI entries
* documentation of the API magic number (there should be
some scheme for this)

I think that datetimeapi.h should be enough for a developer
to read in order to use the interface.

Logged In: YES 
user_id=38388

Any progress on this ?

Logged In: YES 
user_id=619560

You haven't yet responded to my previous comments -- it
looks like SourceForge put them __after__ your comments so
it is quite understandable why you didn't notice them. If
you can give me the answers to those questions or provide
additional questions that I need to answer, then I think we
can make progress again. I was waiting for __you__ :-)

Logged In: YES 
user_id=619560

Oops! It looks like my own misunderstanding of SourceForge
and how it works is to blame. :-( I'll take a look at this
and get back to you as soon as possible -- ignore my
previous comment.

Logged In: YES 
user_id=38388

Anthony, have you had a chance to look at the comments ?

Logged In: YES 
user_id=619560

Sorry for the delay. Things here have been quite busy
recently both at work and at home. Let me see if I can
answer your questions.

1) are you suggesting comments that include the macro names
in datetimeapi.h similar to what I put in the comments
below? Or something else? See modified datetimeapi.h for
more info.

2) TimeFromTicks() is a very DB API specific routine. In
chatting with others about the datetime module, they were
not keen on adding such a beast. I guess there will have to
be some discussion about whether datetimeapi.h is also a
sort of half implementation for a C module implementing the
DB API. I was intending to do this in my cx_Oracle module
but not in datetimeapi.h. Could you clarify?

3) I could add C APIS for the three DB API ticks interfaces
but I was simply intending to do that in cx_Oracle. Again,
see #2. I can do it either way. :-)

4) I have added limited documentation in datetimeapi.h but
is that really the right place for it? Is it even remotely
close to what is needed? I'm not sure what is needed here
exactly.

5) the API magic number is simply an arbitrary number which
is stored in datetime.h and need not be understood by the
implementor. For your information I simply took the name
"datetime", converted each letter to its ordinal value in
the alphabet, modded by 16 and used that hex character --
does that make any sense? This can be changed to whatever
makes sense, though.

6) Whether or not datetimeapi.h should be sufficient for a
developer or whether he should look at documentation in the
usual locations (html pages, for example) I'll leave up to
the Python development team. Let me know how I can assist.

Logged In: YES 
user_id=38388

Thanks for the comments...

1)  Please put *all* the code for the C API into
datetimeapi.h. I don't like
your current mix of putting some things into datetime.h and
others into
datetimeapi.h (e.g. the PyDateTime_CAPI definition).

2) Since it's the only API missing if you want to use
datetime objects
in database interfacing, please add it.

3) dito.

4) Yes, the header file is the right place. Have a look at
unicodeobject.h
for what I have in mind (or mxDateTime.h for that matter).

5) Why add yet another magic number ? Isn't the Python API
number
enough ?

6) Since you don't provide a doc patch, please put comments into
the header file. There can never be enough documentation and 
developers tend to look at the code rather than the docs ;-)

Logged In: YES 
user_id=619560

I am back at work after finishing my move so I think I can
finally devote a little more time to this issue.

1-4) The main reason for datetimeapi.h as opposed to simply
modifying datetime.h was that the internal build needs to
define things __differently__ from an external build of a
module and there doesn't appear to be any way of fixing that
-- so if you know a way I think I can manage to create a
reasonable patch; otherwise, I'm out of my depth!

5) Guido was suggesting that a new PyIMPORT macro be made
available which would verify that the magic number defined
in the header file actually matches the magic number
exported by the built module; this is simply a developer
protection scheme which should eliminate really stupid uses;
you can ask him about this or I can e-mail you part of the
thread that brought this up

6) Makes sense. I'd like to get the questions above resolved
first before proceeding any further, though.

Logged In: YES 
user_id=619560

Any chance to look at this? Any hope of this being included
in Python 2.4? I know a number of people (including Guido)
who would appreciate that.... :-)

Logged In: YES 
user_id=38388

Sorry for not getting back to you earlier: I didn't see you last
message.

1-4) Python defines a macro during the Python build process
that is not defined when compiling extensions: Py_BUILD_CORE
You can use that macro to compile things differently for
core things vs. extensions.

5) Fair enough; I don't see a point in creating a new
numbering for each and every module. Please just use
1, 2, 3, 4, ... 

6) Hope this makes sense :-)

Logged In: YES 
user_id=619560

I spent some time getting the two include files merged
together, only to discover that Py_BUILD_CORE is __NOT__
defined when building modules, only when building stuff
linked directly into the core interpreter. Perhaps this
ought to be defined when building the modules with setup.py?
At any rate, I cannot proceed along the route suggested
without this problem being resolved.

On the DateFromTicks(), TimestampFromTicks(),
TimeFromTicks() issue, I'm not sure that there is much
benefit to including a C API for this since if the datetime
module becomes the standard method for managing datetime
objects, there is no point in having the DB API modules
provide something that the datetime module already provides,
right? During the transition period the few lines of code
needed can be posted. If you disagree I can certainly
provide the few lines as part of the patch but I suspect you
will find resistance to having this included by Tim Peters,
the maintainer (I believe) of the datetime module. Feel free
to correct me... :-)

Logged In: YES 
user_id=38388

About the Py_BUILD_CORE issue:

Sorry, I didn't know that Python does not define this macro for
core modules. Nevertheless, looking at the datetime module code
I don't really understand why you would need such a macro 
in the first place. Since the datetime module is the only code
including the datetime.h header file it should be easy to add
a macro definition just before importing datetime.h.

On the TimeFromTicks() issue:

The whole point of this discussion is to make the datetime
module easily usable for authors of database modules.
Since the DB API specifies that the module will need
to export the three functions, two of which are available
through the datetime module already, I don't really see
your point in not wanting to add it.

BTW, you are wrong in the assumption that the DB API spec
will move away from a generic API for date/time objects.
The DB API has always been defined in terms of interfaces
and does not force a certain set of tools onto the developer.
The datetime module will become one possible choice for
DB API authors, others will want to stick to mxDateTime,
use strings, ticks integers or more specific objects for 
interfacing. That won't change, so there is no 
"transition period".

Logged In: YES 
user_id=619560

About the Py_BUILD_CORE issue:

You are suggesting that I define this in datetimemodule.c
just prior to the include of datetime.h? Or are you
suggesting some other macro? I am not sure about the comment
about "not needing a macro in the first place" since there
are two groups that will be importing datetime.h: the first
being the datetime module itself and the second being the DB
API modules. The macro is definitely required to distinguish
between the two -- unless I am missing something?

On the TimeFromTicks() issue:

You obviously consider it important that the three methods
be available at the C level for DB API module authors to use
directly. My objection is simply a reiteration of an
objection raised by Tim Peters. I think you'll have to argue
that one with him. :-) I'll provide my implementations for
cx_Oracle as part of the patches and then you can fight it
out and I'll stay out of it. :-)

As for the transition period, I assumed that with the
introduction of a standard datetime module, that it would
become the suggested implementation for database interfaace
modules. That said, I don't really have any authority to say
anything on this front so I'll leave that to you and others
to decide. :-)

Logged In: YES 
user_id=38388

Macro:

I don't know whether you need the macro or not (you probably
do for the type checks). In any case, you can do without
relying on Python providing you this information via the build
process, since only datetimemodule.c is using the header
file while Python is being built.

TimeFromTicks(): 

I' ve stated my point. I don't think there's anything
to add - I'm tired of fighting for these things... one of
the reasons I stopped actively discussing things on 
python-dev. 

IMHO, a product that's aimed at developers
should make things easy for the developer and try to
avoid code duplication if possible. The API is needed
in order to support database interfaces that use Unix
ticks as basis for date/time, so if you don't add it to
the module, the ten or twenty module authors out
there will have to duplicate that piece of work in their
implementation.

Transition:

This was already discussed on the db-api list. We love
freedom of choice. Nothing to add.

Logged In: YES 
user_id=31435

I don't understand what TimeFromTicks() is supposed to do.  
Seconds-from-the-epoch is a point in time, not a time-of-
day.  If some DB API requires guessing some transformation 
from seconds-from-the-epoch to time-of-day, that's fine, but 
it doesn't belong in Python's datetime module.

The datetime module should certainly have a method to 
construct a datetime.datetime from a seconds-from-the-
epoch argument -- seconds-from-the-epoch is a way of 
specifying a datetime.datetime.  Given that, if the intent is to 
throw away the datetime.date portion of the 
datetime.datetime object, retaining only the datetime.time 
portion, then that's trivially accomplished by invoking dt.time
() (where dt is the datetime.datetime object).

Do any databases really store time-of-day as a seconds-from-
the-epoch value?  Google's top hit on TimeFromTicks() today 
happens to be a msg from Anthony saying that Oracle does 
not.

Logged In: YES 
user_id=38388

From the DB API specs:

        TimeFromTicks(ticks)
          
            This function constructs an object holding a
time value
            from the given ticks value (number of seconds
since the
            epoch; see the documentation of the standard
Python time
            module for details).
            
The reason for this having this API is to be able to construct
an object suitable for passing to the database given a Unix
ticks value. Since the ticks value contains both a date and 
a time part and most databases have a special column type
for time value, we need two constructor APIs, one to extract
just the date part and one for the time part.

Here's the mxDateTime implementation for reference:

def TimeFromTicks(ticks,
                  # Locals:
                 
DateTimeDelta=DateTimeDelta,localtime=_time.localtime):

    """ TimeFromTicks(ticks)

        Constructs a DateTimeDelta instance pointing to the
local time
        indicated by the given ticks value. The date part is
ignored.

    """
    return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6])

Logged In: YES 
user_id=619560

Attached are the modified patches based on your suggestions.
I have tested them with my own module cx_Oracle and they
appear to work correctly. Any further suggestions would be
appreciated. I would like to see this included for Python
2.4 if at all possible. Please let me know what I need to do
(if anything) to get this to happen.

As for the TimeFromTicks() routine, Oracle has absolutely no
use for a "time-only" data type and does not support it. It
clearly does not have broad support so I really have no
desire to implement it since I will never use it myself. If
you consider it important, you could add it yourself or we
could wait until someone else who requires it for their
implementation of the DB API requires it. I'm not trying to
be rude or trying to denigrate the DB API, simply trying to
be practical. Please let me know if this is offensive in any
way. Thanks.

Logged In: YES 
user_id=38388

Thanks, but I don't have time to review it.

As for the TimeFromTicks() API: 

I can understand that you don't feel like implementing it,
but hey, I won't use datetime either. The only reason I'm
trying to push full support of the DB API specs is that
Guido et al. wanted to see a note in the DB API that
datetime also provides a usable interface to do date/time
interfacing.

I'm not very interested in datetime myself, since I'll
continue to use mxDateTime, so the situation for me is
somewhat similar to yours: investing time into something
that I myself will probably not use much (I will add support
to mxODBC for those who like to use datetime instead of
mxDateTime, but won't use it myself).

Feel free to check in your changes. Thanks.

Logged In: YES 
user_id=38388

Tim, please review. Thanks.

Logged In: YES 
user_id=31435

Under the belief that the *intent* of this patch is to add a C 
API to Python's datetime module, it should stick to exposing C 
ways to spell what Python programmers can already do from 
Python using datetime.  Since nothing like TimeFromTicks() 
exists in the Python datetime API, it simply doesn't make 
sense to invent one available only from C.  It *may* make 
sense to invent one in a DB API wrapper around datetime, but 
I view that as out of scope for this patch.

I would not add TimeFromTicks() to the Python datetime API 
either, because the docs are ambiguous.  The sample 
implementation Marc-Andre posted here uses localtime() to 
resolve the ambiguity in the docs, but that has to be a wrong 
choice for some time-of-day apps.  For example, there's 
apparently no portable way to spell "noon" using 
TimeFromTicks():  the value you get back depends on which 
time zone localtime() happens to use at the time it's called.

If people want time-of-day from a POSIX timestamp, it 
remains easy to get it from what datetime already supplies, 
but to do so you have to be explicit about whether you want 
UTC or local time interpretation.  Both are easily spelled 
already, and it's a Good Thing that you need to be explicit if 
you want to do such a thing.

Logged In: YES 
user_id=619560

I would agree with your assessment of the issue regarding
the TimeFromTicks() API and find it quite reasonable. Either
implement both at the C and Python levels or not at all.
Since there is no consensus on this, none it is. :-)

Did you have a chance to review the actual code? The changes
are fairly minimal but you might have questions about
readability, names, etc. It would be great if this could be
included in an upcoming 2.4 alpha/beta so that I can provide
support in cx_Oracle for the next release. Thanks for your time.

Logged In: YES 
user_id=31435

Assigned to me for the next review.  Can't do it immediately, 
but will do my best to free up some time for it "soon".

Logged In: YES 
user_id=619560

Sure. What's the chance of it being reviewed prior to the
release of Python 2.4 a1? I just saw the preannouncement
today -- it is scheduled for the end of this month, I believe.

Logged In: YES 
user_id=31435

I intend to do it this week, but have no time today or 
tomorrow.

Logged In: YES 
user_id=31435

Heh.  Just noticed there are 6 patches attached to this 
report.  Are all of them part of the suggested change, or, if 
not, which are the current ones (if that's the case, the best 
way to answer the question is to delete the out-of-date 
patches)?

Logged In: YES 
user_id=31435

Assuming the current state of the proposal consists of the 
first two patches on the list ("patch for datetimemodule.c" 
and "patch for datetime.h"), I'm probably happy to commit 
them.  Can't say for sure right now, because SF CVS is dead, 
and can't get a current checkout to try them with.

The one necessary thing I don't see are docs:  how are users 
supposed to know this exists, let alone know how to use it?  
We need a new section for datetime objects in the Python/C 
API Reference Manual, in the Other Objects part of the 
Concrete Objects Layer section.  You don't have to write 
LaTeX, plain text is fine.  Look at the other sections in that 
chapter for what's needed (an overview, and a precise 
account of what's in the new C API and how to use it).

Don't let that discourage you -- you're very close!  If I don't 
force docs out of you now, docs will never get written.  
That's why I'm trying to force docs out of you now <wink>.

Logged In: YES 
user_id=619560

Yes, the top two files are the ones involved. As per your
suggestion I deleted the other four files to avoid further
confusion.

I understand the concern for documentation. I'll see what I
can do to get at least a preliminary patch on that front
ready this week. Earlier discussions suggested that the
documentation reside solely in the include file but I think
adding a section to the Python/C API documentation would
make a lot more sense. Please stand by. :-)

Logged In: YES 
user_id=619560

I have attached a patch for the documentation as requested.
Not being certain of the format and requirements for the
content, your comments are appreciated. I have a few
questions that might help focus your comments.

1) Should the API structure be documented and users of
datetime.h encouraged to use it directly? Or should it be
"implementation details" and only those methods that I have
provided should be available? To help answer this question,
is it intended to ever bring datetime objects into the
"core" rather than as a separate module?

2) The DB API helper functions are provided because Marc
Andre Lemburg thought it would be a good idea. Please take a
look and see if it makes sense now that it is documented. :-)

Please review and give me your feedback. Thanks.

Logged In: YES 
user_id=31435

Thanks for your patience, Anthony!  I checked this in after 
minor fiddling.  Biggest change was to supply a new macro 
and docs, to expose the unreferenced Delta_FromDelta 
member of the C API struct.  I do wish we had test cases -- 
as is, these maros could be totally busted and we'd have no 
way to know that.  But virtually all of the C API is untested 
that way, so it's not a fatal objection.

Doc/api/concrete.tex; new revision: 1.45
Include/datetime.h; new revision: 1.5
Misc/ACKS; new revision: 1.267
Misc/NEWS; new revision: 1.1008
Modules/datetimemodule.c; new revision: 1.74

The answers to your questions are implicit in what I checked 
in <wink>.  Note that in 2.4, datetimemodule.c *is* compiled 
as part of the core.  Guido nevertheless didn't want the 
datetime API to take advantage of that (I'm unclear on 
why).  I'm content to document the macros and leave it at 
that.

The DB API functions don't bother me one way or the other, 
so left them in.

Logged In: YES 
user_id=619560

Yes, this has been a long time coming. Collaboration by
e-mail with people working on completely different projects
can be quite a challenge! Thanks for getting this checked
in. When 2.4a1 comes out I will be providing a new version
of cx_Oracle that takes advantage of these functions.

I just noticed that I missed the macros that are defined in
datetime.h that were not modified with my patch --
specifically those used to acquire the components of the
date/time objects. Should those be documented as well? I'm
assuming the answer is yes -- so would it be helpful if I
did so and attached a patch to this set of patches or do I
need to create a new patch?

Logged In: YES 
user_id=31435

Good eye!  Yes, the accessor macros should be documented 
too, and it would be great if you whipped up a patch for 
that.  This report has already been closed, so best to open a 
new patch report for it.  You can assign the report to me and 
I'll make time to get it checked in.

Logged In: YES 
user_id=619560

The additional changes to the documentation are available in
patch 986010.

Logged In: YES 
user_id=591851

This API is seriously broken. datetime.h says 
static PyDateTime_CAPI *PyDateTimeAPI;

which means that including this file from two C (or C++) 
files leads to two independent PyDateTimeAPI symbols - 
calling PyDateTime_IMPORT in one file still leaves 
PyDateTimeAPI being 0 in the other!

I double checked, and this happens with both C and C++.

This needs to be revised so that there is indeed only one 
PyDateTimeAPI symbol, no matter how many files include that 
file. s/static/extern/ and defining "PyDateTime_CAPI 
*PyDateTimeAPI;" in one of the c files in python.