classification
Title: cporting docs recommend using Include/intobject.h, which was removed in 3.1?
Type: behavior Stage: resolved
Components: Documentation, Interpreter Core Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Ramchandra Apte, benjamin.peterson, christian.heimes, dmalcolm, docs@python, eric.smith, georg.brandl, gvanrossum, iustin, lemburg, mark.dickinson, python-dev, skrah
Priority: normal Keywords: patch

Created on 2009-11-18 21:05 by dmalcolm, last changed 2013-01-20 22:33 by skrah. This issue is now closed.

Files
File name Uploaded Description Edit
issue7353.diff skrah, 2013-01-17 20:22 review
Messages (14)
msg95455 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2009-11-18 21:05
I'm attempting to package Python 3 for a Linux distribution, together with a stack of python extension 
modules; I'm currently using Python-3.1.1.   (see https://fedoraproject.org/wiki/Features/Python3F13 )

Many of these extension modules are close to compiling under both python 2 and python 3.

The page "http://wiki.python.org/moin/PortingExtensionModulesToPy3k" refers to using the macros in 
intobject.h, so that all PyInt_* calls in the python 3 build are aliased to PyLong_ API hooks.

Similarly, the page: http://docs.python.org/howto/cporting.html recommends using this file.

However, that header file was removed in this commit:
  http://svn.python.org/view?view=rev&revision=71697
with this message:
  Issue #4910:  PyNumber_Int is deprecated in 3.0.1; will be removed in 3.1.
(which links to this issue: http://bugs.python.org/issue4910 ; that issue refers to removal of the 
nb_long slot).

This seems to make it harder to port modules.

Is is acceptable if I ship that header file in my distribution packages of python-3.1.1 ?  (possibly 
with a reworded deprecation warning?)

Alternatively, is the fix to migrate all usage of the PyInt_ API to the PyLong_ equivalents ?  That 
would cause a change of behavior for the python 2 builds, assuming a shared source tree.

(I'd much prefer to ship the latest in the py3k branch than to stay with 3.0.1 for this)


In any case, it seems like the porting documentation isn't in sync with the code.

Hope this is helpful.
msg95470 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-11-19 14:07
> However, that header file was removed in this commit:
>  http://svn.python.org/view?view=rev&revision=71697

Hmm.  That was me.

> with this message:
>   Issue #4910:  PyNumber_Int is deprecated in 3.0.1; will be removed in
> 3.1.

That's the message for r69517, I think, which just added an entry to
Include/intobject.h.  The message for r71697 was:

"The comments at the top of intobject.h say that it will be removed in
3.1.  Make it so."

I'm not too sure of the history here, but I suspect that
Include/intobject.h was at least partly an aid to porting the Python
extension modules from 2.x to 3.0, rather than third-party modules.

Christian, Benjamin:  any comments?

I'm -1 on resurrecting Include/intobject.h in its old location.  I don't
much like the idea of having a random include file that's no longer used
by Python itself in the main Include directory;  it'll likely succumb to
bitrot before long.  But maybe there's a place for 2-to-3 porting aids
in the Tools directory, where there's less expectation that files are
well-maintained?

> Is is acceptable if I ship that header file in my distribution
> packages of python-3.1.1 ?  (possibly with a reworded deprecation
> warning?)

I can't see any problem with this.

I agree the porting documentation needs to be updated, whatever happens.
msg95471 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-11-19 14:09
I didn't mean to assign this to Georg.  Apologies.
msg95478 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-11-19 14:49
You didn't.  Doc bugs are automatically assigned to Georg by the tracker.
msg95516 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-11-19 23:03
Hmm, I wish intobject.h hadn't been removed so soon. I'm not really sure
how a file of #defines could suffer bitrot. This point is probably moot,
though because there's little point in having its presence skip a
version. I suppose sticking it in Tools or even Doc/includes is the
second best option.
msg95546 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-11-20 14:10
[Benjamin]
>I wish intobject.h hadn't been removed so soon.

Yes;  I'm sorry about that.

> I'm not really sure how a file of #defines could suffer bitrot.

Good point.  Me neither.
msg95586 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-11-21 17:04
I don't think it would hurt to put it back, would it?  I think the "remove 
this in 3.1" note was made when we expected 3.1 to be happening 1.5 years 
after 3.0 rather than a few months.
msg95621 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2009-11-23 12:24
MvL made this comment in
http://www.mail-archive.com/python-dev@python.org/msg43844.html

I'm copying it here so it doesn't get lost and because I think he makes
a good point that many people would miss (at least I didn't think of it).
-----------------------------------------------

The macros (unfortunately) allowed
to make non-obvious mistakes. Now that they are gone, people need to
really think of what precisely they want to do.

For example, consider

if (PyInt_Check(o)){
  long val = PyInt_AsLong(o);
  // process
} else if (PyLong_Check(o)) {
  long long val = PyLong_AsLongLong(o);
  // check for overflow
  // process
}

With intobject.h, this code would continue to compile, but work
incorrectly, as the second case will never be executed. It would
be better to port this as

#if Py2.x
if (PyInt_Check(o)){
  long val = PyInt_AsLong(o);
  // process
} else
#endif
if (PyLong_Check(o)) {

i.e. eliminating the int case altogether. For another example,

long foo = PyInt_AsLong(Foo);

has a hidden error in 3.x, with intobject: PyLong_AsLong might
overflow, which the 2.x case doesn't.

So eliminating intobject.h likely helps avoiding subtle errors.

Regards,
Martin
msg95624 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2009-11-23 12:52
Eric pointed me to this ticket after having raised the question on
python-dev: http://www.mail-archive.com/python-dev@python.org/msg43841.html

I think the discussion should be continued there instead of on this ticket.

Just for completeness, I'm copying my reply to Martin's reply here
(http://www.mail-archive.com/python-dev@python.org/msg43849.html):

"""
> For example, consider
> 
> if (PyInt_Check(o)){
>   long val = PyInt_AsLong(o);
>   // process
> } else if (PyLong_Check(o)) {
>   long long val = PyLong_AsLongLong(o);
>   // check for overflow
>   // process
> }
> 
> With intobject.h, this code would continue to compile, but work
> incorrectly, as the second case will never be executed. It would
> be better to port this as
> 
> #if Py2.x
> if (PyInt_Check(o)){
>   long val = PyInt_AsLong(o);
>   // process
> } else
> #endif
> if (PyLong_Check(o)) {
> 
> i.e. eliminating the int case altogether.

Sure, but that assumes that the original code already had support
for Python longs, which a lot of code doesn't.

In an ideal world, developers would add that code to their
extensions right away. In the real world, where developers only
have limited resources available, you'll get more 3.x ports
by making such ports as painless as possible while at the
same time not forcing them to alienate their 2.x user base.

The long support could then be added in later releases
of the extensions, giving the developers more time adapt.

> For another example,
> 
> long foo = PyInt_AsLong(Foo);
> 
> has a hidden error in 3.x, with intobject: PyLong_AsLong might
> overflow, which the 2.x case doesn't.

That's not quite true: PyInt_AsLong(obj) will try the
nb_int slot on non-integer objects which can return errors
(it returns -1 and sets the error message).

> So eliminating intobject.h likely helps avoiding subtle errors.

In the long run, yes. In the short run, other criteria are
more important, IMHO.
"""

IMO, it would be worthwhile collecting all Python 2.x compatibility C
APIs in two new files:

 * py2compat.h
 * py2compat.c

These could then be used in extensions and make the use of such
compatibility APIs explicit in the extension.
msg96911 - (view) Author: Iustin Pop (iustin) * Date: 2009-12-27 15:43
Hi,

Might I suggest that, whatever the outcome of the re-adding intobject.h
discussion, the documentation is updated? I think I'm not the only
module author which spent time trying to understand why the 3.1
documentation refers to non-existent header before finally finding this
bug :)

thanks a lot,
iustin
msg180140 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2013-01-17 16:45
Bump... is this still valid?
msg180151 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-01-17 20:22
I tend to agree with the argument that the removal of intobject.h was
a good thing, since it avoids subtle errors.

Probably no one wants to reinstate intobject.h at this point, so unless
there are objections, I'll update the docs in a couple of days.
msg180299 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-01-20 16:15
Go ahead!

The intobject header file used to make porting to Python 3 easier. Nowadays it's no longer required.
msg180326 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-01-20 22:31
New changeset 6df456f8fc6d by Stefan Krah in branch '3.3':
Issue #7353: Remove references to Include/intobject.h in the C-porting howto.
http://hg.python.org/cpython/rev/6df456f8fc6d
History
Date User Action Args
2013-01-20 22:33:00skrahsetstatus: open -> closed
stage: needs patch -> resolved
resolution: fixed
versions: + Python 2.7, Python 3.3, Python 3.4, - Python 3.1, Python 3.2
2013-01-20 22:31:07python-devsetnosy: + python-dev
messages: + msg180326
2013-01-20 16:15:16christian.heimessetmessages: + msg180299
2013-01-17 20:22:59skrahsetfiles: + issue7353.diff

nosy: + skrah
messages: + msg180151

keywords: + patch
2013-01-17 16:45:32Ramchandra Aptesetnosy: + Ramchandra Apte
messages: + msg180140
2010-08-24 22:54:35eric.araujosetassignee: docs@python

nosy: + docs@python
2010-05-20 20:35:23skip.montanarosetnosy: - skip.montanaro
2009-12-27 15:43:00iustinsetnosy: + iustin
messages: + msg96911
2009-11-23 12:52:53lemburgsetnosy: + lemburg
messages: + msg95624
2009-11-23 12:24:13eric.smithsetnosy: + eric.smith
messages: + msg95621
2009-11-21 17:04:07gvanrossumsetnosy: + gvanrossum
messages: + msg95586
2009-11-20 14:10:15mark.dickinsonsetmessages: + msg95546
2009-11-19 23:03:26benjamin.petersonsetmessages: + msg95516
2009-11-19 14:50:15r.david.murraysetnosy: - r.david.murray
2009-11-19 14:49:24r.david.murraysetpriority: normal

type: behavior

title: Why was Include/intobject.h removed in 3.1? -> cporting docs recommend using Include/intobject.h, which was removed in 3.1?
nosy: + r.david.murray
versions: + Python 3.2
messages: + msg95478
stage: needs patch
2009-11-19 14:09:55mark.dickinsonsetassignee: georg.brandl -> (no value)
messages: + msg95471
2009-11-19 14:07:24mark.dickinsonsetnosy: + benjamin.peterson, mark.dickinson, christian.heimes, georg.brandl
messages: + msg95470

assignee: georg.brandl
components: + Documentation
2009-11-18 22:16:22skip.montanarosetnosy: + skip.montanaro
2009-11-18 21:05:40dmalcolmcreate