Google just released ipaddr.py, a module that knows how to manipulate IP
addresses (both IPv4 and IPv6).

I have nothing to do with this module, but I suggest considering it for
inclusion in the standard library.

It fills a gap for address manipulations that will become more important
to fill as IPv6 becomes more widespread.

Where can we find this?

It would help if I added a link to the Google release:

http://code.google.com/p/ipaddr-py/

Description:

"

An IPv4/IPv6 manipulation library in Python.

This library is used to create/poke/manipulate IPv4 and IPv6 addresses
and prefixes.

"

Almost time machine. :)

I see a list of owners in the code (although it's difficult to infer
real names or email addresses from that list). I think we should not
include the code without their explicit approval.

The question will then always be: what is the official master copy of
the code? The one in Python, or the one on Google code? Whose
responsibility would it be to keep those synchronized, and incorporate
changes from one copy into the other?

I would prefer if the copy in Python (say, 2.7) becomes the master copy,
and the copy on Google code eventually disappears (when interest in
older Python versions has died).

I would object to a mere fork of the code (i.e. where one of the regular
Python committers incorporates, from time to time, the changes that
Google made)

as one of the owners listed in the code.google.com project (same name),
and the original author, you certainly have my approval for inclusion in
python.

On Wed, Sep 24, 2008 at 9:07 PM, Martin v. Löwis <report@bugs.python.org> wrote:

Martin v. Löwis <martin@v.loewis.de> added the comment:
I see a list of owners in the code (although it's difficult to infer
real names or email addresses from that list). I think we should not
include the code without their explicit approval.

I know they *want* this to happen, no worries on this front.

The question will then always be: what is the official master copy of
the code? The one in Python, or the one on Google code? Whose
responsibility would it be to keep those synchronized, and incorporate
changes from one copy into the other?

I would prefer if the copy in Python (say, 2.7) becomes the master copy,
and the copy on Google code eventually disappears (when interest in
older Python versions has died).

I'm in favor of this, and I believe the authors at Google are too --
it was written out of necessity, and once integrated, the need for a
separate Google copy will go away.

I would object to a mere fork of the code (i.e. where one of the regular
Python committers incorporates, from time to time, the changes that
Google made)

Agreed.

I'm the maintainer of IPy library. Another library for IPv4/IPv6
manipulation. The code is old (was written for Python 2.2?), but
released under BSD license. Main issue of this library: it's unable to
manipulation "invalid ranges": 10.20.4.0/30 is valid but
10.20.4.1-10.20.4.5 is invalid since it's not possible to create a
bitmask matching the range. I don't care if you choose Google library,
maybe better. But you might check IPy to reuse some idea ;-)

Ooops, the website: http://software.inl.fr/trac/wiki/IPy

Another Python library: http://erlug.linux.it/~da/soft/iplib/

Now that 2.6 and 3.0 are out of the way, how should we move forward?
Will a PEP be required?

Now that 2.6 and 3.0 are out of the way, how should we move forward?

I think some committer needs to take charge and work with the authors
on merging the code. If pmoody wants commit access, that should be set
up, and he should add it himself in a commit-after-review manner (with
an actual patch, including documentation and tests).

If he doesn't want to do the integration, some committer needs to
volunteer - I personally won't (lack of time).

In any case, we would need a copyright form signed, unless Guido can
confirm that the code is covered by some agreement between Google and
the PSF.

Will a PEP be required?

I doubt that; a quick announcement to python-dev might be sufficient
(in general, a PEP should only be done if dissent is likely). It
seems that Victor didn't actually disagree to integrating this
package; we should somehow encourage IPy users to review the code
and comment on the (possibly lack of) functionality.

I think this might be worth a look before any hard and fast decisions
are made :-

http://code.google.com/p/netaddr/

I'm biased ;) but I don't see what netaddr provides over ipaddr. it
also seems to be in the neighborhood of 50% slower (at least on my mac
mini).

pmoody@mini - 04:52 PM - ~/Downloads/ipaddr-1.0.1
-> python -m timeit 'import ipaddr;\
ipaddr.IP("1.1.1.1")'
10000 loops, best of 3: 46.2 usec per loop

pmoody@mini - 04:52 PM - ~/Downloads/netaddr-0.5.2
-> python -m timeit 'import netaddr;\
netaddr.IP("1.1.1.1")'
10000 loops, best of 3: 71.9 usec per loop

dealing with netmasks, the difference is even greater.
pmoody@mini - 04:52 PM - ~/Downloads/ipaddr-1.0.1
-> python -m timeit 'import ipaddr;\
ipaddr.IP("1.1.1.1/255.255.255.0")'
10000 loops, best of 3: 78.4 usec per loop

pmoody@mini - 04:49 PM - ~/Downloads/netaddr-0.5.2
-> python -m timeit 'import netaddr;\

netaddr.IP("1.1.1.1/255.255.255.0")'
1000 loops, best of 3: 247 usec per loop

(and it doesn't seem to deal with hostsmasked addresses).

it also seems to be layed out in an unintuitive way, treating address
ranges a something very different than network addresses. hopefully
someone who isn't intimately familiar with both libraries can comment,
but ipaddr feels a lot cleaner to me.

Performance shouldn't be a major concern here. Utility is more
important and the implementation can be optimized later.

My initial impression of netaddr is pretty good. One thing it has going
for it is documentation. The netaddr google page gives a really great
overview of the functionality provided. In comparison, the ipaddr-py
page is extremely sparse; the documentation in the repository is no
better (as there isn't really any).

There are a lot of minor things about ipaddr-py which are off-putting.
It uses a naming convention that's not like any real Python software I'm
familiar with. In particular, CamelCase method names really draw the
eye (in a bad way) and distract from the actual functionality being
provided. I assume that if ipaddr-py is selected for inclusion, the API
will be adapted to fit PEP-8. In comparison, netaddr appears to largely
comply with PEP-8 already.

Regarding the treatment of addresses versus address ranges, I prefer
netaddr's solution. An address is not the same as an address range. It
seems odd to try to represent them both with the same type as ipaddr-py
does. This feels somewhat APLesque (where everything is a vector),
which isn't bad in itself, but generally if I want to manipulate an IP
address, I'm not thinking about it as a list of one address. If I
wanted to do that, then I'd make a list and put the address in it. By
comparison, the separate CIDR type in netaddr makes a lot of sense. It
preserves a memory-efficient representation of the address range, but
does it with a different type than the fundamental address type.

This is far from a complete review of either library, obviously. I've
done little more than glance over the documentation of each and do a
small amount of interactive playing around. A lot more effort seems to
have gone into making netaddr accessible and appealing, and I think it
has succeeded. Perhaps if some documentation for ipaddr-py is written
it will be easier to see where that library excels. Of course, I assume
this is a prerequisite for inclusion in the standard library as well.

hm, all addresses have a subnet, even if its an implied /32, so
specifying a network as ("1.1.1.0", "1.1.1.255") seems a lot more
off-putting than "1.1.1.0/24". You're also much more likely to see the
latter in network devices. I guess I don't see the utility in an address
range of .1 - .22 (or something arbitrary, something which doesn't fall
on a power-of-2 boundary); when dealing with ranges of addresses, i've
always only wanted to/needed to manipulate sub-networks of addresses.

and my expectation was always that method names, etc. would have to be
converted to more closely match other python code; ipaddr was written to
google's python style guide, which I understand can be different than
the python.org style guide ;)

also, ipaddr does make extensive use of pydoc, but you're right, the
webpage is spartan.

happy (PST) new year.

Cheers,
/peter

hm, all addresses have a subnet, even if its an implied /32, so
specifying a network as ("1.1.1.0", "1.1.1.255") seems a lot more
off-putting than "1.1.1.0/24". You're also much more likely to see the
latter in network devices.

I'm not sure which API in netaddr you're referring to. If you want to
construct that /24 with netaddr, then I would use
netaddr.address.CIDR("1.1.1.0/24"). Offhand, I can't find an API which
accepts two endpoints of a range to construct a network in netaddr.
When I wrote about having separate types for individual addresses vs
ranges of addresses in my previous comment, I had IP and CIDR
respectively in mind, as opposed to ipaddr-py's single IPv4 class which
can represent either.

I'm not sure which API in netaddr you're referring to. If you want to
construct that /24 with netaddr, then I would use
netaddr.address.CIDR("1.1.1.0/24"). Offhand, I can't find an API which

netaddr.AddrRange

class AddrRange(__builtin__.object)
 |  A block of contiguous network addresses bounded by an arbitrary start and
 |  stop address. There is no requirement that they fall on strict bit mask
 |  boundaries, unlike L{CIDR} addresses.
 |

| __init__(self, first, last, klass=<class 'netaddr.address.Addr'>)
| Constructor.
|
| @param first: start address for this network address range.
|
| @param last: stop address for this network address range.
|
| @param klass: (optional) class used to create each object returned.
| Default: L{Addr()} objects. See L{nrange()} documentations for
| additional details on options.

when looking through the code.google.com wiki, I couldn't find any
examples of creating addresses with netmasks and AddrRange was the
first thing I found when looking through pydoc.

accepts two endpoints of a range to construct a network in netaddr.
When I wrote about having separate types for individual addresses vs
ranges of addresses in my previous comment, I had IP and CIDR
respectively in mind, as opposed to ipaddr-py's single IPv4 class which
can represent either.

and still having two separate classes represent the same thing seems odd to me.

ipaddr appears to be on a fast track for "batteries included" status
without much consultation in the wider Python community. As BDFL it's
ultimately Guido's call, but it would be disappointing to see one
solution being chosen wholesale without much additional discussion.

There is no point in wrangling over the individual features of one
library at revision x versus another at revision y. Software is a moving
target. Omissions and bugs are easily dispatched. Ultimately we all want
the same thing, better software that makes our lives easier.

As for netaddr it is intended to cover more ground that all the other
solutions out there. Here is a handy list of them :-

http://code.google.com/p/netaddr/wiki/YetAnotherPythonIPModule

This may preclude its suitability for the standard library in any event
being intended to support any and all network address types, not just
IPv4 and IPv6. It is also currently, very much in beta, being about 75%
feature complete with 6-9 months work of (part-time) work left on it
before it can reach version 1.0 ready. A Python 3.0 version is also
planned for release soon.

In answer to various specific issues raised about netaddr.

The need for separate IP and CIDR classes is set out in the FAQ page :-

http://code.google.com/p/netaddr/wiki/NetAddrFAQ

FYI, AddrRange is a (not yet abstract) base class with a start and end
address covering the shared functionality of 3 subclasses (IPRange,
CIDR, Wildcard) which should all interchangeable (within reason). It is
also supports generic address (Addr objects) rather than just IP
specific ones. Base 2 slices is not always the best way of expressing
groups of network hosts, they are *one* way.

With netaddr, I wanted to avoid the 'super object' syndrome that seems
to dog virtuall all other existing efforts out there, and not only those
in Python.

In any event confusion will hopefully be dealt with as the docs improve.

As for performance, netaddr's current speed for certain operations areas
may be slower than other implementations but the optimisation phase of
development hasn't even begun. There are loads more features to add and
get working before that starts.

One point I will make is on licensing. netaddr is BSD, so you can do
whatever you want with it and contribute as you see fit.

At the present time, contributing to ipaddr for anyone outside Google
seems like the software equivalent of climbing Mount Everest! Even if I
personally wanted to contribute to this particular project's efforts,
having to sign the individual CLA would automatically preclude me from
doing so. This is a *major* sticking point.

Regardless of whatever decision is taken, netaddr will always be
available as long as it has a user base, either as built in, optional
extra or as a part of some wider standardisation effort.

I think this might be worth a look before any hard and fast decisions
are made :-

http://code.google.com/p/netaddr/

Looking at code isn't really helpful for determining whether it can
be included into Python. What matters more is whether the authors
want to contribute it, how they want to support it afterwards, and
so on.

One point I will make is on licensing. netaddr is BSD, so you can do
whatever you want with it and contribute as you see fit.

That is not sufficient for inclusion to Python. We also want support
from the author (along with a promise to eventually give up the external
project).

That is not sufficient for inclusion to Python. We also want support
from the author (along with a promise to eventually give up the external
project).

As Guido had mentioned in a previous message, this is along the lines
of my thinking wrt ipaddr.

Cheers,
/peter

I am a contributor to netaddr, having deprecated my own old and crufty
IP address library in its favor. JP's comments on the library in this
ticket are included in the set of reasons that I initially chose it to
replace my own.

When I first looked at the ipaddr code a month ago, the features I was
delighted to see in the ipaddr project include address exclusion and
address collapsing (we'd been discussing these features in netaddr since
my old project had similar functionality).

Perhaps the following might be a prudent course:

• determine how small or big a standard library ip address module or
  subpackage should be;
• have a mutual discussion on the netaddr and ipaddr mail lists to
  determine what would need to be changed in each project in order to
  support inclusion in the standard lib;
• choose the option that requires the least code changes (simple naming
  convention changes should probably be given less weight than any changes
  to logic).

As for shutting down any project that is chosen, does such an action not
leave older Python versions out in the cold? Shouldn't the project
remain open to support Python versions < 2.7, with a highly visible note
that the code is included in 2.7/3.1+? (I am completely ignorant of
related Python development policy.)

Perhaps the following might be a prudent course:

This sounds all good. It is better if the experts in a domain
make a recommendation, than if some random committer makes
a choice.

As for shutting down any project that is chosen, does such an action not
leave older Python versions out in the cold? Shouldn't the project
remain open to support Python versions < 2.7, with a highly visible note
that the code is included in 2.7/3.1+?

Of course - hence I said "eventually". The project could continue to
maintain the external version as long as they please, provided it
doesn't diverge from the in-core version (unless it is the in-core
version itself that diverges). What I don't want to see happen is that
the community recommends at some point to ignore the outdated crappy
version in the core, and replace it with the more-powerful bug-fixed
version available separately. This has happened in the past, so I'm
extremely cautious here.

> As for shutting down any project that is chosen, does such an action
not
> leave older Python versions out in the cold? Shouldn't the project
> remain open to support Python versions < 2.7, with a highly visible
note
> that the code is included in 2.7/3.1+?

Of course - hence I said "eventually". The project could continue to
maintain the external version as long as they please, provided it
doesn't diverge from the in-core version (unless it is the in-core
version itself that diverges). What I don't want to see happen is that
the community recommends at some point to ignore the outdated crappy
version in the core, and replace it with the more-powerful bug-fixed
version available separately. This has happened in the past, so I'm
extremely cautious here.

Fantastic! Thanks for the clarification.

David, in the event of netaddr's complete or partial inclusion, are you
+1 with this and the maintenance of an ip/net library in Python?

Guido, what is your preference regarding feature set/size for an ip/net
library in Python?

If there's going to be lots of discussion, perhaps it should be taken to
python-dev?

David, in the event of netaddr's complete or partial inclusion,
are you +1 with this and the maintenance of an ip/net library
in Python?

Yes, I would be very happy to help with this.

If there's going to be lots of discussion, perhaps it should be
taken to python-dev?

Good idea.

Peter M. and the ipaddr contributors, are you all happy to proceed in
accordance with Duncan's suggestions? If so, let's kick off a thread on
python-dev to get the ball rolling and see what we can come up with.

Good idea.

Peter M. and the ipaddr contributors, are you all happy to proceed in
accordance with Duncan's suggestions? If so, let's kick off a thread on
python-dev to get the ball rolling and see what we can come up with.

I'm fine with this. But as Duncan mentioned, some guidance from the
benevolent powers that be on what exactly they're looking for before
we launch into a bunch of work would be appreciated.

Cheers,
/peter

I'm fine with this. But as Duncan mentioned, some guidance from the
benevolent powers that be on what exactly they're looking for before
we launch into a bunch of work would be appreciated.

I think Guido's original message summarizes that: a module that
fills a gap for address manipulations... In addition, it should
have all the organisational qualities (happy user base, determined
maintainers, copyright forms, documentation, tests). As to what
precisely its API should be - that is for the experts (i.e. you)
to determine. I personally think performance is important, in
addition to a well-designed, useful API. Conformance to PEP-8
is also desirable.

I've been on vacation and unable to follow this, and won't have time to
catch up now. Note that I have no vested interest in Google's module
except knowing it has many happy users (I have never used it myself --
but Collin Winter has and he tells me it's great).

Since I haven't used either module and am not particularly interested in
this functionality, I'm withdrawing myself from the set of experts who
can decide which is better.

Duncan McGreggor asked some questions on python-dev; here are my responses:

• do we want to limit it to IP (i.e. no EUI/MAC support)?

I don't want to exclude EUI/MAC support, but it seems quit a separate
(and much more specialized) application area, so it's probably best to
keep it separate (even if it may make sense to use a common (abstract
or concrete) base class or just have similar APIs).

• do we want a single module or is a package acceptable?

I don't care either way.

• other thoughts?

How about a merger?

r72210 PEP-8-ified the test names.

Since Python 2.7 and 3.1 have not yet shipped, I hope it's not too late
to
continue this discussion. I have no vested interest in either ipaddr or
netaddr, but I am interested in seeing a well-designed and usable IP
address
library included in the stdlib.

From reading the comments in this issue, it seems most of the discussion
focused on netaddr's API. There seems to have been little discussion
about
the merits of ipaddr's API, which is unfortunate given that ipaddr is
the
library being included in the stdlib. There was some technical
discussion on
the ipaddr and netaddr lists, but it amounted to each developer
defending his
library, and neither party seemed willing to compromise on key design
decisions. At the end of the reconciliation period, ipaddr looks much
the
same as it did before, albeit with prettier indentation.

When looking for an IP address library for use in a network scanning and
discovery application I wrote last year, I decided against ipaddr
because of
its conflation of address and network -- it seems strange to me to have
one
class modeling both concepts.

After reading the technical discussion on the ipaddr and netaddr lists,
it is
clear to me that ipaddr's designers have a somewhat limited
understanding of
IP addressing, and have not designed an API that articulates IP concepts
particularly well. For example, see pmoody's earlier comments in this
ticket:

all addresses have a subnet, even if its an implied /32

This is not only incorrect, but it demonstrates a deep misunderstanding
of
how IP works. IP addresses most certainly do not have a "subnet", or a
mask
of any sort. An IPv4 address is simply a 32-bit number.

IPv4 *networks* have masks, of course, hence the name "netmask". These
network masks are used by the routing process to lookup the next hop (or
directly connected network) for an IP datagram by matching the
datagram's
destination address against the networks and masks defined in the
routing
table.

specifying a network as ("1.1.1.0", "1.1.1.255") seems a lot more
off-putting than "1.1.1.0/24".

Since networks are commonly represented in at least three forms, the API
should support at least these:

CIDR/prefix notation: "192.168.1.0/24"
address+mask notation: "192.168.1.0/255.255.255.0"
range notation: "192.168.1.0-192.168.1.255"

I guess I don't see the utility in an address range of .1 - .22
(or something arbitrary, something which doesn't fall on a power-of-2
boundary); when dealing with ranges of addresses, i've always only
wanted to/needed to manipulate sub-networks of addresses.

This statement demonstrates an extremely narrow view of the problem
domain.
Access lists and DHCP pools are two common examples of where arbitrary
address ranges are useful.

At least three concepts deserve first-class treatment in any meaningful
IP
address library:

• addresses
• ranges of addresses
• networks

To conflate these is a mistake.

My hope is that now that a library has been selected, it can be improved
before Python 2.7 and 3.1 ship. Now is the best time to make
backwards-incompatible API changes; once ipaddr ships with the stdlib,
we're
stuck with it. If it remains as-is, it will be deadweight for those app
developers like myself who prefer an API that more accurately reflects
the
problem domain.

To this end, now that ipaddr is moving to Python stdlib, will Google's
contributor license agreement restriction be lifted? I would like to
contribute, but have not been able to secure a corporate CLA from my
employer.

Cheers,

Clay

On Mon, Jun 1, 2009 at 9:39 AM, Clay McClure <report@bugs.python.org> wrote:

Since Python 2.7 and 3.1 have not yet shipped, I hope it's not too late
to continue this discussion. I have no vested interest in either ipaddr or
netaddr, but I am interested in seeing a well-designed and usable IP
address library included in the stdlib.

Python 3.1 is as good as shipped. It is in the release candidate
stage which means no code changes other than bug fixes can go in.

Improvements will have to wait for 2.7 and 3.2. In the mean time I
suggest writing up improvements as either subclasses of the existing
ipaddr module (so that they can be used both with this standalone
project and with the version shipped in python 3.1).

> specifying a network as ("1.1.1.0", "1.1.1.255") seems a lot more
> off-putting than "1.1.1.0/24".

Since networks are commonly represented in at least three forms, the API
should support at least these:

CIDR/prefix notation: "192.168.1.0/24"
address+mask notation: "192.168.1.0/255.255.255.0"
range notation: "192.168.1.0-192.168.1.255"

I think range notation parsing could be implemented pretty easily.

Access lists and DHCP pools are two common examples of where arbitrary
address ranges are useful.

At least three concepts deserve first-class treatment in any meaningful
IP address library:

• addresses
• ranges of addresses
• networks

To conflate these is a mistake.

My hope is that now that a library has been selected, it can be improved
before Python 2.7 and 3.1 ship. Now is the best time to make
backwards-incompatible API changes; once ipaddr ships with the stdlib,
we're stuck with it. If it remains as-is, it will be deadweight for those app
developers like myself who prefer an API that more accurately reflects
the problem domain.

Too late for 3.1. But I see no reason for the existing API to be
considered bad. There is always room for improvement. I believe you
could add support for network address ranges on top of it without too
much difficulty.

Subclass the existing classes and inherit from a mixin that keeps
track of a start and end of range and extends the 'in' operator to
check if the comparison address is >= and <= the start and end
addresses.

To this end, now that ipaddr is moving to Python stdlib, will Google's
contributor license agreement restriction be lifted? I would like to
contribute, but have not been able to secure a corporate CLA from my
employer.

You can sign a Python Software Foundation contributor agreement
instead. http://www.python.org/psf/contrib/contrib-form/ We've been
a bit relaxed on requiring these from people at times but it is a good
thing to do for Python's sake.

No Google CLA will ever be required to make changes to anything in the
python project itself.

My hope is that now that a library has been selected, it can be improved
before Python 2.7 and 3.1 ship.

That is fairly unlikely. The 3.1 release candidate has been produced,
so the only options possible at this point are to either go ahead with
what is in the code, or withdraw the library from 3.1 if it can be
demonstrated to have severe flaws.

On Mon, Jun 1, 2009 at 1:09 PM, Martin v. Löwis <report@bugs.python.org> wrote:

> My hope is that now that a library has been selected, it can be improved
> before Python 2.7 and 3.1 ship.

That is fairly unlikely. The 3.1 release candidate has been produced,
so the only options possible at this point are to either go ahead with
what is in the code, or withdraw the library from 3.1 if it can be
demonstrated to have severe flaws.

False
>>> ipaddr.IPv4('192.168.1.1') == ipaddr.IPv4('192.168.1.1/32')
True

ipaddr makes no distinction between two fundamentally different
concepts -- to my mind, that is a serious flaw.

ipaddr has many other quirks that, while not technically flaws, are
design warts that deserve to be fixed before its target audience is
amplified by its inclusion in the stdlib.

To those arguing for ipaddr's inclusion in the stdlib, how many of you
will actually use ipaddr to develop software? As an actual developer
of network scanning and discovery software, I can tell you that I
would rather roll my own library than use ipaddr as it exists today.

Clay

Strangely, the leading line of my last response was eaten by the bug
tracker. It read:

>> 1 == (1,)

On Mon, Jun 1, 2009 at 1:33 PM, Clay McClure <report@bugs.python.org> wrote:
>
> Clay McClure <clay@daemons.net> added the comment:
>
> On Mon, Jun 1, 2009 at 1:09 PM, Martin v. Löwis <report@bugs.python.org> wrote:
>
>>> My hope is that now that a library has been selected, it can be improved
>>> before Python 2.7 and 3.1 ship.
>>
>> That is fairly unlikely. The 3.1 release candidate has been produced,
>> so the only options possible at this point are to either go ahead with
>> what is in the code, or withdraw the library from 3.1 if it can be
>> demonstrated to have severe flaws.
>
> False
>>>> ipaddr.IPv4('192.168.1.1') == ipaddr.IPv4('192.168.1.1/32')
> True
>
> ipaddr makes no distinction between two fundamentally different
> concepts -- to my mind, that is a serious flaw.

I don't see these a fundamentally different, I guess. can you
demonstrate how this equivalency makes ipaddr unusable?

ipaddr has many other quirks that, while not technically flaws, are
design warts that deserve to be fixed before its target audience is
amplified by its inclusion in the stdlib.

I haven't seen any new issues on code.google.com (and I haven't heard
of any being reported on the python bugtracker), so since you're using
this thread to report issues, can you elaborate?

To those arguing for ipaddr's inclusion in the stdlib, how many of you
will actually use ipaddr to develop software? As an actual developer
of network scanning and discovery software, I can tell you that I
would rather roll my own library than use ipaddr as it exists today.

have used it to develop software and will continue to use it to
develop software.

Cheers,
/peter

Clay

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue3959\>

---

ipaddr makes no distinction between two fundamentally different
concepts -- to my mind, that is a serious flaw.

Do you have an application in mind where this lack of distinction
would prevent writing the application in a straight-forward way?
IOW, could you do something if they were distinct that you can't
do because they are not?

>>> ipaddr.IPv4('192.168.1.1') == ipaddr.IPv4('192.168.1.1/32')
True

As a network engineer I don't see any inherent problem with that equality.
In fact I make use of that conceptual equality on a regular basis.

Further, if you were to add a specifically 'address-without-netmask'
type, the above equality would still be true, because then the above
would be comparing two addresses-with-netmasks and you would want to
apply the hostmask to a bare address for convenience. To get inequality,
you'd be comparing two different object types...which comparison would
be False by default.

On Mon, Jun 1, 2009 at 5:02 PM, R. David Murray <report@bugs.python.org> wrote:

> >>> ipaddr.IPv4('192.168.1.1') == ipaddr.IPv4('192.168.1.1/32')
> True

As a network engineer I don't see any inherent problem with that equality.
In fact I make use of that conceptual equality on a regular basis.

For an example of why 192.168.1.1 != 192.168.1.1/32, look no further
than ifconfig:

# ifconfig en0 192.168.1.1/32
# ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 192.168.1.1 netmask 0xffffffff broadcast 192.168.1.1
...

# ifconfig en0 192.168.1.1
# ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
...

Can you provide an example of when 192.168.1.1 does in fact equal
192.168.1.1/32?

Further, if you were to add a specifically 'address-without-netmask'
type, the above equality would still be true, because then the above
would be comparing two addresses-with-netmasks and you would want to
apply the hostmask to a bare address for convenience.  To get inequality,
you'd be comparing two different object types...which comparison would
be False by default.

I don't follow. Assuming hypothetical Address and Network classes, as
accurately models the problem domain, we would have:

False

That seems to me the correct behavior, since an address is in fact not
the same thing as a network.

Clay

On Mon, Jun 1, 2009 at 4:41 PM, Clay McClure <report@bugs.python.org> wrote:
>
> Clay McClure <clay@daemons.net> added the comment:
>
> On Mon, Jun 1, 2009 at 5:02 PM, R. David Murray <report@bugs.python.org> wrote:
>
>>> >>> ipaddr.IPv4('192.168.1.1') == ipaddr.IPv4('192.168.1.1/32')
>>> True
>>
>> As a network engineer I don't see any inherent problem with that equality.
>> In fact I make use of that conceptual equality on a regular basis.
>
> For an example of why 192.168.1.1 != 192.168.1.1/32, look no further
> than ifconfig:
>
> # ifconfig en0 192.168.1.1/32
> # ifconfig en0
> en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>        inet 192.168.1.1 netmask 0xffffffff broadcast 192.168.1.1
>        ...
>
> # ifconfig en0 192.168.1.1
> # ifconfig en0
> en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
>        ...
>
> Can you provide an example of when 192.168.1.1 does in fact equal
> 192.168.1.1/32?

what this shows is that your copy of darwin defaults to a /24
prefixlen; ipaddr assumes a /32 prefixlen. I don't see anything
particularly *more* intuitive with darwin, but in any event, it seems
to provide support for assuming a prefixlen when none is supplied.

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue3959\>

---

In pre-CIDR days, assuming a prefixlen of 24 for a 192.168.x.x address
made sense. Nowadays it is better not to make that assumption. So I
find ipaddr's default of 32 to be "safer" than using a class based default.

The larger point, however, is that there _is_ a mask associated with the
address in ifconfig. There _must_ be one. So that is not an example
that shows that a separate address class is useful.

As for the == thing, I agreed with you that address compared to network,
if you had an address class, would yield false. My point was that

HypotheticalNetworkClass('192.168.1.1') ==

HypotheticalNetworkClass('192.168.1.1/32')

should yield True, because, as I said above, in a CIDR world using a
default of a hostmask for an otherwise unadorned address makes the most
sense.

As for an example of when the equivalence is useful, it is useful every
time I set up an access rule or route that applies to a single host.
Otherwise, I _must_ give a specific netmask, because in real life the
classfull default is often not the correct netmask. Most networking
software that I've dealt with requires explicit netmasks (often with a
shorthand to specify an ip/hostmask pair). It is true that when a
netmask isn't required it generally defaults to the classful netmask,
but having such a default is becoming more rare with time, in my
experience (because of CIDR).

On Mon, Jun 1, 2009 at 4:54 PM, Martin v. Löwis <report@bugs.python.org> wrote:

Do you have an application in mind where this lack of distinction
would prevent writing the application in a straight-forward way?
IOW, could you do something if they were distinct that you can't
do because they are not?

Consider applications that use ipaddr.IPv4 objects to configure
network interfaces:

ifconfig: 255.255.255.0/32: bad value

That's because ipaddr wrongly appends a prefix length to all
ipaddr.IPv4 objects, even those representing addresses, which do not
have prefix lengths.

Consider applications that need to validate addresses (or networks,
but not both) supplied as user input:

address = ipaddr.IP(input)

if isinstance(address, ipaddr.IPv4):
    if address.prefixlen != 32:
        raise TypeError("Expecting IP address, not network")
elif isinstance(address, ipaddr.IPv6):
    if address.prefixlen != 128:
        raise TypeError("Expecting IP address, not network")

Given its myriad quirks, it is really rather surprising that ipaddr is
being considered for inclusion in the Python stdlib.

Clay

On Mon, Jun 1, 2009 at 5:47 PM, Clay McClure <report@bugs.python.org> wrote:

Clay McClure <clay@daemons.net> added the comment:

On Mon, Jun 1, 2009 at 4:54 PM, Martin v. Löwis <report@bugs.python.org> wrote:

> Do you have an application in mind where this lack of distinction
> would prevent writing the application in a straight-forward way?
> IOW, could you do something if they were distinct that you can't
> do because they are not?

Consider applications that use ipaddr.IPv4 objects to configure
network interfaces:

ifconfig: 255.255.255.0/32: bad value

That's because ipaddr wrongly appends a prefix length to all
ipaddr.IPv4 objects, even those representing addresses, which do not
have prefix lengths.

I'm not sure what you're trying to do here, can you elaborate?

Consider applications that need to validate addresses (or networks,
but not both) supplied as user input:

address = ipaddr.IP(input)

if isinstance(address, ipaddr.IPv4):
   if address.prefixlen != 32:
       raise TypeError("Expecting IP address, not network")
elif isinstance(address, ipaddr.IPv6):
   if address.prefixlen != 128:
       raise TypeError("Expecting IP address, not network")

i'm not sure what's onerous about this code. you're missing a
try/except around ipaddr.IP(), but otherwise it seems fine.

Given its myriad quirks, it is really rather surprising that ipaddr is
being considered for inclusion in the Python stdlib.

it's actually already been included, but that's beside the point. I'm
now asking you a second time to submit bug reports if there are issues
which you see; perhaps these 'myriad of quirks' can be fixed, perhaps
not. yelling here doesn't actually do anything productive.

Clay

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue3959\>

---

On Mon, Jun 1, 2009 at 4:51 PM, pmoody <report@bugs.python.org> wrote:

>>>> ipaddr.IPv4('192.168.1.1') == ipaddr.IPv4('192.168.1.1/32')
> True
>
> ipaddr makes no distinction between two fundamentally different
> concepts -- to my mind, that is a serious flaw.

I don't see these a fundamentally different, I guess.  can you
demonstrate how this equivalency makes ipaddr unusable?

Fortunately, it's not up for debate: RFC-791 defines an IP address as
a 32-bit number, with no provision for a mask. Networks are defined by
their address and their mask. To correctly model them in an
object-oriented system, we would say that a Network has-a Address,
certainly not that a Network is-a Address.

I haven't seen any new issues on code.google.com (and I haven't heard
of any being reported on the python bugtracker), so since you're using
this thread to report issues, can you elaborate?

I will go ahead and open issues on code.google.com.

have used it to develop software and will continue to use it to
develop software.

I'd like to hear from application developers outside of Google. The
two that have commented on this issue seem not to prefer ipaddr's API.

Clay

On Mon, Jun 1, 2009 at 7:51 PM, pmoody <report@bugs.python.org> wrote:

> For an example of why 192.168.1.1 != 192.168.1.1/32, look no further
> than ifconfig:
>
> # ifconfig en0 192.168.1.1/32
> # ifconfig en0
> en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>        inet 192.168.1.1 netmask 0xffffffff broadcast 192.168.1.1
>        ...
>
> # ifconfig en0 192.168.1.1
> # ifconfig en0
> en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
>        ...

what this shows is that your copy of darwin defaults to a /24
prefixlen; ipaddr assumes a /32 prefixlen.  I don't see anything
particularly *more* intuitive with darwin, but in any event, it seems
to provide support for assuming a prefixlen when none is supplied.

The example demonstrates one case where the strings '192.168.1.1' and
'192.168.1.1/32' are not equivalent -- it wouldn't be hard to find
other examples -- yet you seem to think that (a) this is
Darwin-specific behavior, and that (b) this discrepancy is acceptable
and does not constitute a design flaw.

You're wrong on both fronts, since in fact all IP implementations
understand classful addressing (as per RFC-791), not just Darwin, and
your insistence on the equality of '192.168.1.1' and '192.168.1.1/32'
means that your library is unusable in applications that pass
ipaddr.IPv4 objects to ifconfig.

Clay

On Mon, Jun 1, 2009 at 8:05 PM, R. David Murray <report@bugs.python.org> wrote:

In pre-CIDR days, assuming a prefixlen of 24 for a 192.168.x.x address
made sense.  Nowadays it is better not to make that assumption.  So I
find ipaddr's default of 32 to be "safer" than using a class based default.

Sorry, but why should you determine what is better for my application?

The larger point, however, is that there _is_ a mask associated with the
address in ifconfig.  There _must_ be one.  So that is not an example
that shows that a separate address class is useful.

Again, you're wrong. The mask that you see in ifconfig is associated
with the network to which the interface is attached, not the interface
address. You also see a broadcast address in the ifconfig output, but
certainly you don't believe that the broadcast address is a property
of the interface address? No, of course not; it's a property of the
network to which the interface is attached -- just like the mask.

That's why we call it a "netmask".

As for an example of when the equivalence is useful, it is useful every
time I set up an access rule or route that applies to a single host.

Host routes are routes like all others -- they have a destination
address and a mask. That a host route has a prefix length of 32 does
not imply that the host route is equivalent to the host address.
You're confusing the two concepts.

Most networking
software that I've dealt with requires explicit netmasks (often with a
shorthand to specify an ip/hostmask pair).

By "ip/hostmask" pair, you actually mean "ip/netmask" pair, and yes,
this is a convenient notation for expressing two distinct but related
concepts: an address, and a mask. Addresses do not have masks;
networks do. I am not sure how to be any more clear about that point,
yet you still seem not to understand.

It is true that when a
netmask isn't required it generally defaults to the classful netmask,
but having such a default is becoming more rare with time, in my
experience (because of CIDR).

I'm not advocating classful routing, I'm merely stating (emphatically
and without ambiguity) that addresses and networks are different:
networks have masks; addresses do not. The ipaddr library forces a
mask on me every time I specify an address. In my view, that is a
design flaw.

Clay

On Mon, Jun 1, 2009 at 9:03 PM, pmoody <report@bugs.python.org> wrote:

> ifconfig: 255.255.255.0/32: bad value
>
> That's because ipaddr wrongly appends a prefix length to all
> ipaddr.IPv4 objects, even those representing addresses, which do not
> have prefix lengths.

I'm not sure what you're trying to do here, can you elaborate?

Let's say I have a UI that prompts users for two pieces of
information: interface address, and network mask. I then take those
two strings and make ipaddr.IPv4 objects out of them. This lets me
validate their correctness, and lets me perform convenient
calculations and tests (things the ipaddr library has done well).
Next, I take the IPv4 objects, coerce them to strings, and pass them
to ifconfig to configure an interface.

Assuming the user has supplied this information:

Interface address = '192.168.1.1'
Network mask = '255.255.255.0'

the following would get passed to ifconfig:

ifconfig en0 192.168.1.1/32 netmask 255.255.255.0/32

Clearly this is not what we expect, nor even legal. The principle of
least surprise is violated here, if nothing else.

We could work around this, of course, but why should we have to work
around our libraries when simply fixing them isn't that hard?

> if isinstance(address, ipaddr.IPv4):
>    if address.prefixlen != 32:
>        raise TypeError("Expecting IP address, not network")
> elif isinstance(address, ipaddr.IPv6):
>    if address.prefixlen != 128:
>        raise TypeError("Expecting IP address, not network")

i'm not sure what's onerous about this code.  you're missing a
try/except around ipaddr.IP(), but otherwise it seems fine.

Your definition of onerous apparently differs from mine :)

In my own applications, when I'm expecting a network, I use an
IPNetwork class, and when I'm expecting an address, I use an IPAddress
class. This means I can simply rely on the class constructors to do
the type checking for me. With ipaddr, I have to do the validation
myself. Simple isinstance() testing and duck typing don't work because
networks and addresses are represented (wrongly) by the same class.

it's actually already been included, but that's beside the point.  I'm
now asking you a second time to submit bug reports if there are issues
which you see; perhaps these 'myriad of quirks' can be fixed, perhaps
not.  yelling here doesn't actually do anything productive.

Rest assured, I've opened one issue and will open one or two more
before the night is out.

I'm sorry that you think "yelling" is unproductive. I happen to think
that healthy debate breeds better code.

Cheers,

Clay

On Mon, Jun 1, 2009 at 7:38 PM, Clay McClure <report@bugs.python.org> wrote:
>
> Clay McClure <clay@daemons.net> added the comment:
>
> On Mon, Jun 1, 2009 at 9:03 PM, pmoody <report@bugs.python.org> wrote:
>
>>> ifconfig: 255.255.255.0/32: bad value
>>>
>>> That's because ipaddr wrongly appends a prefix length to all
>>> ipaddr.IPv4 objects, even those representing addresses, which do not
>>> have prefix lengths.
>>
>> I'm not sure what you're trying to do here, can you elaborate?
>
> Let's say I have a UI that prompts users for two pieces of
> information: interface address, and network mask. I then take those
> two strings and make ipaddr.IPv4 objects out of them. This lets me
> validate their correctness, and lets me perform convenient
> calculations and tests (things the ipaddr library has done well).
> Next, I take the IPv4 objects, coerce them to strings, and pass them
> to ifconfig to configure an interface.
>
> Assuming the user has supplied this information:
>
> Interface address = '192.168.1.1'
> Network mask = '255.255.255.0'
>
> the following would get passed to ifconfig:
>
> ifconfig en0 192.168.1.1/32 netmask 255.255.255.0/32

I suppose that might be the case if you didn't know how to use ipaddr.
knowing that ipaddr accepts an ipaddress/netmask, I'd pass it

my_ip = ipaddr.IP('%s/%s' % (ipaddress, netmask))

and then check for any exceptions. I'd then ifconfig en0 str(my_ip)

Clearly this is not what we expect, nor even legal. The principle of
least surprise is violated here, if nothing else.

We could work around this, of course, but why should we have to work
around our libraries when simply fixing them isn't that hard?

>> if isinstance(address, ipaddr.IPv4):
>>    if address.prefixlen != 32:
>>        raise TypeError("Expecting IP address, not network")
>> elif isinstance(address, ipaddr.IPv6):
>>    if address.prefixlen != 128:
>>        raise TypeError("Expecting IP address, not network")
>
> i'm not sure what's onerous about this code.  you're missing a
> try/except around ipaddr.IP(), but otherwise it seems fine.

Your definition of onerous apparently differs from mine :)

In my own applications, when I'm expecting a network, I use an
IPNetwork class, and when I'm expecting an address, I use an IPAddress
class. This means I can simply rely on the class constructors to do
the type checking for me. With ipaddr, I have to do the validation
myself. Simple isinstance() testing and duck typing don't work because
networks and addresses are represented (wrongly) by the same class.

so ipaddr doing this validation for the user actually could make some
sense, but this could be fixed by something as easy

  ipaddr.IP(ip, network=False)
or
  ipaddr.IP(net, network=True)

anyway (more below)...

> it's actually already been included, but that's beside the point.  I'm
> now asking you a second time to submit bug reports if there are issues
> which you see; perhaps these 'myriad of quirks' can be fixed, perhaps
> not.  yelling here doesn't actually do anything productive.

Rest assured, I've opened one issue and will open one or two more
before the night is out.

cool, I see that you did that. as an aside, this probably isn't the
best place for continued discussion on ipaddr. ipaddr has already been
shipped with python, so i'm not sure that debate on it's inclusion is
able to influence that decision. All of the folks who've worked on
ipaddr + the python committer are on ipaddr-py-dev@googlegroups.com
(looks like you are too), so might I suggest we continue this
discussion there?

I'm sorry that you think "yelling" is unproductive. I happen to think
that healthy debate breeds better code.

I think debate is healthy, too. but I measure my success at a debate
by my ability to convince other people of my side. I find I have
better success when I listen other people and respond with respect.
yelling promotes neither of those.

Cheers,
/peter

Cheers,

Clay

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue3959\>

---

Consider applications that need to validate addresses (or networks,
but not both) supplied as user input:

address = ipaddr.IP(input)

If that is a frequent need, it would be reasonable to add an API

address = ipaddr.IP(input, allow_mask=False)

which would raise an exception if a mask was specified (as an
old-style bit mask, or in CIDR form).

if isinstance(address, ipaddr.IPv4):
if address.prefixlen != 32:
raise TypeError("Expecting IP address, not network")
elif isinstance(address, ipaddr.IPv6):
if address.prefixlen != 128:
raise TypeError("Expecting IP address, not network")

With the current API, you don't need to write it in such a quirky
way. Instead

if address.numhosts != 1:
   raise TypeError("Expecting IP address, not network")

would do as well.

Given its myriad quirks

Well, you deliberately make it appear more quirky than it actually is.

I will go ahead and open issues on code.google.com.

If you want to see them fixed in Python, please report them to this
tracker.

On Tue, Jun 2, 2009 at 1:21 AM, Martin v. Löwis <report@bugs.python.org> wrote:

> I will go ahead and open issues on code.google.com.

If you want to see them fixed in Python, please report them to this
tracker.

I'd like to see the issues fixed upstream, and the library removed
from Python until it is satisfactory to the developers who will
actually use it. To my knowledge, every developer (outside of Google)
who has commented on the issue has indicated a preference for a
different API.

Thanks,

Clay

Consider applications that need to validate addresses (or networks,
but not both) supplied as user input:

address = ipaddr.IP(input)

if isinstance(address, ipaddr.IPv4):
   if address.prefixlen != 32:
       raise TypeError("Expecting IP address, not network")
elif isinstance(address, ipaddr.IPv6):
   if address.prefixlen != 128:
       raise TypeError("Expecting IP address, not network")

Support for this can be added (its too late for Python 3.1). User
input validation is a good use case. For now I suggest the simpler
code:

if '/' in input:
    raise TypeError("Expecting IP address")
address = ipaddr.IP(input)

Or for a more pedantic test prior to calling ipaddr.IP.

if re.match('^[0-9a-fA-F:.]+$', input):
    raise TypeError("Invalid characters in IP address")

Please file a feature request on bugs.python.org for this one if you
haven't already. We could add optional parameter(s) to ipaddr.IP to
enable only accepting host addresses or network addresses in the
future.

I'd like to see the issues fixed upstream, and the library removed
from Python until it is satisfactory to the developers who will
actually use it. To my knowledge, every developer (outside of Google)
who has commented on the issue has indicated a preference for a
different API.

That's not true - I'm outside of Google, and have not indicated such
a preference.

On Tue, Jun 2, 2009 at 2:18 AM, Martin v. Löwis <report@bugs.python.org> wrote:

> I'd like to see the issues fixed upstream, and the library removed
> from Python until it is satisfactory to the developers who will
> actually use it. To my knowledge, every developer (outside of Google)
> who has commented on the issue has indicated a preference for a
> different API.

That's not true - I'm outside of Google, and have not indicated such
a preference.

You've indicated no preference either way, and have said:

"I personally have no plans for using this library, or any other IP
address library"

I should think you would seek the opinion of those developers who
actually do have plans to use an IP address library. As far as I can
tell, every developer in that category, outside of Google, that has
commented on this issue here or in python-dev has advocated a
different API.

Clay

I should think you would seek the opinion of those developers who
actually do have plans to use an IP address library.

That's what this has been doing for the last 8 1/2 months.

As far as I can
tell, every developer in that category, outside of Google, that has
commented on this issue here or in python-dev has advocated a
different API.

Is there some sort of conspiracy theory-ish reason that a google
software engineer might be somehow unfairly influenced?

Cheers,
/peter

On Tue, Jun 2, 2009 at 2:52 AM, pmoody <report@bugs.python.org> wrote:

> As far as I can
> tell, every developer in that category, outside of Google, that has
> commented on this issue here or in python-dev has advocated a
> different API.

Is there some sort of conspiracy theory-ish reason that a google
software engineer might be somehow unfairly influenced?

From reading your comments and the code, it is clear that concepts
that aren't relevant at Google have been neglected. For that reason,
developers at Google who are already familiar with ipaddr might
consider its API quite natural because of their institutionalized
thinking. But since this library is now intended for general purpose
use outside Google, I should think it is important to consult
developers outside Google who have been exposed to a broader range of
IP addressing issues. I don't believe that "good enough for Google"
ought to be our acid test.

The fact that developers outside Google seem to prefer a different API
is not new -- comments in this issue dating back several months
reflect that fact. What I don't see is a comment that explains why
their concerns were not considered.

Clay

This issue is closed.

Please take discussion up on a mailing list.