With transition from Python2.5 to Python2.6 on current Debian stable I noticed that the python2.6 executable has now 2x size of python2.5's.
Half of lib-dynload/*  obviously have been embedded into the executable by default. While most of the selections may be somewhat reasonable, I want to protest against static inclusion of _ssl.so, which now draws libssl*.so and libcryto*.so at each Python startup. This module is rarely needed, and the draw is almost as fat as the Python binary itself and those libs are not genarally loaded in the system. Those 2 dependencies solely are against detailed versions even!! See below.
Besides load time and resource wastage, there are now e.g. likely problems with frozen python scripts due to the detailed version deps. (binding with unversioned libssl.so may be ok for future separate _ssl.so module?)


$ ldd /usr/bin/python2.5
        linux-gate.so.1 =>  (0xb78dc000)
        libpthread.so.0 => /lib/i686/cmov/libpthread.so.0 (0xb78c1000)
        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb78bd000)
        libutil.so.1 => /lib/i686/cmov/libutil.so.1 (0xb78b8000)
        libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7892000)
        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb774c000)
        /lib/ld-linux.so.2 (0xb78dd000)

$ ldd /usr/bin/python2.6
        linux-gate.so.1 =>  (0xb76e7000)
        libpthread.so.0 => /lib/i686/cmov/libpthread.so.0 (0xb76cc000)
        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb76c8000)
        libutil.so.1 => /lib/i686/cmov/libutil.so.1 (0xb76c3000)
        libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0xb7679000)
        libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0xb751d000)
        libz.so.1 => /usr/lib/libz.so.1 (0xb7509000)
        libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb74e3000)
        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb739c000)
        /lib/ld-linux.so.2 (0xb76e8000)



Note: "missing files" consumed from lib-dynload/ since Python2.5:

_functools.so 6780
_hashlib.so 11392
math.so 12492
array.so 32432
_socket.so 54228
strop.so 21616
spwd.so 7132
collections.so 21116
unicodedata.so 474792
itertools.so 29684
rgbimg.so 12416
select.so 12816
time.so 16412
grp.so 6868
_locale.so 15760
binascii.so 17344
_weakref.so 4816
cStringIO.so 17076
cPickle.so 68968
syslog.so 5824
_ssl.so 15452
_bisect.so 7568
operator.so 25392
fcntl.so 13536
_struct.so 24832
zlib.so 21708
_random.so 10368


(python2.7 not tested, as it is not available via apt-get so far.)

+1

I see this effect in the stock ubuntu 10.04 python 2.6.

I can't see it in my selfcompiled binaries for Solaris 10 and 2.7 ubuntu.

The reason seems to be that my compilated code uses the python shared libs, while the stock Ubuntu 10.04 python 2.6 is statically linked.

In any case, the shared lib version doesn't link against ssl:

"""
jcea@ubuntu:~$ ldd /usr/local/lib/libpython2.7.so.1.0
	linux-vdso.so.1 =>  (0x00007fff715ff000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00007f5ec3079000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f5ec2e75000)
	libutil.so.1 => /lib/libutil.so.1 (0x00007f5ec2c71000)
	libm.so.6 => /lib/libm.so.6 (0x00007f5ec29ee000)
	libc.so.6 => /lib/libc.so.6 (0x00007f5ec266b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5ec3698000)
"""

Of course, as soon as we use sockets, we will bring SSL in:

"""
Python 2.7.2 (default, Jul 14 2011, 00:30:51) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getpid()
5701
>>> import socket
[...]
jcea@ubuntu:~$ cat /proc/5701/maps|grep -i ssl
7f2f467cd000-7f2f46818000 r-xp 00000000 fc:0a 131554                     /lib/libssl.so.0.9.8
7f2f46818000-7f2f46a17000 ---p 0004b000 fc:0a 131554                     /lib/libssl.so.0.9.8
7f2f46a17000-7f2f46a19000 r--p 0004a000 fc:0a 131554                     /lib/libssl.so.0.9.8
7f2f46a19000-7f2f46a1e000 rw-p 0004c000 fc:0a 131554                     /lib/libssl.so.0.9.8
7f2f46a47000-7f2f46a4f000 r-xp 00000000 fc:0a 403582                     /usr/local/lib/python2.7/lib-dynload/_ssl.so
7f2f46a4f000-7f2f46c4e000 ---p 00008000 fc:0a 403582                     /usr/local/lib/python2.7/lib-dynload/_ssl.so
7f2f46c4e000-7f2f46c4f000 r--p 00007000 fc:0a 403582                     /usr/local/lib/python2.7/lib-dynload/_ssl.so
7f2f46c4f000-7f2f46c50000 rw-p 00008000 fc:0a 403582                     /usr/local/lib/python2.7/lib-dynload/_ssl.so
"""

Importing "socket" will bring SSL in with any python at least since 2.3 (the older version I can try now).

In any case Python 2.6 is closed. Any change will need to address 2.7 and up. I would say that 3.1 is off too, and possibly 3.2 too.

PS: Can you try to compile python using the shared lib?.

So isn't this saying that this is a problem with the distribution packaging and not with CPython itself?

"Of course, as soon as we use sockets, we will bring SSL in"

Indeed, as it is now. 
Suggestions:


* urllib.URLOpener.open_https  shall always exist, but fail on runtime.  non-existance with strange "AttributeError" is inelegant..bogus.
Note: concept of late import ftplib, .. is otherwise ok in urllib. same style shall be used for ssl on demand. see python-Bugs-1046077)

* httplib.HTTPSConnection.connect shall late-import ssl

* httplib.HTTPSConnection,HTTPS,FakeSocket shall always exist but error on runtime if ssl is not available; same reason as with open_https

(* httplib.test already late-imports ssl)

* imaplib.IMAP4_SSL.ssl,open shall late-import ssl

* smtplib.starttls should late-import ssl

* smtplib.SMTP_SSL._get_socket  should late-import ssl

* smtplib.SSLFakeFile shall always exist (same reason as with open_https)

* poplib.POP3_SSL.__init__ shall late-import ssl

* deprecated socket.ssl() shall late-import _ssl/ssl (and possibly      RAND_add, RAND_egd, RAND_status too if they need to exist globally for compabtibilty; constants to be entered fix into socket or _socket; sslerror perhaps a builtin in _socket, which _ssl then uses )

"Can you try to compile python using the shared lib"

yes I can try lot of things, but I'd need to do this on many machines. Yet I didn't create this issue for some local purpose ;-)  
99% of Pythons are installed by apt-get, .msi etc.
And now this turns out as bigger issue with the early import of ssl in those mentioned locations.

(A little surprising for me is that the Python2.6 of current Debian stable shall already be outdated. Its the new thing here ;-)  
Hope for Python 2.7 not beeing outdated so soon. Py3 has already 4 version - who can use Py3 in real word? Is the Python dev team too fast for reality? ;-) )

From a fresh-compiled Python 2.7:

$ ldd ./python
	linux-vdso.so.1 =>  (0x00007fff85cde000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f772f725000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f772f521000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f772f31e000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f772f09c000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f772ed2b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f772f941000)

(same with 3.3)

This is therefore a problem with the Debian package, not with the vanilla Python build (when using default options).

Also, the reason ssl is imported when socket is imported in Python 2 is for compatibility reasons. It doesn't happen in Python 3:

$ grep ssl Lib/socket.py 
$ ldd build/lib.linux-x86_64-3.3-pydebug/_socket.cpython-33dm.so 
	linux-vdso.so.1 =>  (0x00007fffa2f21000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f227ff8f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f227fc1e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f22803e1000)

Therefore, closing as not a Python bug.

> Is the Python dev team too fast for reality? ;-)

Actually, releasing a new version every 18 months (not counting bugfix releases) makes us fairly conservative (perhaps too much) in the modern software ecosystem ;)

"It doesn't happen in Python 3"

Yet the cheap/unnecessary pre-imports of ssl in those other mentioned socket using libs (urllib (cgi!),httplib,smtplib,pop,imap...) exist there. socket is rarely used directly, so not much difference to Py2 in effect overall.
And Python2.7 lives - which is important for the majority of users perhaps.

Thus I'd request to not close this issue so swift. This is IHMO really a point to make python startup significantly faster, with a rather simple means.

Also the linkage of _ssl solely against a detailed version of libssl/libcrypto is still questionable.


"This is therefore a problem with the Debian package"

I'm not into the Python build files. Just to ask/double-check: is that observed _semi_ static link selection (which is good otherwise - somebody must have done surprisingly lots of care) really from Debian or is there maybe a sort of 2nd default option bundle somewhere in Pythons configure? (If really not so I would go for Debian BTS.)

> "It doesn't happen in Python 3"
> 
> Yet the cheap/unnecessary pre-imports of ssl in those other mentioned
> socket using libs (urllib (cgi!),httplib,smtplib,pop,imap...) exist
> there. socket is rarely used directly, so not much difference to Py2
> in effect overall.

Well, by this measure, we probably have unnecessary imports all over the
place, since the general guideline is to import modules at the top-level
rather than inside functions (the reason is partly technical, to avoid
potential deadlocks with the import lock).

> Thus I'd request to not close this issue so swift. This is IHMO really
> a point to make python startup significantly faster, with a rather
> simple means.

If you are using a network library such as urllib or others you
mentioned, then startup time will surely be small compared to the time
spent sending and retrieving data over the network, no?

> Also the linkage of _ssl solely against a detailed version of
> libssl/libcrypto is still questionable.

I don't know the reasons (if any). Perhaps you can open a separate issue
about that?

> "This is therefore a problem with the Debian package"
> 
> I'm not into the Python build files. Just to ask/double-check: is that
> observed _semi_ static link selection (which is good otherwise -
> somebody must have done surprisingly lots of care) really from Debian
> or is there maybe a sort of 2nd default option bundle somewhere in
> Pythons configure? (If really not so I would go for Debian BTS.)

Well, seeing as Mageia's Python 2.7 doesn't have the problem, I really
think it must be Debian-specific:

$ ldd /usr/bin/python
	linux-vdso.so.1 =>  (0x00007fff4dd13000)
	libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0
(0x00007f45809ff000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f45807e3000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4580472000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f458026e000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f458006b000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f457fde9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4580dc0000)
$ ldd /usr/lib64/libpython2.7.so.1.0
	linux-vdso.so.1 =>  (0x00007fffb53ff000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc0d6455000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fc0d6251000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fc0d604d000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fc0d5dcb000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc0d5a5a000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc0d6a51000)

Also, I've just compiled a fresh Python 2.6 and I get similar results:

$ ldd ./python
	linux-vdso.so.1 =>  (0x00007fffa9795000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5f76439000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f5f76235000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f5f76032000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f5f75db0000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f5f75a3f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5f76655000)

>> "It doesn't happen in Python 3"
>>
>> Yet the cheap/unnecessary pre-imports of ssl in those other mentioned
>> socket using libs (urllib (cgi!),httplib,smtplib,pop,imap...) exist
>> there. socket is rarely used directly, so not much difference to Py2
>> in effect overall.
> Well, by this measure, we probably have unnecessary imports all over the
> place, since the general guideline is to import modules at the top-level
> rather than inside functions (the reason is partly technical, to avoid
> potential deadlocks with the import lock).

see e.g. issue #1046077. 
As said, in this case here its even about libs almost as big as the python binary itself (on many platforms).
When there are only few link points, and a huge effect (->usage probabilities), late imports shall be used. Python is dynamic.
The list, as posted, is short here. Grep "ssl" in Python lib.
Its not reasonable to draw the big libcrypto and libssl almost always.

>> Thus I'd request to not close this issue so swift. This is IHMO really
>> a point to make python startup significantly faster, with a rather
>> simple means.
> If you are using a network library such as urllib or others you
> mentioned, then startup time will surely be small compared to the time
> spent sending and retrieving data over the network, no?

no. This is to cheap here. I'd vote for some discipline regarding such levels of resource usage. 
(I often wonder why software today isn't much faster than years ago - though the nominal speed of hardware increases tremendously. package sizes grow, without appropriate growth of functionality. This is one example how the rescources are wasted too careless.)

For example each cgi script (which has to respond fast and does only a small job), which does  "import cgi" and a few lines; or a script which just uses e.g., urllib string format functions ... : the whole thing is drawn.

>
>> Also the linkage of _ssl solely against a detailed version of
>> libssl/libcrypto is still questionable.
> I don't know the reasons (if any). Perhaps you can open a separate issue
> about that?

Yet the issue of this library is here now. Why procrastinate?

>
>> "This is therefore a problem with the Debian package"
>>
>> I'm not into the Python build files. Just to ask/double-check: is that
>> observed _semi_ static link selection (which is good otherwise -
>> somebody must have done surprisingly lots of care) really from Debian
>> or is there maybe a sort of 2nd default option bundle somewhere in
>> Pythons configure? (If really not so I would go for Debian BTS.)
> Well, seeing as Mageia's Python 2.7 doesn't have the problem, I really
> think it must be Debian-specific:

as emphasized in my sentence such reasoning alone would be sloppy. Thats why I asked.
Does sb actually know, if this optimized semistatic module list is really not in Pythons configure somewhere?
(The Debians would have gone rather deep into issues when they really created that fine tuning on their own. almost can't believe. If so I'd even recommend to adopt that (except _ssl.so) generally into Pythons standard configuration - at least for Linux)

> (I often wonder why software today isn't much faster than years ago -
> though the nominal speed of hardware increases tremendously. package
> sizes grow, without appropriate growth of functionality. This is one
> example how the rescources are wasted too careless.)

I don't know of any evidence that software slowness has to do with code
size. When code isn't called, it doesn't pollute the instruction caches
and hence shouldn't affect execution speed.

I understand the concern about py2exe and similar distribution systems
(although distribution size should be much less important nowadays than
10 years ago). But, really, it's a separate issue.

> For example each cgi script (which has to respond fast and does only a
> small job), which does  "import cgi" and a few lines; or a script
> which just uses e.g., urllib string format functions ... : the whole
> thing is drawn.

Well, CGI scripts are a wasteful way to do programmatic page serving. If
you care about performance, you should have switched to something like
FastCGI or mod_wsgi.

> >> Also the linkage of _ssl solely against a detailed version of
> >> libssl/libcrypto is still questionable.
> > I don't know the reasons (if any). Perhaps you can open a separate issue
> > about that?
> 
> Yet the issue of this library is here now. Why procrastinate?

This sentence sounds like you want to dictate us what and how we should
work on. That won't fly, sorry. The reason we want to avoid tackling
multiple issues in a single tracker entry is simply so that the entries
stay readable and searchable.

(and, really, most projects' bug trackers work that way, for the same
reasons)

> (The Debians would have gone rather deep into issues when they really
> created that fine tuning on their own. almost can't believe.

There's nothing magical about libssl that would make us link it
statically to the executable; it's far too optional a dependency for
that. Perhaps Debian has its own bootstrapping requirements that mandate
it, or perhaps they simply made a mistake and nobody complained before?
Why don't you open an issue on their bug tracker, or at least try to
contact them? You would get a definite answer about it.

>> (I often wonder why software today isn't much faster than years ago -
>> though the nominal speed of hardware increases tremendously. package
>> sizes grow, without appropriate growth of functionality. This is one
>> example how the rescources are wasted too careless.)
> I don't know of any evidence that software slowness has to do with code
> size. When code isn't called, it doesn't pollute the instruction caches
> and hence shouldn't affect execution speed.

With slowness under the subject here I mean long startup time  (and slowness by overall memory impact on small systems like laptops, non-cutting edge hardware, even embedded systems.). Thats what is mainly unpleasant and what seems to not improve appropriately overall. 
Developers carelessly link bigger and bigger libraries which eats the hardware gains ...
There are however some good efforts: For example when Google came out with Chrome (and many after fast responding apps), fast startup time was the striking issue. And the old browsers meanwhile were challenged by that, and improved as well.

Please keep Python fast as well. I'm using it since 1.5.2, and that careless fatty degeneration is one of the main things I don't like. 
Python is a universal language. Most Python progs are small scripts.

Overall I wonder why you post here on the main topic "resource usage", when you don't care about issues of magnitude 2x memory usage. Why not close this topic for Python at all with your arguments?

>
> I understand the concern about py2exe and similar distribution systems
> (although distribution size should be much less important nowadays than
> 10 years ago). But, really, it's a separate issue.

that is not really a separate issue (because module decoupling is a pre-requisite therefor, a sort of show stopper).

And as mentioned its by far not the only issue.

>
>> For example each cgi script (which has to respond fast and does only a
>> small job), which does  "import cgi" and a few lines; or a script
>> which just uses e.g., urllib string format functions ... : the whole
>> thing is drawn.
> Well, CGI scripts are a wasteful way to do programmatic page serving. If
> you care about performance, you should have switched to something like
> FastCGI or mod_wsgi.

And how about other "scripts" ;-) 

I'm sure you find everywhere something how you can make all app programmers busy and not take care of the few cheap fixes mentioned in the system to make Python faster und usable easily for everybody.
I created this issue to improve Python and make experience significantly faster.
You seem to me being too interested in closing issues fast.

>If you care about performance

You have this sort of black-white arguments which are green and somehow really I think, you are perhaps misplaced in this category "resource usage".

>
>>>> Also the linkage of _ssl solely against a detailed version of
>>>> libssl/libcrypto is still questionable.
>>> I don't know the reasons (if any). Perhaps you can open a separate issue
>>> about that?
>> Yet the issue of this library is here now. Why procrastinate?
> This sentence sounds like you want to dictate us what and how we should
> work on. That won't fly, sorry. The reason we want to avoid tackling
> multiple issues in a single tracker entry is simply so that the entries
> stay readable and searchable.
>
> (and, really, most projects' bug trackers work that way, for the same
> reasons)

You are free to divide the issue if you really think its worth multiple.
But why close it swift-handed before that is sorted out / set up? I indeed wonder about that careless style here meanwhile. Its not as it was years ago.

To me this "issues" seem to belong rather so close together, that the possible fix should perhaps be made in one go.

>> (The Debians would have gone rather deep into issues when they really
>> created that fine tuning on their own. almost can't believe.
> There's nothing magical about libssl that would make us link it
> statically to the executable; it's far too optional a dependency for
> that. Perhaps Debian has its own bootstrapping requirements that mandate
> it, or perhaps they simply made a mistake and nobody complained before?
> Why don't you open an issue on their bug tracker, or at least try to
> contact them? You would get a definite answer about it.

So are you definitely saying/knowing, there is really no such mentioned optimized module selection  (~50% of so modules since Python2.5 on Debian) somewhere in the Python build files?
(I ask first here to not create unnecessary lots of issues)

> Overall I wonder why you post here on the main topic "resource usage",
> when you don't care about issues of magnitude 2x memory usage.

Because you are talking about a fixed overhead of a mere 2MB (IIUC),
which is moreover shared between all processes using libssl/libcrypto
(and chances are these libraries are already loaded by something else on
your system - crypto is useful after all - so Python would actually not
add anything substantial in that regard). It is IMHO not interesting at
all, but still, you are welcome to post an issue about it if you are
concerned.

On the other hand, we do care about "resource usage" issues when they
are about dynamic memory consumption, e.g. reducing the size of objects.
Or, of course, memory leaks.

[...]
> You have this sort of black-white arguments which are green and
> somehow really I think, you are perhaps misplaced in this category
> "resource usage".
[...]
> I indeed wonder about that careless style here meanwhile. Its not as
> it was years ago.

Nobody here is interested in exchanging personal attacks, I'm afraid.

> You are free to divide the issue if you really think its worth multiple.
> But why close it swift-handed before that is sorted out / set up?

It *is* sorted out. You complained about libssl being linked into the
executable and it turned out to be Debian-specific. Just because you
think there is some kind of "big picture" problem involved doesn't make
it ok to turn this issue into a catch-all for all related issues.

> So are you definitely saying/knowing, there is really no such
> mentioned optimized module selection  (~50% of so modules since
> Python2.5 on Debian) somewhere in the Python build files?
> (I ask first here to not create unnecessary lots of issues)

Let's say that I don't know about such a selection, and neither probably
do other core developers on this issue, otherwise they would already
have chimed in. There is certainly no such documented or tested thing,
anyway, so even if there were I wonder why the Debian people would
decide to use it.

I just checked the file python2.6_2.6.6-8.diff.gz from the Debian python2.6 package: http://packages.debian.org/squeeze/python2.6

This diff file contains a """patch to build _hashlib and _ssl extensions statically""" that modifies Modules/Setup.dist
So it was definitely Debian choice to link to libssl.