I work on a large app and we noticed that a surprising portion of our heap was filenames embedded the the bytecode.

This one-line patch to intern filenames reduces our on-disk size about ~15% and brings down our heap and in-memory object count by a similar percentage.

How exactly does it bring down your disk space?

If you have a file with say a hundred functions, and each function contains
the full path of that file on disk, your pyc file will contain about
(100*(path_size+overhead)) bytes. In some cases, this is pretty
significant.

On Thu, May 26, 2011 at 12:47 PM, Benjamin Peterson
<report@bugs.python.org>wrote:

>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> How exactly does it bring down your disk space?
>
> ----------
> nosy: +benjamin.peterson
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12190>
> _______________________________________
>

2011/5/26 Mike Solomon <report@bugs.python.org>:
>
> Mike Solomon <msolo@gmail.com> added the comment:
>
> If you have a file with say a hundred functions, and each function contains
> the full path of that file on disk, your pyc file will contain about
> (100*(path_size+overhead)) bytes. In some cases, this is pretty
> significant.

I see. The support for saving interned strings no longer exists in Py3
where this "feature" would have to be added.

New changeset 27359a4e0f8c by Benjamin Peterson in branch 'default':
try to use the same str object for all code filenames when compiling or unmarshalling (#12190)
http://hg.python.org/cpython/rev/27359a4e0f8c

As you can see, I've implemented a similar solution in 3.3. It should have the same memory savings but not disk space saving. (This would require reintroducing the marshal feature for interned strings.)

The in-memory fix is really the most important - the disk space was a bonus
and an easy metric to gather.

Unfortunately, our app won't be upgrading to python 3.x.

On Fri, May 27, 2011 at 7:10 AM, Benjamin Peterson
<report@bugs.python.org>wrote:

>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> As you can see, I've implemented a similar solution in 3.3. It should have
> the same memory savings but not disk space saving. (This would require
> reintroducing the marshal feature for interned strings.)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12190>
> _______________________________________
>

Okay, I'll close.