_zip_directory_cache has no single test nor piece of documentation. The
only place where it's mentioned is a call to method .clear() on it in
tests. Besides, it can be whatever. Despite _ prefix, which I would
consider to be private, distutils crash obscurely if it's not there or
does not contain sequences as keys.

I can't find a reference to _zip_directory_cache in distutils.  Do you
mean setuptools?

Seems I meant setuptools indeed. Note that brett's importlib contains
some tests for _zip_directory_cache.

That's fortunate, because I can just reassign this to him :)

Actually importlib does not test this; that is an old thing from an old
sandbox version of importlib that I no longer use.

I am assigning to Tarek as the better solution is for distutils to stop
using this "private" attribute.

_zip_directory_cache is part of the zipimport module (C module) you can
find in Modules/zipimport.c.

The tests are located here: ./Lib/test/test_zipimport.py

_zip_directory_cache is a dict used by zipimporter_init to speed up the
lookup.

Notice that if it is successfully compiled, zipimport is used by pkgutil.

So Maciek, did you have a bug with this ?

The bug with this is that setuptools uses it (and even relies on what is
there and in what order), while it has no single tests and no documentation.

> while it has no single tests and no documentation."

what "it" refers to ? zipimport, distutils or setuptools ?

it == _zip_directory_cache

I am not sure what you would like to see added.

in zipimport.c:

"- _zip_directory_cache: a dict, mapping archive paths to zip directory
  info dicts, as used in zipimporter._files.
"

while there are no direct test, Lib/test/test_zipimport.py covers it
becauseit is used when you instanciate a zipimport class.

would you like to see a section explaining how it works at 
http://docs.python.org/library/zipimport.html ?

maybe what's missing is a public API for the clear() method since
setuptools uses it by calling it directly

setuptools at least deletes stuff from _zip_directory_cache (via _uncache
function in easy_install.py). it also iterates over it and checks for
existance of elements.

What about documenting details about _zip_directory_cache:
* does it need to be a normal python dictionary or anything?
* what methods it needs to support?
* what api zipimporter exposes?

Right now the situation is:
if I want my library not to be cached any more, I need to grab object
called _zip_directory_cache, and remove things from it. That's a very
non-obvious API for caching. Is it documented in some PEP?

The way I stomped on it - just passing the tests of zipimporter module
actually does not guarantee that you've implemented it correctly. You
also need to expose at least _zip_directory_cache otherwise setuptools
won't be too happy.

Stdlib is not only about testing cpython, it's also about what you
define as python these days.

I propose to have a clean api on zipimporter how to invalidate it's
caches. Maybe a general mechanism for that, I don't know.

Does distutils use _zip_directory_cache in any way? If not then setuptools
is in the wrong here for using a private attribute and it is on them to make
it work if the attribute does not exist, period.

Otherwise someone will need to come up with a very simple API or set or
rules to follow for clearing the cache and get setuptools to switch to it
(IMO it should be handled by simply deleting all entries for a module in
sys.modules, removing the path hook from sys.path_hooks, and then clear out
sys.path_importer_cache for all paths associated with the module). Be aware,
though, this is all leading down the road of special-casing zip imports when
you may very well run into the same issues if people start to develop other
importers -- e.g. tarball, dbm, or sqlite3 importers -- and this will all
start up again.

> Does distutils use _zip_directory_cache in any way?

Not at all. But pkgutil does, and uses to get a dirlist,
so I guess a new API should be created in zipimport to handle this case
as well. (I think it's some setuptools code originallyt, that landed in
pkgutil)

2009/4/5 <"Tarek Ziadé <report@bugs.python.org>"@psf.upfronthosting.co.za>

>
> Tarek Ziadé <ziade.tarek@gmail.com> added the comment:
>
> > Does distutils use _zip_directory_cache in any way?
>
> Not at all. But pkgutil does,

Ah, OK.

> and uses to get a dirlist,

Couldn't one just open the zip file with zipfile to get the directory list?
Is there any real requirement that the cache must be used?

>
> so I guess a new API should be created in zipimport to handle this case
> as well. (I think it's some setuptools code originallyt, that landed in
> pkgutil)

Probably.

Could the recently-added functools.lru_cache help here?

Eric, as you know importlib, can you comment about this?  If zipimport now works with the invalidate_caches system, then we do have a documented way of clearing the cache, and I would recommend closing this bug (the existing attribute should stay for compat, but we should not document or test it more IMO).

I'll take a look but can't say when.  My wife's is having a baby in the next couple days so I'm a bit swamped.  :)

No problem (and congratulations :)