This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: refactoring and documenting ModuleFinder
Type: Stage:
Components: None Versions:
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: theller Nosy List: jvr, nnorwitz, theller
Priority: normal Keywords: patch

Created on 2002-11-25 18:52 by theller, last changed 2022-04-10 16:05 by admin. This issue is now closed.

File name Uploaded Description Edit
modulefinder.patch jvr, 2002-11-27 11:02 better missing module reporting, take 2.
thomas1.patch theller, 2002-11-28 12:04 remove some warnings about missing modules
modulefinder_just#3.patch jvr, 2002-11-29 10:16 better missing module reporting, take 3
Messages (16)
msg41771 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-25 18:52
This is not (yet) a complete patch, it is thought as a 
workspace for improvements to 
tools/freeze/moduulefinder, so that it can eventually go 
into the standard library.
msg41772 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-25 18:58
Logged In: YES 

First question for anyone listening: isn't the windows 
_try_registry stuff unneeded, because imp.find_module 
handles this itself? Even in python15, imp.find_module
("pythoncom") returns 'c:\\winnt\\system32\\pythoncom15.dll' 
on my system.
msg41773 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-25 19:59
Logged In: YES 

Assign to Mark, hoping for an answer.
msg41774 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-26 07:57
Logged In: YES 

Mark writes in private email: "Modules listed in the registry 
was a dumb idea.  This whole scheme can die.
AFAIK, no one in the world uses it (including win32all since 
the last build)."
msg41775 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-26 08:11
Logged In: YES 

I've removed in current CVS the support for modules listed in 
the registry.
msg41776 - (view) Author: Just van Rossum (jvr) * (Python triager) Date: 2002-11-27 01:26
Logged In: YES 

Here's a patch. Summary: any_missing() returns less bogus

- I've rewritten scan_code() more or less from scratch,
factored bits and pieces out for readability.
- keep track of global assignments and failed imports per
module; use this to determine whether the Y in "from X
import Y" is a submodule or just a global name. This is not
100% doable: you can't tell which symbols are imported when
doing a star import of a non-Python module short of actually
importing it.
- added a new method to ModuleFinder: any_missing_maybe(),
which returns *two* lists, one with certain misses, one with
possible misses. The possible misses are *very* often false
alarms, so it's useful to keep this list separate.
any_misses() now simply returns the union of

I pretty sure this is all 100% backward compatible, so after
some more testing it could be checked in.
msg41777 - (view) Author: Just van Rossum (jvr) * (Python triager) Date: 2002-11-27 01:30
Logged In: YES 

Oh, I just realized should still be
patched to use any_missing(), or even better,
any_missing_maybe(). Will do if this gets through.
msg41778 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-27 08:27
Logged In: YES 

Running the current CVS ModuleFinder on a simple script 
which contains 'import os' with Python 2.2 on Win2k finds 27 
modules, while with this patch it only finds 13.

(and unassign Mark)
msg41779 - (view) Author: Just van Rossum (jvr) * (Python triager) Date: 2002-11-27 11:02
Logged In: YES 

Whoops, that was a useless patch :-( I managed to forgot to
copy the last three lines of the original scan_code() which
takes care of delayed imports... Fixed in the new patch.

I now also patched to actually use the
new any_missing_maybe() feature.

(Note for the lib doco to be written: we should document
that people should use any_missing[_maybe]() instead of
looking directly at mf.badmodules. mf.badmodules is the raw
list, any_missing_maybe() filters out the false positives.)

I've ran before/after tests on all of Lib/*.py and
Lib/test_*.py and the only differences I found are those
that this patch address, ie. less false positives when
reporting missing modules.
msg41780 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-28 12:04
Logged In: YES 

Now that it works really fine, I would like to make progress.
This new patch 'thomas1.patch' is against the patched 
modulefinder, it is easier to read this way.

First it passes all changes to self.badmodules through the 
_add_badmodule() method.

Then, and this part is not really polished, but you may get the 
idea, it introduces a new ignored instance variable which is 
thought to suppress most of the remaining missing modules 
msg41781 - (view) Author: Just van Rossum (jvr) * (Python triager) Date: 2002-11-28 18:22
Logged In: YES 

What's the difference between this "ignore" variable and the
"exclude" list? And why do you need to ignore modules based
on importing module? In my OSX appbuilder stuff I simply
have a list of modules that will be removed from the
'missing' list when reporting:

MAYMISS_MODULES = ['mac', 'os2', 'nt', 'ntpath', 'dos',
	'win32api', 'ce', '_winreg', 'nturl2path', 'sitecustomize',
	'org.python.core', 'riscos', 'riscosenviron', 'riscospath'

(I think I could just add this to the exclude list and get
the same effect.)

Let's please keep it simple!
msg41782 - (view) Author: Just van Rossum (jvr) * (Python triager) Date: 2002-11-28 20:17
Logged In: YES 

Thomas, your _add_badmodule patch actually broke my code...
It took me a while to figure out why, and while at it, I
discovered a blatant stupidity in my own patch: somehow I
completely overlooked the fact that mf.badmodules already
references the modules that did the failed import, so my
module.badimports thing can go. It does the exact same
thing. Almost... and your patch made that clear: the lines
that you patched actually make the badmodules reference lie
(and with your patch it also makes mod.badimports lie ;-):
if module X imports Y.Z, and Z isn't found, it lists Y as an
importer of Y.Z, which isn't true. My any_maybe_missing()
code needs to know whether the tried to failed
import _itself_, because if it did (and failed) it's certain
that the submodule is missing. But if it _didn't_, the
"missing submodule" should be listed as "maybe missing", as
it's most likely just a global in the I'll try
to revise my patch tomorrow, it should become simpler.

In the meantime I think it might indeed be useful to
distinguish between "may be missed" and "must be excluded".
Is that what your "ignore" patch tried to do?
msg41783 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2002-11-28 20:28
Logged In: YES 

Just, I trust your analysis while I don't understand it ;-).
Thinking more about my "ignore" concept shows that it is 
unneeded. Warnings can be better removed after 
modulefinder has been run by examining the data it collects.

To answer your question: "excludes" does not mean 'may be 
missed', it means 'must be excluded' IMO, and no, this was 
not the purpose of my patch.
msg41784 - (view) Author: Just van Rossum (jvr) * (Python triager) Date: 2002-11-29 10:16
Logged In: YES 

Here's a revised patch. The previous version actually was
too lenient: it could list submodules as "maybe missing"
when they were missing for sure. Fixed by tracking
unresolvable starimports.

(Removed the module.badimports attribute of my previous
patch, it now uses mf.badimports appropriately.)

Meta question: are we going to commit fixes to in the freeze directory and do the move when
the doco is done, or do we not commit anything until we're
ready for the move?
msg41785 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2003-04-12 01:36
Logged In: YES 

What's the status of this patch now that has
been moved into the library?
msg41786 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2003-04-14 16:53
Logged In: YES 

I think it can be closed (although the docs are still missing).
Date User Action Args
2022-04-10 16:05:56adminsetgithub: 37535
2002-11-25 18:52:02thellercreate