classification
Title: Allow to restrict ModuleFinder to get "direct" dependencies
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: eric.araujo, eric.snow, ethan.furman, iritkatriel, jvr, mikecmcleod, misc, ncoghlan, nnorwitz, theller
Priority: normal Keywords: easy, patch

Created on 2005-09-08 09:28 by misc_from_metz, last changed 2021-12-01 16:33 by mikecmcleod.

Files
File name Uploaded Description Edit
modulefinder.no_recurse.diff misc_from_metz, 2005-09-08 09:28 Add a parameter to ModuleFinder, to not recurse when searching module.
modulefinder-no-recurse.diff eric.araujo, 2011-07-29 13:53 review
1.html mikecmcleod, 2021-12-01 16:31 bpo-1284670
2.html mikecmcleod, 2021-12-01 16:33 fix non_recursive_import_test_1 and set debug to 0
Messages (15)
msg48725 - (view) Author: Scherer Michael (misc_from_metz) Date: 2005-09-08 09:28
this patch add a parameter to ModuleFinder, to not
recurse when searching module.
This way, someone can compute the modules needed by a
specific file, without having to compute all other
imported module, which may take time for result that
are no t needed. 

With the current system, you cannot create a clean
graph of modules dependancy, since  they are all on the
same level.

This patch do not change the default behavior.


msg48726 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2005-10-01 03:39
Logged In: YES 
user_id=33168

Thomas are you the defacto maintainer of modulefinder?  Any
comments?
msg48727 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2007-06-08 19:48
I have no time or interest in this patch, so unassigning.
msg114551 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-21 19:19
Anyone interested in this or can it be closed?
msg114553 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-21 19:33
Adding jvr and people interested in import machinery, per Misc/maintainers.rst

Michael, can you refresh your patch against the py3k branch? Please also generate a unified diff from the top level of the source distribution, as detailed in http://www.python.org/dev/patches/ The feature seems useful to me and the patch is straightforward. Tests and doc updates are needed; do you want to add them?

Mark, I think that feature requests are never closed just because they’re old; they have to be rejected, implemented or obsoleted. (Thank you for all your triage work.)
msg114645 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-08-22 08:42
The generalist in me is inclined to suggest a "depth" parameter (with depth=1 equivalent to direct dependencies only, and depth = None meaning all dependencies), but I must admit I don't have a concrete use case for the extra generality. So the simpler, recurse/don't recurse approach is probably a better option (building a depth-limited search on top of the recursion flag wouldn't be difficult anyway).

Aside from missing docs and unit test updates, the idea seems sound.
msg114646 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-08-22 08:45
Although I do find it a little concerning that there is no mention of sys.path_hooks or sys.meta_path in the modulefinder source code. I suspect this module only works correctly with vanilla filesystem based imports and can't handle anything imported via PEP 302.
msg124928 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-12-30 18:37
The depth parameter idea sounds like YAGNI, so let’s stay with a recurse boolean :)
msg140602 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-18 16:23
I applied the patch, added a test and found a bug.  Here’s my progress so far; someone can start from it to write more tests and fix the code.
msg141381 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-29 13:53
I modernized modulefinder a bit in 1521d9837d16; here’s a refreshed patch.
msg406579 - (view) Author: mike mcleod (mikecmcleod) * Date: 2021-11-19 12:17
I would like to help with this issue.

I note the history (using Git Dag) of the modulefinder.py shows that it has been modified with changes related here:

Author: Éric Araujo <merwok@netwok.org>  2011-07-28 22:35:29
Committer: Éric Araujo <merwok@netwok.org>  2011-07-28 22:35:29
Parent: 867754e3e3c907f87d1bb0282df628deb392cd62 (merge 11164)
Child:  cee6bb549946bb096aa380a49f3b11daf2d8713e (Fix bug I unwittingly added in 1521d9837d16 (found by Ezio Melotti))
Branches: main, remotes/origin/3.10, remotes/origin/3.6, remotes/origin/3.7, remotes/origin/3.8, remotes/origin/3.9, remotes/origin/bpo-45514/deprecate-importlib-resources-legacy, remotes/origin/buildbot-custom, remotes/origin/main, remotes/upstream/3.10, remotes/upstream/3.6, remotes/upstream/3.7, remotes/upstream/3.8, remotes/upstream/3.9, remotes/upstream/bpo-45514/deprecate-importlib-resources-legacy, remotes/upstream/buildbot-custom, remotes/upstream/main, remotes/upstream/windeepfreeze
Follows: v3.2.1
Precedes: v3.3.0a1

    Modernize modulefinder module and tests a bit.
    
    The tests don’t use an internal distutils function anymore, and use
    regular assertEqual with sorted lists instead of a convoluted manual
    diff.

and

Author: Éric Araujo <merwok@netwok.org>  2011-08-01 14:29:07
Committer: Éric Araujo <merwok@netwok.org>  2011-08-01 14:29:07
Parent: 1e3a68d36b08cd9d59084a37c8cb6c2d911868ce (Modernize modulefinder module and tests a bit.)
Child:  cb66eb0deca1d5cd232f97c76a215ecaab958d30 (Issue #13959: Deprecate imp.get_suffixes() for new attributes on)
Branches: main, remotes/origin/3.10, remotes/origin/3.6, remotes/origin/3.7, remotes/origin/3.8, remotes/origin/3.9, remotes/origin/bpo-45514/deprecate-importlib-resources-legacy, remotes/origin/buildbot-custom, remotes/origin/main, remotes/upstream/3.10, remotes/upstream/3.6, remotes/upstream/3.7, remotes/upstream/3.8, remotes/upstream/3.9, remotes/upstream/bpo-45514/deprecate-importlib-resources-legacy, remotes/upstream/buildbot-custom, remotes/upstream/main, remotes/upstream/windeepfreeze
Follows: v3.2.1
Precedes: v3.3.0a1

    Fix bug I unwittingly added in 1521d9837d16 (found by Ezio Melotti)

And the modulefinder.py file has been subsequently modified by another 28 changes.

So I am wondering if this issue is still relevant?
msg406981 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-11-25 09:04
Mike, from looking at the code the change proposed here is not there, so while the patch may not apply cleanly anymore, the commits you mention do not make this issue irrelevant.
msg407139 - (view) Author: mike mcleod (mikecmcleod) * Date: 2021-11-27 11:10
Ok, I will work on this soon and make further comments.
msg407470 - (view) Author: mike mcleod (mikecmcleod) * Date: 2021-12-01 16:31
I have made the changes as indicated in the diff files. I have tested against the latest from my GitHub copy.
The result is:
./python -m test -j0
== CPython 3.11.0a2+ (heads/main:3a91617590, Dec 1 2021, 15:11:41) [GCC 11.2.0]
== Linux-5.13.0-22-generic-x86_64-with-glibc2.34 little-endian
== cwd: /home/dougal/Documents/cpython/build/test_python_300237æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
...
0:01:34 load avg: 8.77 [221/430/1] test_modulefinder passed
== Tests result: FAILURE ==
409 tests OK.
1 test failed:
    test_embed
20 tests skipped:
    test_curses test_devpoll test_epoll test_ioctl test_kqueue
    test_msilib test_ossaudiodev test_smtpnet test_socketserver
    test_startfile test_tix test_tk test_ttk_guionly test_urllib2net
    test_urllibnet test_winconsoleio test_winreg test_winsound
    test_xmlrpc_net test_zipfile64
Total duration: 5 min 30 sec
Tests result: FAILURE

The first diff is attached. But there was a bug in the code and I will upload in next comment.
msg407471 - (view) Author: mike mcleod (mikecmcleod) * Date: 2021-12-01 16:33
Minor fix after test.
History
Date User Action Args
2021-12-01 16:33:24mikecmcleodsetfiles: + 2.html

messages: + msg407471
2021-12-01 16:31:56mikecmcleodsetfiles: + 1.html

messages: + msg407470
2021-11-27 11:10:48mikecmcleodsetmessages: + msg407139
2021-11-25 09:05:00iritkatrielsetnosy: + iritkatriel
messages: + msg406981
2021-11-19 17:16:06ethan.furmansetnosy: + ethan.furman
2021-11-19 12:17:36mikecmcleodsetnosy: + mikecmcleod
messages: + msg406579
2013-01-25 19:18:16brett.cannonsetnosy: - brett.cannon
2011-07-29 13:53:58eric.araujosetfiles: + modulefinder-no-recurse.diff

messages: + msg141381
2011-07-29 13:52:42eric.araujosetfiles: - modulefinder-no-recurse.diff
2011-07-18 16:32:20eric.snowsetnosy: + eric.snow
2011-07-18 16:23:56eric.araujosetfiles: + modulefinder-no-recurse.diff
nosy: + misc
messages: + msg140602

2010-12-30 18:37:02eric.araujosetversions: + Python 3.3, - Python 3.2
nosy: - misc_from_metz, BreamoreBoy

messages: + msg124928

assignee: eric.araujo
stage: test needed -> patch review
2010-08-22 08:45:21ncoghlansetmessages: + msg114646
2010-08-22 08:42:18ncoghlansetmessages: + msg114645
2010-08-21 19:33:56eric.araujosetnosy: + eric.araujo, jvr, brett.cannon, ncoghlan
messages: + msg114553
2010-08-21 19:19:00BreamoreBoysetnosy: + BreamoreBoy

messages: + msg114551
versions: + Python 3.2, - Python 2.7
2009-04-22 05:08:10ajaksu2setkeywords: + easy
2009-03-31 01:19:22ajaksu2setstage: test needed
type: enhancement
versions: + Python 2.7
2005-09-08 09:28:31misc_from_metzcreate