classification
Title: Test pkgutil.walk_packages in test_pkgutil instead of test_runpy
Type: enhancement Stage: needs patch
Components: Tests Versions: Python 3.5
process
Status: open Resolution:
Dependencies: 15343 15376 Superseder:
Assigned To: Nosy List: BreamoreBoy, brett.cannon, chris.jerdonek, eric.snow, ncoghlan
Priority: normal Keywords:

Created on 2012-07-15 09:54 by ncoghlan, last changed 2015-06-28 06:03 by ncoghlan.

Messages (11)
msg165516 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-15 09:54
Created to record the egregious hack of relying on the test_runpy infrastructure in order to test pkgutil.walk_packages.

It gets the job done, but is a really messy way of going about it. Worth cleaning up by factoring the support code out to a helper module when there isn't a release deadline looming.
msg165519 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-15 11:25
Also, a second test case should be added to cover the zipimporter.zipimporter component.
msg165540 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-15 17:33
Such helper functionality could also be used in the tests of unittest.TestLoader.loadTestsFromName().  See, for example, the tests proposed for issue 7559.
msg165616 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-16 14:31
The walk_packages tests should also be enhanced to ensure correctly handling of namespace packages
msg165617 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-16 14:33
Note that the problem described in #14787 will limit the scope of what can be tested.
msg165682 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-17 07:05
I'd like to work on this if that's okay.  I've already written test cases for issue 7559 and issue 15299 that required creating temporary packages, so I understand the need.  And I'd also like to create test cases for issue 14787, which should really be done after this.

I think this should be split into two parts though: (1) refactor the create-package code in test_runpy into a separate helper module, and (2) move the refactored walk_package tests from test_runpy into test_pkgutil (so it's easier to see what has changed).

I can create a new issue for (1).
msg165686 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-17 08:11
That sounds good, and thanks for working on this - if we can get the tests cleaned up for the 3.3 release, that will make ongoing maintenance a lot easier.
msg221983 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-30 21:16
Has anyone made any progress with this issue or others referenced like #7559 or #14787 ?  Regardless I'd like to help out directly if possible as I'm suffering from an acute case of triagitis :-)
msg222001 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-07-01 02:09
Dropped down my priority list (it's a nice to have to make the code base a
bit easier to work on, rather than something essential).
msg222055 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-01 18:36
Hardly a good start :-(  From the IDLE shell.

Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> from pkgutil import walk_packages
>>> packages = list(walk_packages())
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    packages = list(walk_packages())
  File "C:\Python34\lib\pkgutil.py", line 86, in walk_packages
    __import__(name)
  File "C:\Python34\lib\site-packages\pip\__init__.py", line 9, in <module>
    from pip.log import logger
  File "C:\Python34\lib\site-packages\pip\log.py", line 8, in <module>
    from pip import backwardcompat
  File "C:\Python34\lib\site-packages\pip\backwardcompat\__init__.py", line 27, in <module>
    console_encoding = sys.__stdout__.encoding
AttributeError: 'NoneType' object has no attribute 'encoding'

Then from the command line.

C:\Users\Mark\MyPython>type wptest.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from pkgutil import walk_packages

packages = list(walk_packages())

C:\Users\Mark\MyPython>wptest.py
c:\python34\lib\site-packages\wx\core.py:22: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
C:\Users\Mark\MyPython\wptest.py in <module>()
      4 from pkgutil import walk_packages
      5
----> 6 packages = list(walk_packages())

c:\python34\lib\pkgutil.py in walk_packages(path, prefix, onerror)
     99                 path = [p for p in path if not seen(p)]
    100
--> 101                 yield from walk_packages(path, name+'.', onerror)
    102
    103

c:\python34\lib\pkgutil.py in walk_packages(path, prefix, onerror)
     99                 path = [p for p in path if not seen(p)]
    100
--> 101                 yield from walk_packages(path, name+'.', onerror)
    102
    103

c:\python34\lib\pkgutil.py in walk_packages(path, prefix, onerror)
     99                 path = [p for p in path if not seen(p)]
    100
--> 101                 yield from walk_packages(path, name+'.', onerror)
    102
    103

c:\python34\lib\pkgutil.py in walk_packages(path, prefix, onerror)
     99                 path = [p for p in path if not seen(p)]
    100
--> 101                 yield from walk_packages(path, name+'.', onerror)
    102
    103

c:\python34\lib\pkgutil.py in walk_packages(path, prefix, onerror)
     84         if ispkg:
     85             try:
---> 86                 __import__(name)
     87             except ImportError:
     88                 if onerror is not None:

c:\python34\lib\site-packages\wx\lib\pubsub\core\arg1\__init__.py in <module>()
     14
     15 msg = 'Should not import this directly, used by pubsub.core if applicable'
---> 16 raise RuntimeError(msg)

RuntimeError: Should not import this directly, used by pubsub.core if applicable

But then.

C:\Users\Mark\MyPython>cd c:\cpython\pcbuild

c:\cpython\PCbuild>python_d.exe C:\Users\Mark\MyPython\wptest.py

c:\cpython\PCbuild>

No problems above!!!

Given Brett's comment in msg165456 "But if we do this then I want to deprecate pkgutil in Python 3.4" I don't believe that this is worth pursuing, or have I missed something?
msg222115 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-07-02 18:29
I got overruled on the deprecation idea due to utility code like walk_packages that pkgutil contains.
History
Date User Action Args
2015-06-28 06:03:55ncoghlansetassignee: ncoghlan ->
2014-07-02 18:29:07brett.cannonsetmessages: + msg222115
2014-07-01 18:36:25BreamoreBoysetnosy: + brett.cannon
messages: + msg222055
2014-07-01 02:09:36ncoghlansetmessages: + msg222001
2014-06-30 21:16:09BreamoreBoysetnosy: + BreamoreBoy

messages: + msg221983
versions: + Python 3.5, - Python 3.3
2012-11-13 05:02:39eric.snowsetnosy: + eric.snow
2012-07-30 10:28:14ncoghlansetassignee: ncoghlan
2012-07-17 08:11:57ncoghlansetmessages: + msg165686
2012-07-17 07:11:43chris.jerdoneksetdependencies: + Refactor the test_runpy walk_package support code into a common location
2012-07-17 07:05:41chris.jerdoneksetmessages: + msg165682
2012-07-16 14:33:48ncoghlansetmessages: + msg165617
2012-07-16 14:31:47ncoghlansetmessages: + msg165616
2012-07-15 17:33:59chris.jerdoneksetmessages: + msg165540
2012-07-15 17:21:10chris.jerdoneksetnosy: + chris.jerdonek
2012-07-15 11:25:50ncoghlansetpriority: low -> normal
2012-07-15 11:25:42ncoghlansetmessages: + msg165519
2012-07-15 10:12:10ncoghlansetdependencies: + "pydoc -w <package>" writes out page with empty "Package Contents" section
2012-07-15 09:54:30ncoghlancreate