classification
Title: test_pydoc loads all Python modules
Type: Stage: resolved
Components: Tests Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, eric.araujo, eric.snow, haypo, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2011-05-04 08:29 by haypo, last changed 2013-05-19 13:52 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
test_pydoc_imports.patch pitrou, 2013-05-18 20:35 review
Messages (9)
msg135100 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-04 08:29
PydocUrlHandlerTest.test_url_requests() of test_doc checks "search?key=pydoc" URL. This operation has to load all Python modules to get their documentation. The problem is that loading all modules can have border effects with other tests. The most recent example: test_signal fails if it runs after test_pydoc. test_pydoc imports _tkinter and _tkinter loads the Tcl library which creates a thread waiting events in select(). The problem is that test_signal supposes that there is only one running thread.

Can we run tests loading all modules in a subprocess?

I have a fix to test_signal, so this bug is not really blocker, but it's better if a test doesn't touch the environment too much.

---

To see which modules are imported by pydoc, you can use:
import pydoc; pydoc.ModuleScanner().run(lambda *args: 0, 'XXX')

Result on my host (using python -v):

__future__
_ast
_bz2
_compat_pickle
_ctypes
_curses
_datetime
_hashlib
_json
_multiprocessing
_pickle
_posixsubprocess
_random
_socket
_sqlite3
_struct
_symtable
_tkinter
array
atexit
bdb
binascii
bz2
cmd
code
codeop
concurrent
concurrent
concurrent.futures
concurrent.futures
concurrent.futures._base
concurrent.futures.process
concurrent.futures.thread
contextlib
copy
ctypes
ctypes
ctypes._endian
ctypes.macholib
ctypes.macholib
ctypes.test
ctypes.test
curses
curses
curses.wrapper
datetime
dbm
dbm
difflib
distutils
distutils
distutils.command
distutils.command
distutils.tests
distutils.tests
doctest
email
email
email.mime
email.mime
encodings.cp1252
encodings.idna
encodings.koi8_r
faulthandler
fcntl
fnmatch
gc
getopt
gettext
grp
hashlib
html
html
http
http
idlelib
idlelib
importlib
importlib
importlib._bootstrap
importlib.test
importlib.test
importlib.test.builtin
importlib.test.builtin
importlib.test.extension
importlib.test.extension
importlib.test.frozen
importlib.test.frozen
importlib.test.import_
importlib.test.import_
importlib.test.source
importlib.test.source
json
json
json.decoder
json.encoder
json.scanner
lib2to3
lib2to3
lib2to3.btm_matcher
lib2to3.btm_utils
lib2to3.fixer_base
lib2to3.fixer_util
lib2to3.fixes
lib2to3.fixes
lib2to3.fixes.fix_imports
lib2to3.fixes.fix_imports2
lib2to3.fixes.fix_urllib
lib2to3.main
lib2to3.patcomp
lib2to3.pgen2
lib2to3.pgen2
lib2to3.pgen2.driver
lib2to3.pgen2.grammar
lib2to3.pgen2.literals
lib2to3.pgen2.parse
lib2to3.pgen2.pgen
lib2to3.pgen2.token
lib2to3.pgen2.tokenize
lib2to3.pygram
lib2to3.pytree
lib2to3.refactor
lib2to3.tests
lib2to3.tests
lib2to3.tests.support
lib2to3.tests.test_all_fixers
lib2to3.tests.test_fixers
lib2to3.tests.test_main
lib2to3.tests.test_parser
lib2to3.tests.test_pytree
lib2to3.tests.test_refactor
lib2to3.tests.test_util
logging
logging
logging.handlers
marshal
math
msilib
msilib
multiprocessing
multiprocessing
multiprocessing.dummy
multiprocessing.dummy
multiprocessing.dummy.connection
multiprocessing.forking
multiprocessing.process
multiprocessing.queues
multiprocessing.synchronize
multiprocessing.util
myfixes
myfixes
optparse
pdb
pickle
pprint
pwd
pydoc_data
pydoc_data
queue
random
select
shutil
socket
sqlite3
sqlite3
sqlite3.dbapi2
sqlite3.test
sqlite3.test
stringprep
struct
subprocess
tarfile
tempfile
test
test
test.encoded_modules
test.encoded_modules
test.json_tests
test.json_tests
test.leakers
test.leakers
test.support
test.test_email
test.test_email
test.tracedmodules
test.tracedmodules
textwrap
threading
tkinter
tkinter
tkinter.constants
tkinter.test
tkinter.test
tkinter.test.test_tkinter
tkinter.test.test_tkinter
tkinter.test.test_ttk
tkinter.test.test_ttk
turtledemo
turtledemo
unicodedata
unittest
unittest
unittest.case
unittest.loader
unittest.main
unittest.result
unittest.runner
unittest.signals
unittest.suite
unittest.test
unittest.test
unittest.util
urllib
urllib
wsgiref
wsgiref
xml
xml
xml.dom
xml.dom
xml.dom.domreg
xml.dom.minicompat
xml.etree
xml.etree
xml.parsers
xml.parsers
xml.sax
xml.sax
xml.sax._exceptions
xml.sax.handler
xml.sax.xmlreader
xmlrpc
xmlrpc
xxsubtype
msg135101 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-04 08:43
By the way, it is NOT POSSIBLE TO UNLOAD modules implemented in C, even if there is no more Python reference to the module.
msg135110 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-04 10:35
See also #11998 for test_signal and _tkinter issue (test_pydoc + test_signal).
msg135147 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-05-04 18:39
It might be possible to use a part of patch for issue #1674555.
A newer version of that patch can be found in:
http://overlays.gentoo.org/svn/proj/python/overlays/python/dev-lang/python/files/3.3-20110417/23_all_tests_environment.patch
msg135312 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-05-06 16:02
Isn’t the root problem that pydoc uses import rather than imp or PEP 302 things?  We could change that.
msg135418 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-07 08:34
> Isn’t the root problem that pydoc uses import rather than imp or PEP 302 things?  We could change that.

It's possible to get the documentation of some module without loading them: pydoc already supports get_source() method of a module loader.

But my concern is more about C modules: you have to load them to get their documentation, but you can never a unload C modules (you have to exit Python).
msg189547 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-18 20:35
Here is a patch.
msg189589 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-19 13:46
New changeset 58ace614df6b by Antoine Pitrou in branch '3.3':
Issue #11995: test_pydoc doesn't import all sys.path modules anymore.
http://hg.python.org/cpython/rev/58ace614df6b

New changeset 13d817cb72e7 by Antoine Pitrou in branch 'default':
Issue #11995: test_pydoc doesn't import all sys.path modules anymore.
http://hg.python.org/cpython/rev/13d817cb72e7
msg189590 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-19 13:51
New changeset 68d2337688c1 by Antoine Pitrou in branch '2.7':
Issue #11995: test_pydoc doesn't import all sys.path modules anymore.
http://hg.python.org/cpython/rev/68d2337688c1
History
Date User Action Args
2013-05-19 13:52:12pitrousetstatus: open -> closed
stage: resolved
resolution: fixed
versions: - Python 3.2
2013-05-19 13:51:21python-devsetmessages: + msg189590
2013-05-19 13:46:46python-devsetnosy: + python-dev
messages: + msg189589
2013-05-18 20:35:52pitrousetfiles: + test_pydoc_imports.patch
versions: + Python 3.4
nosy: + pitrou

messages: + msg189547
2013-01-26 08:44:29eric.snowsetnosy: + eric.snow
2013-01-03 01:44:44Arfreversetfiles: - python-3.3-issue_11995.patch
2013-01-03 01:44:41Arfreversetfiles: - python-3.2-issue_11995.patch
2013-01-03 01:44:36Arfreversetfiles: - python-2.7-issue_11995.patch
2011-05-26 22:06:02Arfreversetversions: + Python 2.7, Python 3.2
2011-05-26 22:05:47Arfreversetfiles: + python-3.3-issue_11995.patch
2011-05-26 22:05:26Arfreversetfiles: + python-3.2-issue_11995.patch
2011-05-26 22:05:02Arfreversetfiles: + python-2.7-issue_11995.patch
keywords: + patch
2011-05-07 08:34:56hayposetmessages: + msg135418
2011-05-06 16:02:31eric.araujosetnosy: + eric.araujo
messages: + msg135312
2011-05-04 18:39:41Arfreversetmessages: + msg135147
2011-05-04 17:53:06Arfreversetnosy: + Arfrever
2011-05-04 10:35:34hayposetmessages: + msg135110
2011-05-04 08:43:57hayposetmessages: + msg135101
2011-05-04 08:29:26haypocreate