classification
Title: doctest.testmod(empty_package) raises TypeError in 3.7 (and no errors in 3.6)
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Dutcho, barry, larabrian, xtreak
Priority: normal Keywords: patch, patch

Created on 2019-03-23 11:54 by Dutcho, last changed 2019-11-07 05:49 by larabrian.

Pull Requests
URL Status Linked Edit
PR 12520 open xtreak, 2019-03-24 06:47
PR 12520 open xtreak, 2019-03-24 06:47
Messages (4)
msg338670 - (view) Author: (Dutcho) Date: 2019-03-23 11:54
In recent Python, a directory without __init__.py is also a package, and hence can be imported. When this directory/package is empty, and a doctest.testmod() is executed, the behaviour changed from 3.6 to 3.7, which I didn't find in the "what's new" documentation.

Minimal example:
>>> import doctest, os
>>> os.mkdir('empty_package')
>>> import empty_package
>>> doctest.testmod(empty_package)

Python 3.6.8 on Windows 7 prints
TestResults(failed=0, attempted=0)

Python 3.7.2 on Windows 7 raises below TypeError in doctest
Traceback (most recent call last):
  File "bug_empty_package.py", line 4, in <module>
    print(doctest.testmod(empty_package))
  File "...\Python37\lib\doctest.py", line 1949, in testmod
    for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
  File "...\Python37\lib\doctest.py", line 932, in find
    self._find(tests, obj, name, module, source_lines, globs, {})
  File "...\Python37\lib\doctest.py", line 982, in _find
    test = self._get_test(obj, name, module, globs, source_lines)
  File "...\Python37\lib\doctest.py", line 1063, in _get_test
    if filename[-4:] == ".pyc":
TypeError: 'NoneType' object is not subscriptable
msg338676 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-03-23 12:51
This might be due to changes introduced in a23d30f64bd9c5655cfae7f359d4279c47f6cab3 where __file__ is set to None. Hence the below returns None but before the commit this used to return an AttributeError and print "missing" . This was not handled in doctest causing error. Adding Barry for confirmation.

# foo.py

import empty_package
print(getattr(empty_package, '__file__', 'missing'))

➜  cpython git:(2b5937ec0a) ✗ git checkout a23d30f64bd9c5655cfae7f359d4279c47f6cab3 && make -s -j4 > /dev/null
Previous HEAD position was 2b5937ec0a bpo-32734: Fix asyncio.Lock multiple acquire safety issue (GH-5466) (#5501)
HEAD is now at a23d30f64b bpo-32303 - Consistency fixes for namespace loaders (GH-5481) (#5503)
➜  cpython git:(a23d30f64b) ✗ ./python.exe foo.py
None
➜  cpython git:(a23d30f64b) ✗ git checkout a23d30f64bd9c5655cfae7f359d4279c47f6cab3~1 && make -s -j4 > /dev/null
Previous HEAD position was a23d30f64b bpo-32303 - Consistency fixes for namespace loaders (GH-5481) (#5503)
HEAD is now at 2b5937ec0a bpo-32734: Fix asyncio.Lock multiple acquire safety issue (GH-5466) (#5501)
➜  cpython git:(2b5937ec0a) ✗ ./python.exe foo.py
missing

There is a unittest with empty package and it's just that doctest.testmod(mod) was not called on the empty package causing this not to be found. A simple patch would be to check for None where None is a valid value for DocTest. Another possible fix would be to have module.__name__ for None but since this is for empty packages I assume there won't be any doctest to parse in DocTest constructor.

diff --git a/Lib/doctest.py b/Lib/doctest.py
index 79d91a040c..e97555ed2f 100644
--- a/Lib/doctest.py
+++ b/Lib/doctest.py
@@ -1060,7 +1060,7 @@ class DocTestFinder:
             filename = None
         else:
             filename = getattr(module, '__file__', module.__name__)
-            if filename[-4:] == ".pyc":
+            if filename and filename[-4:] == ".pyc":
                 filename = filename[:-1]
         return self._parser.get_doctest(docstring, globs, name,
                                         filename, lineno)
diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py
index f1013f2572..b99f2aea2f 100644
--- a/Lib/test/test_doctest.py
+++ b/Lib/test/test_doctest.py
@@ -699,6 +699,7 @@ class TestDocTestFinder(unittest.TestCase):
                 support.forget(pkg_name)
                 sys.path.pop()
             assert doctest.DocTestFinder().find(mod) == []
+            doctest.testmod(mod)


 def test_DocTestParser(): r"""

Without patch the line doctest.testmod(mod) in unittest fails as below with TypeError and with patch the tests pass

$ ./python.exe Lib/test/test_doctest.py
doctest (doctest) ... 66 tests with zero failures
doctest (test.test_doctest) ... 516 tests with zero failures
test_empty_namespace_package (__main__.TestDocTestFinder) ... ERROR

======================================================================
ERROR: test_empty_namespace_package (__main__.TestDocTestFinder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_doctest.py", line 702, in test_empty_namespace_package
    doctest.testmod(mod)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1947, in testmod
    for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 932, in find
    self._find(tests, obj, name, module, source_lines, globs, {})
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 982, in _find
    test = self._get_test(obj, name, module, globs, source_lines)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1063, in _get_test
    if filename[-4:] == ".pyc":
TypeError: 'NoneType' object is not subscriptable

----------------------------------------------------------------------

Ran 1 test in 0.027s

FAILED (errors=1)
Traceback (most recent call last):
  File "Lib/test/test_doctest.py", line 3032, in <module>
    test_main()
  File "Lib/test/test_doctest.py", line 3015, in test_main
    support.run_unittest(__name__)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/support/__init__.py", line 2064, in run_unittest
    _run_suite(suite)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/support/__init__.py", line 1983, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_doctest.py", line 702, in test_empty_namespace_package
    doctest.testmod(mod)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1947, in testmod
    for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 932, in find
    self._find(tests, obj, name, module, source_lines, globs, {})
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 982, in _find
    test = self._get_test(obj, name, module, globs, source_lines)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1063, in _get_test
    if filename[-4:] == ".pyc":
TypeError: 'NoneType' object is not subscriptable
msg338679 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-03-23 14:47
So this was backported to 3.6 too with but caused similar issues reported in issue32872 to be reverted back.
msg356169 - (view) Author: larabrian (larabrian) Date: 2019-11-07 05:49
A subscriptable object is any object that implements the __getitem__ special method (think lists, dictionaries). It is an object that records the operations done to it and it can store them as a "script" which can be replayed. You are trying to subscript an object which you think is a list or dict, but actually is None. NoneType is the type of the None object which represents a lack of value, for example, a function that does not explicitly return a value will return None. 'NoneType' object is not subscriptable is the one thrown by python when you use the square bracket notation object[key] where an object doesn't define the __getitem__ method . You might have noticed that the method sort() that only modify the list have no return value printed – they return the default None. This is a design principle for all mutable data structures in Python. http://net-informations.com/python/err/nonetype.htm
History
Date User Action Args
2019-11-07 05:49:44larabriansetnosy: + larabrian
messages: + msg356169
2019-03-24 06:47:38xtreaksetkeywords: + patch
stage: patch review
pull_requests: + pull_request12472
2019-03-24 06:47:38xtreaksetkeywords: + patch
stage: (no value)
pull_requests: + pull_request12471
2019-03-23 14:47:51xtreaksetmessages: + msg338679
2019-03-23 12:51:02xtreaksetnosy: + barry, xtreak

messages: + msg338676
versions: + Python 3.8
2019-03-23 11:54:24Dutchocreate