Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doctest.testmod(empty_package) raises TypeError in 3.7 (and no errors in 3.6) #80587

Closed
Dutcho mannequin opened this issue Mar 23, 2019 · 7 comments
Closed

doctest.testmod(empty_package) raises TypeError in 3.7 (and no errors in 3.6) #80587

Dutcho mannequin opened this issue Mar 23, 2019 · 7 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@Dutcho
Copy link
Mannequin

Dutcho mannequin commented Mar 23, 2019

BPO 36406
Nosy @warsaw, @brettcannon, @Dutcho, @tirkarthi
PRs
  • bpo-36406: Handle namespace packages in doctest #12520
  • bpo-36406: Handle namespace packages in doctest #12520
  • [3.8] bpo-36406: Handle namespace packages in doctest (GH-12520) #17591
  • [3.7] bpo-36406: Handle namespace packages in doctest (GH-12520) #17592
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-12-13.23:38:43.633>
    created_at = <Date 2019-03-23.11:54:24.504>
    labels = ['3.7', '3.8', 'type-bug', 'library']
    title = 'doctest.testmod(empty_package) raises TypeError in 3.7 (and no errors in 3.6)'
    updated_at = <Date 2019-12-13.23:38:43.632>
    user = 'https://github.com/Dutcho'

    bugs.python.org fields:

    activity = <Date 2019-12-13.23:38:43.632>
    actor = 'xtreak'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-12-13.23:38:43.633>
    closer = 'xtreak'
    components = ['Library (Lib)']
    creation = <Date 2019-03-23.11:54:24.504>
    creator = 'Dutcho'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36406
    keywords = ['patch', 'patch']
    message_count = 7.0
    messages = ['338670', '338676', '338679', '356169', '358348', '358365', '358366']
    nosy_count = 5.0
    nosy_names = ['barry', 'brett.cannon', 'Dutcho', 'xtreak', 'larabrian']
    pr_nums = ['12520', '12520', '17591', '17592']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue36406'
    versions = ['Python 3.7', 'Python 3.8']

    @Dutcho
    Copy link
    Mannequin Author

    Dutcho mannequin commented Mar 23, 2019

    In recent Python, a directory without __init__.py is also a package, and hence can be imported. When this directory/package is empty, and a doctest.testmod() is executed, the behaviour changed from 3.6 to 3.7, which I didn't find in the "what's new" documentation.

    Minimal example:
    >>> import doctest, os
    >>> os.mkdir('empty_package')
    >>> import empty_package
    >>> doctest.testmod(empty_package)

    Python 3.6.8 on Windows 7 prints
    TestResults(failed=0, attempted=0)

    Python 3.7.2 on Windows 7 raises below TypeError in doctest
    Traceback (most recent call last):
      File "bug_empty_package.py", line 4, in <module>
        print(doctest.testmod(empty_package))
      File "...\Python37\lib\doctest.py", line 1949, in testmod
        for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
      File "...\Python37\lib\doctest.py", line 932, in find
        self._find(tests, obj, name, module, source_lines, globs, {})
      File "...\Python37\lib\doctest.py", line 982, in _find
        test = self._get_test(obj, name, module, globs, source_lines)
      File "...\Python37\lib\doctest.py", line 1063, in _get_test
        if filename[-4:] == ".pyc":
    TypeError: 'NoneType' object is not subscriptable

    @Dutcho Dutcho mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Mar 23, 2019
    @tirkarthi
    Copy link
    Member

    This might be due to changes introduced in a23d30f where __file__ is set to None. Hence the below returns None but before the commit this used to return an AttributeError and print "missing" . This was not handled in doctest causing error. Adding Barry for confirmation.

    # foo.py

    import empty_package
    print(getattr(empty_package, '__file__', 'missing'))

    ➜ cpython git:(2b5937e) ✗ git checkout a23d30f && make -s -j4 > /dev/null
    Previous HEAD position was 2b5937e bpo-32734: Fix asyncio.Lock multiple acquire safety issue (GH-5466) (bpo-5501)
    HEAD is now at a23d30f bpo-32303 - Consistency fixes for namespace loaders (GH-5481) (bpo-5503)
    ➜ cpython git:(a23d30f) ✗ ./python.exe foo.py
    None
    ➜ cpython git:(a23d30f) ✗ git checkout a23d30f~1 && make -s -j4 > /dev/null
    Previous HEAD position was a23d30f bpo-32303 - Consistency fixes for namespace loaders (GH-5481) (bpo-5503)
    HEAD is now at 2b5937e bpo-32734: Fix asyncio.Lock multiple acquire safety issue (GH-5466) (bpo-5501)
    ➜ cpython git:(2b5937e) ✗ ./python.exe foo.py
    missing

    There is a unittest with empty package and it's just that doctest.testmod(mod) was not called on the empty package causing this not to be found. A simple patch would be to check for None where None is a valid value for DocTest. Another possible fix would be to have module.__name__ for None but since this is for empty packages I assume there won't be any doctest to parse in DocTest constructor.

    diff --git a/Lib/doctest.py b/Lib/doctest.py
    index 79d91a040c..e97555ed2f 100644
    --- a/Lib/doctest.py
    +++ b/Lib/doctest.py
    @@ -1060,7 +1060,7 @@ class DocTestFinder:
                 filename = None
             else:
                 filename = getattr(module, '__file__', module.__name__)
    -            if filename[-4:] == ".pyc":
    +            if filename and filename[-4:] == ".pyc":
                     filename = filename[:-1]
             return self._parser.get_doctest(docstring, globs, name,
                                             filename, lineno)
    diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py
    index f1013f2572..b99f2aea2f 100644
    --- a/Lib/test/test_doctest.py
    +++ b/Lib/test/test_doctest.py
    @@ -699,6 +699,7 @@ class TestDocTestFinder(unittest.TestCase):
                     support.forget(pkg_name)
                     sys.path.pop()
                 assert doctest.DocTestFinder().find(mod) == []
    +            doctest.testmod(mod)
     def test_DocTestParser(): r"""

    Without patch the line doctest.testmod(mod) in unittest fails as below with TypeError and with patch the tests pass

    $ ./python.exe Lib/test/test_doctest.py
    doctest (doctest) ... 66 tests with zero failures
    doctest (test.test_doctest) ... 516 tests with zero failures
    test_empty_namespace_package (__main__.TestDocTestFinder) ... ERROR

    ======================================================================
    ERROR: test_empty_namespace_package (main.TestDocTestFinder)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "Lib/test/test_doctest.py", line 702, in test_empty_namespace_package
        doctest.testmod(mod)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1947, in testmod
        for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 932, in find
        self._find(tests, obj, name, module, source_lines, globs, {})
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 982, in _find
        test = self._get_test(obj, name, module, globs, source_lines)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1063, in _get_test
        if filename[-4:] == ".pyc":
    TypeError: 'NoneType' object is not subscriptable

    Ran 1 test in 0.027s

    FAILED (errors=1)
    Traceback (most recent call last):
      File "Lib/test/test_doctest.py", line 3032, in <module>
        test_main()
      File "Lib/test/test_doctest.py", line 3015, in test_main
        support.run_unittest(__name__)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/support/__init__.py", line 2064, in run_unittest
        _run_suite(suite)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/support/__init__.py", line 1983, in _run_suite
        raise TestFailed(err)
    test.support.TestFailed: Traceback (most recent call last):
      File "Lib/test/test_doctest.py", line 702, in test_empty_namespace_package
        doctest.testmod(mod)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1947, in testmod
        for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 932, in find
        self._find(tests, obj, name, module, source_lines, globs, {})
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 982, in _find
        test = self._get_test(obj, name, module, globs, source_lines)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py", line 1063, in _get_test
        if filename[-4:] == ".pyc":
    TypeError: 'NoneType' object is not subscriptable

    @tirkarthi tirkarthi added the 3.8 only security fixes label Mar 23, 2019
    @tirkarthi
    Copy link
    Member

    So this was backported to 3.6 too with but caused similar issues reported in bpo-32872 to be reverted back.

    @larabrian
    Copy link
    Mannequin

    larabrian mannequin commented Nov 7, 2019

    A subscriptable object is any object that implements the __getitem__ special method (think lists, dictionaries). It is an object that records the operations done to it and it can store them as a "script" which can be replayed. You are trying to subscript an object which you think is a list or dict, but actually is None. NoneType is the type of the None object which represents a lack of value, for example, a function that does not explicitly return a value will return None. 'NoneType' object is not subscriptable is the one thrown by python when you use the square bracket notation object[key] where an object doesn't define the __getitem__ method . You might have noticed that the method sort() that only modify the list have no return value printed – they return the default None. This is a design principle for all mutable data structures in Python. http://net-informations.com/python/err/nonetype.htm

    @brettcannon
    Copy link
    Member

    New changeset 8289e27 by Brett Cannon (Xtreak) in branch 'master':
    bpo-36406: Handle namespace packages in doctest (GH-12520)
    8289e27

    @brettcannon
    Copy link
    Member

    New changeset d321203 by Brett Cannon (Miss Islington (bot)) in branch '3.7':
    bpo-36406: Handle namespace packages in doctest (GH-12520) (GH-17592)
    d321203

    @brettcannon
    Copy link
    Member

    New changeset aa74a53 by Brett Cannon (Miss Islington (bot)) in branch '3.8':
    bpo-36406: Handle namespace packages in doctest (GH-12520) (GH-17591)
    aa74a53

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants