I originally hit upon this with issue36777 where I have used support.script_helper.make_script which calls importlib.invalidate_caches and then trying to use __import__ on an empty folder causes reference leak. I tried 3.8 to 3.5 and it exists on each version. A sample script as below. I tried using try finally instead of DirsOnSysPath in doubt and it still causes leak. I couldn't find any issues on search and let me know if I am using something in an incorrect manner.

import importlib
import unittest
import os, sys
import os.path
from test import support

def test_importlib_cache():

    with support.temp_dir() as path:
        dirname, basename = os.path.split(path)
        os.mkdir(os.path.join(path, 'test2'))
        importlib.invalidate_caches()

        with support.DirsOnSysPath(dirname):
            __import__("{basename}.test2".format(basename=basename))


class Tests(unittest.TestCase):

    def test_bug(self):
        for _ in range(10):
            test_importlib_cache()

➜  cpython git:(master) ✗ ./python.exe -m test -R 3:3 test_import_bug
Run tests sequentially
0:00:00 load avg: 1.56 [1/1] test_import_bug
beginning 6 repetitions
123456
......
test_import_bug leaked [980, 980, 980] references, sum=2940
test_import_bug leaked [370, 370, 370] memory blocks, sum=1110
test_import_bug failed

== Tests result: FAILURE ==

1 test failed:
    test_import_bug

Total duration: 1 sec 529 ms
Tests result: FAILURE


I also tried __import__('test1.test2') instead of __import__("{basename}.test2".format(basename=basename)) and the program doesn't cause reference leak. Moving importlib.invalidate_caches() above support.temp_dir() also causes leak so I guess it's not something to do with temporary directories that are cleaned up after tests.

➜  cpython git:(master) ✗ mkdir -p test1/test2
➜  cpython git:(master) ✗ ./python.exe -m test -R 3:3 test_import_bug
Run tests sequentially
0:00:00 load avg: 1.97 [1/1] test_import_bug
beginning 6 repetitions
123456
......
test_import_bug passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 557 ms
Tests result: SUCCESS

Interesting, I used tracemalloc to see if it helps and it gave me a line in Lib/tempfile.py . Supplying path to support.temp_dir(path="/tmp/") causes random directory code was not to be hit and there was no memory leak. So I removed the _Random() initialization in _RandomNameSequence in Lib/tempfile.py and instead of self._rng.choice I used random.choice and still had the leak. 

I replaced the code to generate random letters letters = [choose(c) for dummy in range(8)] where choose is random.choice with c = "a" and the memory leak stopped. I tried below combinations at line [0] and ran test to see if the memory leaks. I also tried -R 10:10 just to make sure my limits are higher enough. Using random.shuffle on a set of characters also causes leak. I am not sure why a combination of importlib.invalidate_caches, support.temp_dir using tempfile and __import__ causes these leaks or perhaps I am debugging or using huntrleaks in an incorrect manner.

# No leak

letters = [choose("a") for dummy in range(8)]
letters = ["a" for dummy in range(8)]
letters = [choose(self.characters[0]) for dummy in range(8)]

# Memory leak

letters = [choose("ab") for dummy in range(8)]
letters = [choose(self.characters[:]) for dummy in range(8)]
letters = [choose(list(self.characters)) for dummy in range(8)]

# Below also leaks

characters = list("abcde")  # list("abcd") doesn't leak
self.rng.shuffle(characters)
letters = characters[:8]


from unittest import TestCase
import tracemalloc
import sys
import os
from test import support

def test_importlib_cache_tempdir():

    import importlib
    importlib.invalidate_caches()

    with support.temp_dir() as path:   # with support.temp_dir(path="/tmp") as path: (no leak)
        dirname = os.path.dirname(path)
        basename = os.path.basename(path)
        os.mkdir(os.path.join(path, 'test2'))

        with support.DirsOnSysPath(dirname):
            __import__(f"{basename}.test2".format(basename=basename))


class Tests(TestCase):

    def test_bug(self):
        tracemalloc.start()

        for _ in range(10):
            test_importlib_cache_tempdir()

        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics('traceback')

        print("[ Top 10 ]")
        for stat in top_stats[:10]:
            for line in stat.traceback.format():
                print(line)


$ ./python.exe -m test -R 3:3 test_import_bug_tempdir
Run tests sequentially
0:00:00 load avg: 2.55 [1/1] test_import_bug_tempdir
beginning 6 repetitions
123456
[ Top 10 ]
  File "<frozen importlib._bootstrap_external>", line 1486
  File "<frozen importlib._bootstrap_external>", line 1461
  File "<frozen importlib._bootstrap_external>", line 1469
  File "<frozen importlib._bootstrap>", line 683
  File "<frozen importlib._bootstrap>", line 509
  File "<frozen importlib._bootstrap_external>", line 1378
  File "<frozen importlib._bootstrap>", line 344
  File "<frozen importlib._bootstrap>", line 36
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/tempfile.py", line 136
    self._rng = _Random()
  File "<frozen importlib._bootstrap_external>", line 1342
.[ Top 10 ]
  File "<frozen importlib._bootstrap_external>", line 1486
  File "<frozen importlib._bootstrap_external>", line 1461
  File "<frozen importlib._bootstrap_external>", line 1469
  File "<frozen importlib._bootstrap>", line 683
  File "<frozen importlib._bootstrap>", line 509
  File "<frozen importlib._bootstrap>", line 344
  File "<frozen importlib._bootstrap>", line 36
  File "<frozen importlib._bootstrap_external>", line 64
  File "<frozen importlib._bootstrap_external>", line 1342
  File "<frozen importlib._bootstrap_external>", line 1378
.[ Top 10 ]
  File "<frozen importlib._bootstrap_external>", line 1486
  File "<frozen importlib._bootstrap_external>", line 1461
  File "<frozen importlib._bootstrap_external>", line 1469
  File "<frozen importlib._bootstrap>", line 683
  File "<frozen importlib._bootstrap>", line 509
  File "<frozen importlib._bootstrap>", line 344
  File "<frozen importlib._bootstrap>", line 36
  File "<frozen importlib._bootstrap_external>", line 64
  File "<frozen importlib._bootstrap_external>", line 1342
  File "<frozen importlib._bootstrap_external>", line 1132
.[ Top 10 ]
  File "<frozen importlib._bootstrap_external>", line 1486
  File "<frozen importlib._bootstrap_external>", line 1461
  File "<frozen importlib._bootstrap_external>", line 1469
  File "<frozen importlib._bootstrap>", line 509
  File "<frozen importlib._bootstrap>", line 683
  File "<frozen importlib._bootstrap>", line 344
  File "<frozen importlib._bootstrap>", line 36
  File "<frozen importlib._bootstrap_external>", line 64
  File "<frozen importlib._bootstrap_external>", line 1342
  File "<frozen importlib._bootstrap_external>", line 1132
.[ Top 10 ]
  File "<frozen importlib._bootstrap_external>", line 1486
  File "<frozen importlib._bootstrap_external>", line 1461
  File "<frozen importlib._bootstrap_external>", line 1469
  File "<frozen importlib._bootstrap>", line 509
  File "<frozen importlib._bootstrap>", line 683
  File "<frozen importlib._bootstrap>", line 344
  File "<frozen importlib._bootstrap>", line 36
  File "<frozen importlib._bootstrap_external>", line 64
  File "<frozen importlib._bootstrap_external>", line 1342
  File "<frozen importlib._bootstrap_external>", line 1132
.[ Top 10 ]
  File "<frozen importlib._bootstrap_external>", line 1486
  File "<frozen importlib._bootstrap_external>", line 1461
  File "<frozen importlib._bootstrap_external>", line 1469
  File "<frozen importlib._bootstrap>", line 509
  File "<frozen importlib._bootstrap>", line 683
  File "<frozen importlib._bootstrap>", line 344
  File "<frozen importlib._bootstrap>", line 36
  File "<frozen importlib._bootstrap_external>", line 64
  File "<frozen importlib._bootstrap_external>", line 1342
  File "<frozen importlib._bootstrap_external>", line 1132
.
test_import_bug_tempdir leaked [980, 980, 980] references, sum=2940
test_import_bug_tempdir leaked [370, 370, 370] memory blocks, sum=1110
test_import_bug_tempdir failed

== Tests result: FAILURE ==

1 test failed:
    test_import_bug_tempdir

Total duration: 3 sec 254 ms
Tests result: FAILURE

So, I dug into this here at the PyCon19 sprints and as far as I can see there is no actual leak.

What you are seeing in your code example is from the state, that is kept between successive run of your import. All the cases you reported as not leaking generate a fixed tempdir. However, if the tempdir is random (or at least differs between runs) two new modules are added to sys.modules and one entry is added to the path_importer_cache for each run. These are not cleared by invalidate_caches().

If you append the following lines to test_importlib_cache_tempdir() these objects (and the caches in them) get cleared and your test passes. 

    sys.modules.pop(basename + ".test2")
    sys.modules.pop(basename)
    sys.path_importer_cache.pop(path)

This can also be confirmed using sys.gettotalrefcount().

Thanks Sebastian for looking at it \o/

Thanks for the details.