I have a hunch that this fix here may be causing this: https://github.com/spotify/dh-virtualenv/issues/148

Minimally:

echo 'from setuptools import setup; setup(name="demo")' > setup.py
echo 'import pytest' > tests/__init__.py

$ python setup.py test
running test
running egg_info
writing dependency_links to demo.egg-info/dependency_links.txt
writing demo.egg-info/PKG-INFO
writing top-level names to demo.egg-info/top_level.txt
reading manifest file 'demo.egg-info/SOURCES.txt'
writing manifest file 'demo.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK


$ python3.5 setup.py test
running test
running egg_info
writing demo.egg-info/PKG-INFO
writing dependency_links to demo.egg-info/dependency_links.txt
writing top-level names to demo.egg-info/top_level.txt
reading manifest file 'demo.egg-info/SOURCES.txt'
writing manifest file 'demo.egg-info/SOURCES.txt'
running build_ext
tests (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 462, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/tmp/foo/tests/__init__.py", line 1, in <module>
    import pytest
ImportError: No module named 'pytest'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)