As the results of investigating a very poor performance of pip while trying to install some code I was able to identify that the root cause was the current implementation of distutils.filelist.findall or to be more precise the _find_all_simple function, which does followsymlinks but without any measures for preventing recursivity and duplicates.

To give an idea in my case it was taking 5-10minutes to run while the CPU was at 100%, for a repository with 95k files (most of them temp inside .tox folders). Removal of the symlinks did make it run in ~5s.

IMHO, _find_all_simple should normalize paths and avoid returning any duplicates.


Realted: https://bugs.launchpad.net/pbr/+bug/1933311