On my machine (Lenovo T400, Linux) the load average during running
the 2.7 test suite is about 0.5 (with some exceptions like test_io).
The system is still usable even during test_io.


In 3.6 the load average is > 2.0, with some tests making
the system practically unusable (e.g. test_mmap).

Scratch the comment about test_io in 2.7: It also renders the
system unusable.

Just to diagnose this, try to run the test suite with "nice". If it still make the system unusable, then it's probably an I/O issue rather than CPU scheduling.

Any specific tests or all tests? 3.x has many new features in comparison with 2.x, and new tests for these features. For example zipfile and tarfile tests run much longer due to testing with bzip2 and lzma compressions.

It's very likely an I/O issue. There are several large file
tests that run even if 'largefile' is disabled. It already
helps a lot if the following tests are decorated with requires_resource('largefile'):

test_mmap: class LargeMmapTests()

test_io: large_file_ops()

test_largefile: all tests

Note some of those large file tests work on sparse files, which shouldn't be a problem if they are supported by the filesystem (which is most Linux filesystems, AFAIK).

Thanks, this clears up the mystery! -- I made the mistake of choosing
the eCryptfs-homedir option when installing Ubuntu, so now I have eCryptfs on top of LVM on top of full-disk encryption.

So I can't create sparse files in my home directory, but I *can*
create sparse files with zero overhead in /tmp.


The test suite runs smoothly from /usr/local.  Should we perhaps create
temp files in /tmp rather than in a subdir of the build directory?

> Should we perhaps create
temp files in /tmp rather than in a subdir of the build directory?

That would sound reasonable, I don't know why it hasn't been done like that from the start. Perhaps Ezio or Victor know.

There is not option to choose the temporary directory like $TMPDIR?

I had issues with /tmp when creating large files because /tmp is now a
tmpfs on Fedora by default.

Maybe we can blacklist some filesystems to choose the temporary directory.

Lib/test/regrtest.py:
# When tests are run from the Python build directory, it is best practice
# to keep the test files in a subfolder.  This eases the cleanup of leftover
# files using the "make distclean" command.
if sysconfig.is_python_build():
    TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build')
else:
    TEMPDIR = tempfile.gettempdir()

Yes, the current scheme is probably good for easy buildbot cleanup.


> Maybe we can blacklist some filesystems to choose the temporary directory.

If that's easily possible, it would be a good solution at least
for ecryptfs.


I've removed ecryptfs from my setup, since speedwise it's not
production-ready for encrypting an entire home directory.


The Ubuntu setup is a bit of a trap, since the encryption is
completely transparent to the user (i.e. mounting ecryptfs
occurs automatically on login).

So if you install Ubuntu as the first system on new hardware
and choose the ecryptfs option, after a while you forget about
ecryptfs entirely and blame any slowness on the hardware or
the software.

Actually, for ecryptfs we could just check for ~/.ecryptfs and
then use /tmp.

No activity for 5 years, so far this issue is more an open discussion than a list of concrete changes, so I close the issue as out of date.

If some tests are too CPU or I/O intensive, you can skip them using the -x option. Use --ignorefile to only skip specific test methods.

About the temporary directory, regrtest now has a --tempdir option.