(I pushed the change 245a16f33c4b, but Serhiy asked me to review it and discuss it. So I reverted my change and created this issue.)

I noticed many times that buildbots log "test xxx changed yyy" but this line is not enough to debug the issue. I propose to always display the old and new value to easy debugging these issues.

Sometimes, I try to reproduce the issue, but I fail to reproduce it locally. I may depend on the test execution order and the platform.

Attached patch changes:

* Replace get/restore methods with a Resource class and Resource subclasses
* Create ModuleAttr, ModuleAttrList and ModuleAttrDict helper classes
* Use __subclasses__() to get resource classes instead of using an hardcoded
  list (2 shutil resources were missinged in the list!)
* Don't define MultiprocessingProcessDangling resource if the multiprocessing module is missing
* Nicer diff for dictionaries. Useful for the big os.environ dict
* Reorder code to group resources


I chose to use classes to be able to easily customize how "Before/After" (value diff) is displayed: the new display_diff() method.

I also wrote helper classes to factorize the code.

Example:

    def get_sys_path(self):
        return id(sys.path), sys.path, sys.path[:]
    def restore_sys_path(self, saved_path):
        sys.path = saved_path[1]
        sys.path[:] = saved_path[2]

becomes

   class SysPath(ModuleAttrList):
       name = 'sys.path'

When sys.path is modified, currently Python displays the whole (id, object, object_copy) tuple which is not easily readable. With my change, it only displays object_copy (one list).

I began to write a pretty diff for ModuleAttrList, but it looks non trivial, and I'm not sure that it's worth (since lists are quite short).

Old code looks much clearer to me.

Consider your example with the 'sys.path' resource. With current code I need to understand only 3 simple lines of the code in two close methods. With the patch I need to research 4 classes with 16 complicated methods! This is awful!

That was about first two items.

Now about using __subclasses__(). Is it guarantied that the order of classes in __subclasses__() is the same as the order of declaration and always will be? The order of restoring resources matters. For now if there is needed to change the order of restoring resources (I'm sure there will be need to do this in future) you need to change only one or two lines. With your patch you need to reorder classes declarations. This not only makes the patch larger, but makes harder to review the diff and to track the history of the file.

Nicer diff for dictionaries can be an enhancement, but it can be implemented without such large rewriting.

> Consider your example with the 'sys.path' resource. With current code I need to understand only 3 simple lines of the code in two close methods. With the patch I need to research 4 classes with 16 complicated methods! This is awful!

I replaced get_sys_path()/restore_sys_path() (2 methods) with ModuleAttr.get()+ModuleAttrList.encode_value() / ModuleAttr.restore()+ModuleAttrList.restore_attr() (4 methods).

I want to factorize the code to limit duplicated code.

My use case is that I want to see what changed when I get the "test xxx changed yyy" error. For sys.path, ModuleAttrList.decode_value() helps to get a nicer output since you only display 1 list instead of the (id, list, list_copy) tuple.

How do you suggest to enhance the existing code to get such nicer output?


> Now about using __subclasses__(). Is it guarantied that the order of classes in __subclasses__() is the same as the order of declaration and always will be?

No, it looks random.


> The order of restoring resources matters.

I'm not sure that it's a good thing to have dependencies between "resources". I see that Files depends on Cwd, but maybe the Files check can be integrated into Cwd?

Ok so the order matters, in this case, I can use a metaclass to use the order of class definition. It will make get_resources() simpler, it will be build directly in the metaclass.

> For now if there is needed to change the order of restoring resources (I'm sure there will be need to do this in future) you need to change only one or two lines. With your patch you need to reorder classes declarations.

Yes. Is it really an issue? I only see two resources that have a dependency (Files and Cwd). It's not like this file is modified regulary.


> Nicer diff for dictionaries can be an enhancement, but it can be implemented without such large rewriting.

How do you want to implement that? Currently, a resource has a "get" method which returns a blackbox. It's not easy to display it. There is not standard format.

> I replaced get_sys_path()/restore_sys_path() (2 methods) with ModuleAttr.get()+ModuleAttrList.encode_value() / ModuleAttr.restore()+ModuleAttrList.restore_attr() (4 methods).

I need to examine all other methods scattered around just to conclude that they are overloaded and aren't used indirectly.

Imagine that you are reading the code first time and need to know only about one resource. Three lines in two methods are much simpler then hundreds lines scattered in tens of methods.

> I'm not sure that it's a good thing to have dependencies between "resources". I see that Files depends on Cwd, but maybe the Files check can be integrated into Cwd?

I'm not sure that there are only two depended resources. In any case deterministic behavior is better. It is very unhappy to to research random test failures.

> Ok so the order matters, in this case, I can use a metaclass to use the order of class definition. It will make get_resources() simpler, it will be build directly in the metaclass.

No-no-no. Please don't complicate the code just because you can. This will make reading and modifying the code yet harder.

> How do you want to implement that? Currently, a resource has a "get" method which returns a blackbox. It's not easy to display it. There is not standard format.

There is a number of ways.

1. Just write smart general diff reporter as in unittest. It's Python, use force^W data reflection!

2. Add third method for resources that need special diff reporting. Use simple general diff reporter by default.

3. Split the "sys.path" resource on two simple resources: one test that sys.path identity is not changed, other tests it's content.

New changeset c5944a76697b by Victor Stinner in branch '3.5':
Issue #26643: Add missing shutil resources to regrtest.py
https://hg.python.org/cpython/rev/c5944a76697b

regrtest now always displays before/after when the environment is altered. I consider that it's enough to close this issue.

I'm no more interested to try to enhance the output.