classification
Title: Add a topological sort algorithm
Type: enhancement Stage: patch review
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Jim Fasarakis-Hilliard, Paddy McCarthy, Zahari.Dim, belopolsky, christian.heimes, eric.smith, gaborjbernat, gdr@garethrees.org, lukasz.langa, maresb, martin.panter, miss-islington, orsenthil, pablogsal, remi.lapeyre, rhettinger, terry.reedy, tim.peters, tshepang, vstinner, wim.glenn
Priority: low Keywords: patch, patch, patch

Created on 2013-01-20 21:39 by rhettinger, last changed 2020-08-27 02:09 by Paddy McCarthy.

Files
File name Uploaded Description Edit
mro_merge.py rhettinger, 2013-01-20 21:39
topological_sort.py rhettinger, 2019-05-26 22:13 topological_sort.py
tsort.1.py rhettinger, 2019-05-26 22:16 Early experiment with only a predecessor dict
ts.py tim.peters, 2020-01-15 02:56
Pull Requests
URL Status Linked Edit
PR 11583 merged pablogsal, 2019-01-17 00:35
PR 18155 merged pablogsal, 2020-01-23 20:26
PR 20558 merged pablogsal, 2020-05-31 22:31
PR 20561 merged miss-islington, 2020-05-31 23:41
Messages (61)
msg180319 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2013-01-20 21:39
I suggest adding a topological sort algorithm to the standard library.

In addition to being a fundamental algorithm, it is immediately useful in demonstrating how the MRO computation works and for pure Python implementations of MRO logic.   IIRC, the pgen code was also expressed in pure Python for the same reason.

I've attached a first-draft of the algorithm and an alternative that only implements a topological merge.  This is just an early draft and there are a number of open points:

* which module to put it in
* a better implementation may be possible (perhaps using fewer dictionaries and sets).
msg180321 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-01-20 21:48
Good idea!

I'm using http://pypi.python.org/pypi/topsort/0.9 for a couple of years. The initial code was written by Tim Peters.
msg180329 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2013-01-21 01:13
+1

I'll note (by inspection only) your example code doesn't work under Python 3.x! :)

(print as a statement)
msg190419 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2013-05-31 22:13
+1

I had "tsort" in my own utilities library for so long that I thought it was in stdlib already.  I found this issue after unsuccessfully searching docs.python.org. :-)

I think functools will be a fine place for it.  It is somewhat related to total ordering and solves the problem which is common when implementing functional mini-languages.

Another possibility is shutil given that tsort is a standard POSIX command, <http://pubs.opengroup.org/onlinepubs/009695299/utilities/tsort.html>, but I think this will be too obscure.
msg333806 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2019-01-17 00:43
Here is an implementation that I've used for years. It is somewhat shorter than the one in PR 11583:

class CycleError(LogicalError, ValueError):
    """dependencies cycle detected

    """


def tsort(pairs):
    """topological sort

    Just like unix tsort(1)

    >>> tsort([(1, 2), (7, 8), (8, 10), (7, 4), (2, 3), (4, 10)])
    [1, 7, 2, 8, 4, 3, 10]
    >>> try:
    ...     tsort([(1,2), (2,1)])
    ... except CycleError as e:
    ...     print(e)
    ([], Counter({1: 1, 2: 1}), {1: [2], 2: [1]})
    """
    # inspired by http://mail.python.org/pipermail/python-list/1999-July/002831.html
    successors = {}
    predecessor_counts = collections.Counter()
    for x, y in pairs:
        successors.setdefault(x, []).append(y)
        predecessor_counts.setdefault(x, 0)
        predecessor_counts[y] += 1
    ordered = [x for x in predecessor_counts
               if predecessor_counts[x] == 0]
    for x in ordered:
        del predecessor_counts[x]
        for y in successors.get(x, ()):
            predecessor_counts[y] -= 1
            if predecessor_counts[y] == 0:
                ordered.append(y)
    if predecessor_counts:
        raise CycleError(ordered, predecessor_counts, successors)
    return ordered
msg333810 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-01-17 01:31
Thanks. I have some API nits to work to make this more broadly useful.  I'll all those on the PR comments soonish :-)
msg333816 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-17 02:13
The one in PR 11583 is twice as fast:

>timeit for -> topsort([(2,11),(9,11),(9,8),(9,10),(10,11),(10,3),(11,7),(11,5),(8,7),(8,3)])
12.4 µs ± 59.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>timeit for -> tsort([(2,11),(9,11),(9,8),(9,10),(10,11),(10,3),(11,7),(11,5),(8,7),(8,3)])
29.1 µs ± 147 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
msg333831 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-01-17 09:01
As the name been already discussed ? 


I fear that topsort might only be clear to people already knowing what it does. topoligical_sort would be more discoverable and explicit and one can always do

    from functools import topological_sort as tsort

if he wants to save some typing later.
msg334000 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2019-01-18 18:19
I approve in general with the principle of including a topological sort algorithm in the standard library. However, I have three problems with the approach in PR 11583:

1. The name "topsort" is most naturally parsed as "top sort" which could be misinterpreted (as a sort that puts items on top in some way). If the name must be abbreviated then "toposort" would be better.

2. "Topological sort" is a terrible name: the analogy with topological graph theory is (i) unlikely to be helpful to anyone; and (ii) not quite right. I know that the name is widely used in computing, but a name incorporating "linearize" or "linear order" or "total order" would be much clearer.

3. The proposed interface is not suitable for all cases! The function topsort takes a list of directed edges and returns a linear order on the vertices in those edges (if any linear order exists). But this means that if there are any isolated vertices (that is, vertices with no edges) in the dependency graph, then there is no way of passing those vertices to the function. This means that (i) it is inconvenient to use the proposed interface because you have to find the isolated vertices in your graph and add them to the linear order after calling the function; (ii) it is a bug magnet because many programmers will omit this step, meaning that their code will unexpectedly fail when their graph has an isolated vertex. The interface needs to be redesigned to take the graph in some other representation.
msg334002 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-18 18:36
> 1. The name "topsort" is most naturally parsed as "top sort" which could be misinterpreted (as a sort that puts items on top in some way). If the name must be abbreviated then "toposort" would be better.


I totally agree that `topsort` is a bad name, I used it more or less as a dummy for starting the discussion about the implementation.

> 2. "Topological sort" is a terrible name: the analogy with topological graph theory is (i) unlikely to be helpful to anyone; and (ii) not quite right. I know that the name is widely used in computing, but a name incorporating "linearize" or "linear order" or "total order" would be much clearer.


Topological sort (not as the function name) but as an operation is a very well known concept and is well defined. If you are referring to not use "Topological Sort" in the docstrings or the documentation, I strongly oppose.


Regarding the interface, I am more happy to change it once there is an agreement. I am still awaiting Raymond's comments regarding this so we can start discussing.
msg334010 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2019-01-18 21:09
Just to elaborate on what I mean by "bug magnet". (I'm sure Pablo understands this, but there may be other readers who would like to see it spelled out.)

Suppose that you have a directed graph represented as a mapping from a vertex to an iterable of its out-neighbours. Then the "obvious" way to get a total order on the vertices in the graph would be to generate the edges and pass them to topsort:

    def edges(graph):
        return ((v, w) for v, ww in graph.items() for w in ww)
    order = topsort(edges(graph))

This will appear to work fine if it is never tested with a graph that has isolated vertices (which would be an all too easy omission).

To handle isolated vertices you have to remember to write something like this:

    reversed_graph = {v: [] for v in graph}
    for v, ww in graph.items():
        for w in ww:
            reversed_graph[w].append(v)
    order = topsort(edges(graph)) + [
          v for v, ww in graph.items() if not ww and not reversed_graph[v]]

I think it likely that beginner programmers will forget to do this and be surprised later on when their total order is missing some of the vertices.
msg334012 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-18 21:27
I think 'toposort' is a good name for a function that implements 'topological sorting'.
https://en.wikipedia.org/wiki/Topological_sorting
msg334014 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-01-18 21:39
This is why I prefer the API exposed by https://pypi.org/project/toposort/

list(toposort({2: {11},
               9: {11, 8, 10},
               10: {11, 3},
               11: {7, 5},
               8: {7, 3},
              }))

returns [{3, 5, 7}, {8, 11}, {2, 10}, {9}]

For an node with no edges, use an empty set:

list(toposort({100: set(),
               2: {11},
               9: {11, 8, 10},
               10: {11, 3},
               11: {7, 5},
               8: {7, 3},
              }))
[{3, 100, 5, 7}, {8, 11}, {2, 10}, {9}]

I also don't think we should provide multiple APIs. Let's just provide one, and recipes for any helpers, if needed. For example, to flatten the result into a list. Or to take a list of edges as the input.
msg334100 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-20 20:42
I have updated the PR to receive a dictionary of sets as in Eric V. Smith's package. I have maintained the guts of the current algorithm as it scales much better:

>>> test_data = {x:{x+n for n in range(100)} for x in range(1000)}

>>> %timeit list(toposort.toposort(test_data)) # The one in PyPi
910 ms ± 2.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

>>> %timeit list(functools.toposort(test_data)) # In this PR

69.3 ms ± 280 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

>>> list(functools.toposort(l)) == list(toposort.toposort(l))
True
msg334103 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-20 21:34
Although I think the API with the dictionary may be cleaner, I still do not like it completely.

1) One of the reasons is that implementing C3 directly with the topological sort is not straightforward as the different nodes that are at the same level are unordered and therefore any permutation of all these is a valid topological sort, while C3 comes from a stable sort. The version with pairs is stable on the other hand.

2) Topological sorting usually is well-defined on totally connected graphs, so I do not know what exactly it means to topologically sort two disjoint graphs. This was one of the main drawbacks of the tuple-based approach, but I think it may be a good property.
msg334104 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-01-20 21:42
> 2) Topological sorting usually is well-defined on totally connected graphs, so I do not know what exactly it means to topologically sort two disjoint graphs. This was one of the main drawbacks of the tuple-based approach, but I think it may be a good property.

To give a use-case, I'm currently using topological sort to order a list of tasks where edges represent dependencies between tasks. Sometime a group of tasks does not share a dependency with another group any relative order between those two groups is correct:


A -> B

   C
  / \
 D   E
  \ /
   F

The order (A, B, C, D, E, F) would be correct in this example as would (C, A, E, B, D, F).

I think the general topological sort in Python should be able to handle such inputs.
msg334107 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-20 22:43
(A, B, C, D, E, F) would not be correct as B is order 2 and "C" and "A" is order 1. The correct orders are the inner-group permutations of:

[{'A', 'F'}, {'B', 'D', 'E'}, {'C'}]

(Assuming that the lower diagram goes from top to bottom)
msg343584 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-05-26 22:13
Attaching some of my 2013 work on this.  Here are some API notes:

* spell-out the name topological_sort()

* allow input as ordered pairs like the Unix tsort command
  https://en.wikipedia.org/wiki/Tsort#Examples

* allow more convenient input as dependency sequences (like graphviz):
     [['a', 'b', 'c', 'x], ['b', 'd', 'e', 'y']]
  is short for and equivalent to:
     [(a,b), (b,c), (c,x), (b,d), (d, e), (e, y)]

* return both the sorted sequence and cycles
  (both are individually useful and the latter
   is helpful in debugging in only the former is wanted)

* desire to match the C3 MRO computation


* would like the ordering to be stable and deterministic
  (this precludes picking arbitrary elements from sets).
msg343586 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-26 23:27
> * allow input as ordered pairs like the Unix tsort command
> * allow more convenient input as dependency sequences (like graphviz):

This is how my first proposal started (and I still like it a bit more than the dictionary input), but there are some concerns (check other comments) regarding this API, like representing isolated nodes or disjoint graphs.

> return both the sorted sequence and cycles

Regarding the output, I like returning a collection of sets, where every set represents all possible elements of the same order in the result. This also helps if the user has some expectation regarding the ordering. For example, in:

['ABDGI', 'BEG', 'CEH', 'KCFHJ']

the results starting with

['A', 'B', 'D'

and

['A', 'B', 'K'

are both valid.

With the current implementation, this is the equivalent of C3 linearization:

      from itertools import tee
      from collections import defaultdict
      def c3_linearization(inheritance_seqs):
         graph = defaultdict(set)
         for seq in inheritance_seqs:
             a, b = tee(seq)
             next(b, None)
             for child, parent in zip(a,b):
                 graph[child].add(parent)
         retun ((list(group) for group in functools.toposort(graph)), [])
         return tuple(reversed(order))

      >>> class A: pass
      >>> class B(A): pass
      >>> class C(A): pass
      >>> class D(B, C): pass

       >> D.__mro__
      (__main__.D, __main__.B, __main__.C, __main__.A, object)

       >> c3_linearization([(D, B, A, object), (D, C, A, object)])
      [{__main__.D}, {__main__.B, __main__.C}, {__main__.A}, {object}]

What do you think?
msg344143 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-06-01 03:49
Unless Łukasz gives us a nod to work out this API for the second beta, we'll need to defer this to 3.9.   IMO, the API in the patch is not user friendly.  It started on the right path but became contorted (for both inputs and outputs) to serve unusual cases.
msg359672 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2020-01-09 12:33
I'd like to push back on the idea that graphs with isolated vertices are "unusual cases" as suggested by Raymond.

A very common use case (possibly the most common) for topological sorting is job scheduling. In this use case you have a collection of jobs, some of which have dependencies on other jobs, and you want to output a schedule according to which the jobs can be executed so that each job is executed after all its dependencies.

In this use case, any job that has no dependencies, and is not itself a dependency of any other job, is an isolated vertex in the dependency graph. This means that the proposed interface (that is, the interface taking only pairs of vertices) will not be suitable for this use case. Any any programmer who tries to use it for this use case will be setting themselves up for failure.
msg359702 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-01-10 02:24
Let's stir this up a bit ;-)  I recently had occasion to exchange ideas with Larry Hastings about topsorts for use in a package manager he's writing.  I thought his API for adding edges was ... perfect:

    add(node, *dependson)

So, e.g., add(A, B, C) says A depends on B, and on C, but says nothing else about B and C.  This is almost always the way topsorts show up in real life:  you know what a thing depends *on* directly, but have scant idea how things may be in the opposite direction.  For example, you know that baking a cake requires (among other things) flour, but have no real idea of the universe of other things that require flour.  Likewise Larry knows which packages each package requires, but not the reverse.  Etc.

Nodes with no edges are trivial to add then:  add(A).

If you're building input to a topsort from a graph, also trivial:

    for n, p in node2predecessors.items():
        topsort_instance.add(n, *p)

and it doesn't matter whether the predecessors in the original graph were stored in a list, set, tuple, or any other iterable container.  Nothing special about an empty collection of predecessors either.

The other big thing that came up is that most topsort programs were useless for his goal:  downloading and installing packages takes significant wall clock time, and there's huge opportunity for exploiting parallelism.  But a flat sequence in topsort order gives no clue about what _can_ be done in parallel.  Instead you really want several methods, like

    prepare()

to say that you're done building the graph; and,

    get_ready()

to get all nodes ready to go, which haven't already been returned by get_ready() calls (initially, this is the collection of nodes with no predecessors, which prepare() can know); and,

    done(node)

to say that `node` (returned by a previous call to get_ready()) is finished now, so that the next call to get_ready() can return all (if any) nodes for which `node` was the last non-done predecessor; and,

    is_active()

to say whether the topsort can make more progress (is_active() returns True iff there are still nodes ready to go that haven't yet been passed out by get_ready(), or if the number of nodes marked done() is less than the number that have been passed out by get_ready()).

These are all easy to code, and allow the user to extract all the opportunities for parallelism that theoretically exist.  There is no static order that can do so, since the opportunities that exist at any time depend on the times and order in which nodes are marked done() in real life - and that may vary from one run to the next.

Of course a deterministic static order can be derived from those, like, e.g.,

    def static_order(self):
        self.prepare()
        while self.is_active():
            for node in self.get_ready():
                yield node
                self.done(node)

For parallel use, e.g.,

    self.prepare()
    while instance.is_active():
        for node in instance.get_ready():
            inq.put(node)
        node = outq.get()
        instance.done(node)

where worker threads or processes take nodes to work on off of queue `inq`, then, when the work for a node is done, put the node on queue `outq`.

Am I seriously suggesting this for Python?  Sure.  It's fun to advance the practical state of the art :-)
msg360016 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-15 01:24
> Am I seriously suggesting this for Python?  Sure.  It's fun to advance the practical state of the art :-)


I think this API looks very interesting! I have some questions before start implementing it to play a bit with it:

- I am slightly confused about what .prepare() should do. Why is this step necessary?

- Why we need the .done() method here? Why not instead make get_ready() simply a generator so you can just write

    for node in self.get_ready():

It seems that the .done() is very tight to use this API as a "task scheduler" but maybe I am doing something here in
my understanding of the API.
msg360017 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-01-15 02:03
> I am slightly confused about what .prepare() should do. Why
> is this step necessary?

To say "I'm done adding edges".  Any call to add() after prepare() should raise an exception.  Likewise, e.g., any call to get_ready() before prepare() should raise an exception.  In a bog-standard topsort implementation, saving for each node a sequence of successors and a count of predecessors, this is also the time to collect all the nodes that have no predecessors (the first batch returned by get_ready()).

Much the same could be done without prepare() by get_ready() making a special case out of the first time it's called.  That's more magical, though.  "I'm done adding edges" is utterly non-magical.

> - Why we need the .done() method here? Why not instead make get_ready()
> simply a generator so you can just write
>
>    for node in self.get_ready():

The point of done() is to enable maximum exploitation of parallelism.  As already sketched, if a user doesn't care about that, fine, a different method (like static_order()) can generate all the nodes in _some_ static topsort order, with no need for done().

But suppose a user does care about parallelism.  Consider graph

A -> B
A -> C
A -> D
B -> D

Output A B C D is a topsort, but useless unless the user is content to "do" one node at a time.

Instead get_ready() first returns [A] (or a tuple, or a generator, or a set ... something iterable).  A is handed out to worker processes/threads, but get_ready() will return an empty iterable until done(A) is called.  Indeed, if "doing" A fails, it's NOT the case that anything else can ever be started.

If/when "doing A" succeeds, then done(A) is called, and the next get_ready() returns [B, C].  Those can be done in parallel, but D can't be started until done(B) is called.  done(B) may or may not be called before done(C) is called - the topsort itself has no way to know in advance, nor _should_ it impose its own view of that.  Note that D does not depend on C, so there's no need to wait for _both_ in [B, C] to finish.  It's necessary and sufficient that B be marked done() for D to be ready to go.

> It seems that the .done() is very tight to use this API as a "task
> scheduler" but maybe I am doing something here in my understanding
> of the API.

done() says nothing about how the user "should" schedule work items, but instead allows get_ready() to return all the work items whose predecessors have been marked done() (but haven't already been passed out by get_ready()).  That's the maximum set of nodes that _can_ be worked on at the time.  The topsort class itself has no policy about how or when they "should" be worked on, get_ready() is just identifying all the possibilities that exist.  Which is impossible to know unless the class is also told which nodes it already passed out have finished - the purpose of done().

is_active() eventually returns False when all the nodes passed out by get_ready() have been marked done(), _and_ there are no more nodes ready to pass out.  At that point, there's a cycle in the input relations if and only if there's some node get_ready() never passed out.

In my prototype implementation, that's another thing prepare() does:  checks for a cycle, and raises CycleError if there is one.  The user can catch & ignore that if they like, and continue calling get_ready() and done() until no more progress can be made.  I think it's more likely, though, that the user would stare at the cycle attached to the CycleError instance, do whatever it takes to break the cycle, and start over again.
msg360018 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-15 02:40
Fair enough, I will read this carefully again and try to sketch a prototype soon :)
msg360019 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-01-15 02:56
I'll add ts.py, which was a work-in-progress that implements a minor variation of most everything I typed about.  If nothing else, its _find_cycle is useful as a non-recursive linear-time cycle finder (recursion is deadly here because recursive depth-first search can easily "blow the stack" on larger graphs).

There's also "if 1:"/"else:" blocks that set up parallel cases, using threads or processes, and two ways of managing the parallelism (the one I showed before, and a more elaborate one that puts an upper bound on how large the queues can grow - which is sometimes "a problem" for multiprocessing.queue).
msg360020 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-15 03:01
Disregarding the API, what do you think about the approach of https://github.com/python/cpython/pull/11583 for the implementation? Under my benchmarks (check previous comments) it seems to perform very good with regards to memory and time.
msg360022 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-01-15 03:51
Oh, it's fine!  Kahn's algorithm is what I meant when I wrote the "bog-standard implementation" before.

I don't believe I've ever seen a context in real life where topsort speed made a lick of real difference, so I expect any linear-time (in the sum of the number of nodes and edges) would be fine.  Nevertheless, for recording a node's successors ("children" in your code), I've always used a list rather than a set.  Lists run faster and require less memory than sets, and - unlike sets - in Python inherently preserve insertion order.  Iteration order can become visible (e.g., if B, C, and D depend on A, what's "the" topsort order?  it depends on the order A's children appear when iterating over them - predictable with a list, "it depends" with a set).

Note:  "but we have to guard against redundant edges!" would be a red herring.  Kahn's algorithm couldn't care less, provided that predecessor counts accurately reflect the number of edges (redundant or not) entering a node.
msg360157 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-17 00:07
I have been playing with the API for a while and I have to say that I have fallen in love with it: I tried to reimplement many snippets of parallel and non-parallel topological-order processing of a graph with this and it always fits very nicely.

I have shown this API also to some library maintainers that have to deal with topological sorting (like the tox maintainer) and they are very enthusiastic about the API as well.

Tim, I have updated PR 11583 to use the proposed API with test and docs. Could you make a review when you have some time so we can polish it for the 3.9 release?
msg360214 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-01-17 20:57
I think the new interface feeds better for usage in both sequential or parallel workflows, which means we can use a single UI for both, so 👍 from myself
msg360258 - (view) Author: Zahari Dim (Zahari.Dim) Date: 2020-01-19 14:03
I would like to suggest a `dependency_resolver` API that I have been using that goes in line with what Tim Peters proposes in https://bugs.python.org/issue17005#msg359702

A DAG would be an object that can be iterated in topological order with __iter__ (for simple sequential usage) or have a way of managing all the tasks that can be run in parallel. The later is done with a generator function:

```
    def dependency_resolver(self):
        """Yield the set of nodes that have all dependencies satisfied (which could be an empty set). Send the next
        completed task."""
```

which is used with something like:

```
deps = dag.dependency_resolver()
pending_tasks = deps.send(None)
if not pending_tasks:
    #Graph empty
    return
#Note this is a can be done in parallel/async
while True:
    some_task = pending_tasks.pop()
    complete_task_somehow(some_task)
    try:
       more_tasks = deps.send(some_task)
    except StopIteration:
       #Exit when we have sent in all the nodes in the graph
       break
    else:
        pending_tasks |= more_tasks

```


An implementation I have used for some time is here:


https://github.com/NNPDF/reportengine/blob/master/src/reportengine/dag.py

although I'd make simpler now. In practice I have found that the function I use most of the time to build the graph is:

dag.add_or_update_node(node=something_hashable, inputs={set of existing nodes}, outputs={set of existing nodes}).

which adds the node to the graph if it was not there and maps updates the dependencies to add inputs and outputs, which in my experience matches the way one discovers dependencies for things like packages.
msg360567 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-23 15:30
New changeset 99e6c260d60655f3d2885af545cbc220b808d492 by Pablo Galindo in branch 'master':
bpo-17005: Add a class to perform topological sorting to the standard library (GH-11583)
https://github.com/python/cpython/commit/99e6c260d60655f3d2885af545cbc220b808d492
msg360580 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-23 21:01
New changeset 65ecc390c1fa5acdd6348ae3f9843bbdcd8870d1 by Pablo Galindo in branch 'master':
bpo-17005: Minor improvements to the documentation of TopologicalSorter (GH-18155)
https://github.com/python/cpython/commit/65ecc390c1fa5acdd6348ae3f9843bbdcd8870d1
msg365614 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-04-02 18:48
At some point in the next two or three weeks, I'll have a chance to work on this more and to offer a competing patch.  IMO, the current checkin is over-engineered, both in its API and implementation.  This could have been a simple, fast tool written as one or two short Python functions.

Also, I would like to try to out the API alternatives on some groups of engineers to get some user feedback.

For me, as the API currently stands, I would have to write a wrapper to make it usable for my applications.
msg365618 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-04-02 19:32
Raymond, what application do you have that wouldn't be completely addressed by sticking to just .add() (to record dependencies) and .static_order() (to retrieve a linear order)?

Larry Hastings and I originally worked out the fancier bits of the interface to deal with problems he actually had, and for which no existing Python topsort implementation we could find was of any use:  extract maximal parallelism.  If you don't want that, fine, stick to the two simple bits.

The bits to support parallelism are very easy to use to write correct parallelized code, but of course can seem baffling if you don't give a rip about parallelism.  But in that case you have no need to learn about them either.

If your alternative isn't equally easy to use in a parallelized context, I'll be at best +0.

About "fast", this is linear time, in the sum of the number of items and dependencies.  Including the part checking for a cycle, which is by far the "hardest" part.  So it's asymptotically optimal, although I've never seen a real context in which topsort speed made a lick of difference.

In the real world, in a parallelized context it can be important to check for a cycle _before_ running a topsort:  actions are performed ASAP based on order-deduced-so-far, and it can be no good to find out "oh! I can't finish this" at the end.  There's actually nothing gratuitous here.  If it seems "over-engineered", that's because it's addressing problems you haven't had yet ;-)
msg365622 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-02 20:20
I just want to echo what Tim mentioned with the extra data point that some of the maintainers of some popular and wide-used open-source libraries that indeed have to deal with this problem or the parallel version of the problem (like gaborbernat in this thread, the maintainer of "tox" and "virtualenv") do indeed find the current API desirable.
msg365631 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-04-02 21:45
> If your alternative isn't equally easy to use in a parallelized
> context, I'll be at best +0.

We may need two versions then, a full-featured TopologicalSorter() class and a simple tsort() function that doesn't aspire to be all things to all people.
msg365632 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-02 21:53
> We may need two versions then, a full-featured TopologicalSorter() class and a simple tsort() function that doesn't aspire to be all things to all people.

How this other version would differ from using .add() + .static_order() as Tim mentions? Originally we designed static_order() so it will satisfy the simpler use cases so I would suggest aspiring to simplify that interface if needed instead of adding an extra function.
msg365638 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-04-02 22:20
Possibly, sure.  But I believe it's hard to beat

    add(node, *predecessors)

for usability as a way to build the dependency graph.  For example, a list of pairs is a comparative PITA for most use cases I've had.  Whether it's following a recipe to bake a cake, or tracing a maze of C include files, it seems _most_ natural to get input in the form "this thing depends on these other things".  Not the other way around, and neither a sequence of pairs.

_If_ you buy that, then .add() is screamingly natural, and trying to squash a pile of .add()s into a single sequence-of-sequences argument seems strained.

Typically I don't get input in one big, single gulp.  It's instead discovered one item at a time.  Fine - .add() it and then move on to the next item.  It's certainly possible to append the item and its predecessors to a persistent (across items) list, and call a function once at the end with that list.

But what does that buy?  I'm building the list solely to meet the function's input requirement - the list serves no other purpose.  Instead of calling .add() N times, I call .append() N times.  "add" is 3 letters shorter ;-)
msg365639 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-02 22:28
Is also notable to mention that you can also provide the graph as a dictionary to the constructor:

>>> graph = {D: {B, C}, C: {A}, B: {A}, A:{object}}
>>> ts = TopologicalSorter(graph)
msg365644 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-04-02 22:57
How about I post a PR so we can talk about something concrete.  Then you two can either fight it to its death or you can join me in making it is good as possible, hopefully the latter :-)

I am not happy with the current API but do accept that both of you are in satisfied with it.
msg370488 - (view) Author: Ben Mares (maresb) Date: 2020-05-31 16:01
It's great to have this feature in the standard library, but it really seems to clutter the functools documentation. Everything else in functools applies directly to functions and methods. Suddenly reading graph theory terminology was disorienting for me. (Due to context, I expected "node" to mean some new type of Python function.) It makes the documentation significantly longer, and IMHO abstruse. Would it be sensible to move this to a separate module?
msg370494 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 18:42
Any suggestions for the new module? I assume we can move this to a new topsort/topological_sort or similar...

What do people think?
msg370496 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2020-05-31 18:46
Could it make sense to have this in the often proposed imath module? 

It's integers per se but Both number theory and graph theory are part of discrete mathematics so it may feel more at home there?
msg370500 - (view) Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * Date: 2020-05-31 19:32
It does seem out of place in functools, intensified by it's odd interjection among the other functools objects.

Considering heapq and bisect exist as standalone modules, the idea that topological sorting could go in its own module wouldn't be without precedent.
msg370501 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 20:53
If we move it, I would prefer a new module that somehow makes clear the scope, something like graphutils or graphtheory or graph (although this las one will probably collide with many existing things).
msg370503 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-05-31 20:57
I like graphutils for what it's worth.
msg370506 - (view) Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * Date: 2020-05-31 21:09
The downside I see with any graph prefixed names is the fact that it implies a larger collection of graph operations.

Add that to the fact that people might be more tempted to propose many  graph related algorithms/utilities to a module with the same name.

A more localized name solves that.
msg370508 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 21:18
> The downside I see with any graph prefixed names is the fact that it implies a larger collection of graph operations.

I don't see that an issue. In the general term is better to have a general name and discuss further improvements than just name the module "topological_sort" and now being cornered if we ever want to add new graph-related stuff.

We can always say "no" to new functionality but changing the name of the module is not possible once is released.
msg370509 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 21:21
I would like to hear Raymond and Tim advise on what the best name for the new module should be :)
msg370510 - (view) Author: Ben Mares (maresb) Date: 2020-05-31 21:54
dependencytools?

But I don't think it qualifies yet for the plural...
msg370511 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2020-05-31 21:56
`functools` is clearly a poor place for this. `imath` would also be. `graph_stuff_probably_limited_to_a_topsort` is the only accurate name ;-) Off-the-wall possibilities include `misclib` (stuff that just doesn't fit anywhere else - yet) and `cslib` (Computer Science-y stuff).

Whichever name wins, we should probably look to ensure the name isn't also of a package already in (non-trivial) use.
msg370514 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-05-31 22:02
I vote for it being in its own module (like difflib).  That would also be a good place to put my proposed tsort() function with its simpler API.
msg370515 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 22:14
I checked and from the list of proposed names with "graph" prefixes, only "graphutils" is not colliding with something on Pypi.

For the record "cslib" and "misclib" are also available but the scope of those feel much much bigger.

I am going to open a PR to move this to "graphutils" for now. We can discuss better in the PR for better names given that seems that there is an agreement on moving this out of functools.
msg370516 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-05-31 22:17
I'd prefer a new module just for this. As for names, I like toposort over topsort.

I already have a library on PyPI named toposort. Personally, I don't have a problem with the stdlib taking over that name, and I'd just deprecate mine at 3.9, or whatever. I'm not sure if there are any ramifications of doing that, however.

Or, pick another unused name, like toposortlib or something.
msg370517 - (view) Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * Date: 2020-05-31 22:18
Another option, `graphlib`[1], does exist on PyPI but is not maintained and currently read-only by the author. Other flavors[2][3] of the same name also don't seem to have much adoption so they shouldn't confuse if a name like `graphlib` was chosen.

[1] https://github.com/bruth/graphlib/
[2] https://github.com/MengLiuPurdue/graph_lib
[3] https://github.com/EmileTrotignon/GraphLib
msg370519 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 22:35
Opened https://github.com/python/cpython/pull/20558. I have *initially* proposed "graphlib" as it feels more in-line with other stdlib module names and as Jim pointed it does not collide with anything major. Also, I have proposed this initial name to not keep the scope too short in case we want to add new things in the future (although we can always say "no" and just keep this functionality).
msg370520 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-31 23:41
New changeset 2f172d8f1525defe9bba4d49e967fdfc69151731 by Pablo Galindo in branch 'master':
bpo-17005: Move topological sort functionality to its own module (GH-20558)
https://github.com/python/cpython/commit/2f172d8f1525defe9bba4d49e967fdfc69151731
msg370521 - (view) Author: miss-islington (miss-islington) Date: 2020-06-01 00:01
New changeset 0a674638a3de14dc86b5294a5db067e0c2177a51 by Miss Islington (bot) in branch '3.9':
bpo-17005: Move topological sort functionality to its own module (GH-20558)
https://github.com/python/cpython/commit/0a674638a3de14dc86b5294a5db067e0c2177a51
msg375979 - (view) Author: Paddy McCarthy (Paddy McCarthy) Date: 2020-08-27 02:01
I've been playing with Python 3.9.0rc1 and was looking at a particular graph to see when it released tasks for processing.
I ran the following code:

from functools import reduce
from pprint import pprint as pp
from collections import defaultdict
from graphlib import TopologicalSorter
from heapq import heapify, heappush, heappop, heappushpop

print("""
###
### TASK DEPENDENCY 
###

    A -> B -> C
    ↓    ↓    ↓
    D -> E -> F
""")
graph3 = {"A": {"B", "D"},
          "B": {"C", "E"},
          "C": {"F"},
          "D": {"E"},
          "E": {"F"},
          }
tasktime = {task: 2 for task in 'ABCDEF'}


def simulate(graph, tasktime):
    print("\n## NEW SIMULATION")
    t = 0
    heap = []
    heapify(heap)
    print("\n# Task runtimes:")
    for node, tm in tasktime.items():
        print(f"  {node}: {tm}")

    print("\n# OUTPUT. (:> for task start times, <: for stop times).\n")
    ts = TopologicalSorter(graph)
    ts.prepare()
    while ts.is_active():
        for node in ts.get_ready():
            finish = t + tasktime[node]
            heappush(heap, (finish, node))
            print(f"{'  ' * t}{node}:> @{t}")
        t, node = heappop(heap)
        print(f"{'  ' * t}{node}<: @{t}")
        ts.done(node)

simulate(graph3, tasktime)
tasktime['C'] = 1
simulate(graph3, tasktime)


I got the following output:


###
### TASK DEPENDENCY 
###

    A -> B -> C
    ↓    ↓    ↓
    D -> E -> F


## NEW SIMULATION

# Task runtimes:
  A: 2
  B: 2
  C: 2
  D: 2
  E: 2
  F: 2

# OUTPUT. (:> for task start times, <: for stop times).

F:> @0
    F<: @2
    C:> @2
    E:> @2
        C<: @4
        E<: @4
        B:> @4
        D:> @4
            B<: @6
            D<: @6
            A:> @6
                A<: @8

## NEW SIMULATION

# Task runtimes:
  A: 2
  B: 2
  C: 1
  D: 2
  E: 2
  F: 2

# OUTPUT. (:> for task start times, <: for stop times).

F:> @0
    F<: @2
    C:> @2
    E:> @2
      C<: @3
        E<: @4
        B:> @4
        D:> @4
            B<: @6
            D<: @6
            A:> @6
                A<: @8
>>> 


Note that in the second simulation, C finish, but B isn't then immediately started. I have my own code that also works like this but it isn't optimal.

Thanks guys.
msg375980 - (view) Author: Paddy McCarthy (Paddy McCarthy) Date: 2020-08-27 02:09
Please ignore my earlier Message-id 	<06462575371.issue17005@roundup.psfhosted.org">1598493715.04.0.06462575371.issue17005@roundup.psfhosted.org>.

I missed a dependency in cutting down a larger example. Sorry.
History
Date User Action Args
2020-08-27 02:09:46Paddy McCarthysetmessages: + msg375980
2020-08-27 02:01:55Paddy McCarthysetnosy: + Paddy McCarthy
messages: + msg375979
2020-06-01 00:01:43miss-islingtonsetmessages: + msg370521
2020-05-31 23:41:27miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19802
2020-05-31 23:41:17pablogsalsetmessages: + msg370520
2020-05-31 22:35:21pablogsalsetkeywords: patch, patch, patch

messages: + msg370519
2020-05-31 22:31:25pablogsalsetpull_requests: + pull_request19801
2020-05-31 22:18:53Jim Fasarakis-Hilliardsetmessages: + msg370517
2020-05-31 22:17:34eric.smithsetkeywords: patch, patch, patch

messages: + msg370516
2020-05-31 22:14:04pablogsalsetkeywords: patch, patch, patch

messages: + msg370515
2020-05-31 22:02:24rhettingersetkeywords: patch, patch, patch

messages: + msg370514
2020-05-31 21:56:48tim.peterssetkeywords: patch, patch, patch

messages: + msg370511
2020-05-31 21:54:59maresbsetmessages: + msg370510
2020-05-31 21:21:20pablogsalsetkeywords: patch, patch, patch

messages: + msg370509
2020-05-31 21:18:08pablogsalsetkeywords: patch, patch, patch

messages: + msg370508
2020-05-31 21:09:35Jim Fasarakis-Hilliardsetmessages: + msg370506
2020-05-31 20:57:59gaborjbernatsetmessages: + msg370503
2020-05-31 20:53:31pablogsalsetkeywords: patch, patch, patch

messages: + msg370501
2020-05-31 19:32:30Jim Fasarakis-Hilliardsetnosy: + Jim Fasarakis-Hilliard
messages: + msg370500
2020-05-31 18:46:53remi.lapeyresetmessages: + msg370496
2020-05-31 18:42:30pablogsalsetkeywords: patch, patch, patch

messages: + msg370494
2020-05-31 16:01:29maresbsetnosy: + maresb
messages: + msg370488
2020-04-02 22:57:44rhettingersetkeywords: patch, patch, patch

messages: + msg365644
2020-04-02 22:55:16rhettingersetmessages: - msg365640
2020-04-02 22:30:51rhettingersetkeywords: patch, patch, patch

messages: + msg365640
2020-04-02 22:28:28pablogsalsetkeywords: patch, patch, patch

messages: + msg365639
2020-04-02 22:20:46tim.peterssetkeywords: patch, patch, patch

messages: + msg365638
2020-04-02 21:53:52pablogsalsetkeywords: patch, patch, patch

messages: + msg365632
2020-04-02 21:45:54rhettingersetkeywords: patch, patch, patch

messages: + msg365631
2020-04-02 20:49:51vstinnersetkeywords: patch, patch, patch
nosy: + vstinner
2020-04-02 20:20:18pablogsalsetkeywords: patch, patch, patch

messages: + msg365622
2020-04-02 19:32:56tim.peterssetkeywords: patch, patch, patch

messages: + msg365618
2020-04-02 18:48:59rhettingersetkeywords: patch, patch, patch

messages: + msg365614
2020-02-20 20:57:25wim.glennsetnosy: + wim.glenn
2020-01-23 21:01:57pablogsalsetmessages: + msg360580
2020-01-23 20:26:22pablogsalsetpull_requests: + pull_request17541
2020-01-23 15:30:03pablogsalsetmessages: + msg360567
2020-01-19 14:03:42Zahari.Dimsetnosy: + Zahari.Dim
messages: + msg360258
2020-01-17 20:57:16gaborjbernatsetnosy: + gaborjbernat
messages: + msg360214
2020-01-17 00:07:47pablogsalsetkeywords: patch, patch, patch

messages: + msg360157
2020-01-15 03:51:31tim.peterssetkeywords: patch, patch, patch

messages: + msg360022
2020-01-15 03:01:41pablogsalsetkeywords: patch, patch, patch

messages: + msg360020
2020-01-15 02:56:47tim.peterssetkeywords: patch, patch, patch
files: + ts.py
messages: + msg360019
2020-01-15 02:40:12pablogsalsetkeywords: patch, patch, patch

messages: + msg360018
2020-01-15 02:04:00tim.peterssetkeywords: patch, patch, patch

messages: + msg360017
2020-01-15 01:24:19pablogsalsetkeywords: patch, patch, patch

messages: + msg360016
2020-01-10 02:37:05orsenthilsetkeywords: patch, patch, patch
nosy: + orsenthil
2020-01-10 02:24:43tim.peterssetkeywords: patch, patch, patch
nosy: + tim.peters
messages: + msg359702

2020-01-09 12:33:02gdr@garethrees.orgsetkeywords: patch, patch, patch

messages: + msg359672
2019-06-01 03:49:45rhettingersetkeywords: patch, patch, patch
nosy: + lukasz.langa
messages: + msg344143

2019-05-26 23:27:30pablogsalsetkeywords: patch, patch, patch

messages: + msg343586
2019-05-26 22:16:37rhettingersetkeywords: patch, patch, patch
files: + tsort.1.py
2019-05-26 22:13:09rhettingersetkeywords: patch, patch, patch
files: + topological_sort.py
messages: + msg343584
2019-01-20 22:43:24pablogsalsetkeywords: patch, patch, patch

messages: + msg334107
2019-01-20 21:42:33remi.lapeyresetmessages: + msg334104
2019-01-20 21:34:03pablogsalsetkeywords: patch, patch, patch

messages: + msg334103
2019-01-20 20:43:00pablogsalsetmessages: - msg334099
2019-01-20 20:42:38pablogsalsetkeywords: patch, patch, patch

messages: + msg334100
2019-01-20 20:41:47pablogsalsetkeywords: patch, patch, patch

messages: + msg334099
2019-01-18 21:39:45eric.smithsetkeywords: patch, patch, patch

messages: + msg334014
2019-01-18 21:27:32terry.reedysetkeywords: patch, patch, patch

messages: + msg334012
versions: + Python 3.8, - Python 3.4
2019-01-18 21:09:37gdr@garethrees.orgsetkeywords: patch, patch, patch

messages: + msg334010
2019-01-18 18:36:12pablogsalsetkeywords: patch, patch, patch

messages: + msg334002
2019-01-18 18:19:15gdr@garethrees.orgsetkeywords: patch, patch, patch
nosy: + gdr@garethrees.org
messages: + msg334000

2019-01-17 09:01:08remi.lapeyresetnosy: + remi.lapeyre
messages: + msg333831
2019-01-17 02:14:07pablogsalsetmessages: - msg333815
2019-01-17 02:13:52pablogsalsetkeywords: patch, patch, patch

messages: + msg333816
2019-01-17 02:12:11pablogsalsetkeywords: patch, patch, patch
nosy: + pablogsal
messages: + msg333815

2019-01-17 01:31:50rhettingersetkeywords: patch, patch, patch

messages: + msg333810
2019-01-17 01:08:53pablogsalsetpull_requests: - pull_request11267
2019-01-17 01:08:42pablogsalsetpull_requests: - pull_request11268
2019-01-17 00:43:56belopolskysetkeywords: patch, patch, patch

messages: + msg333806
2019-01-17 00:36:22pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request11268
2019-01-17 00:36:07pablogsalsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11267
2019-01-17 00:35:51pablogsalsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11266
2013-06-17 06:56:43martin.pantersetnosy: + martin.panter
2013-05-31 22:13:41belopolskysetnosy: + belopolsky
messages: + msg190419
2013-01-27 01:41:47terry.reedysetnosy: + terry.reedy
2013-01-26 17:19:30tshepangsetnosy: + tshepang
2013-01-21 01:13:40eric.smithsetnosy: + eric.smith
messages: + msg180329
2013-01-20 23:28:41rhettingersetassignee: rhettinger
2013-01-20 21:48:33christian.heimessetnosy: + christian.heimes
messages: + msg180321
2013-01-20 21:39:12rhettingercreate