OK, more specifically, here's the kind of situation where items in a set are mutable:

company = Session.query(Company).first()

# company.employees is a set()
company.employees

# each employee references the parent company
for e in company.employees:
    assert e.company is company

So nothing is mutated relationally in this case.   It's just a plain old bidirectional structure.    If two objects are related via many-to-many, then you might have a set in both directions.   I'm not sure if this specific situation produces the pickle bug, however, it's been awhile since I've seen which specific case causes the problem.   But there are mutable items in a set in these examples and it doesn't seem unreasonable to me.