classification
Title: Comprehension doc doesn't mention buggy class scope behavior
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.8, Python 3.7
process
Status: closed Resolution:
Dependencies: Superseder: Unintuitive error when using generator expression in class property
View: 26951
Assigned To: docs@python Nosy List: Aaron Hall, bzip2, docs@python, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2018-12-31 10:48 by bzip2, last changed 2019-07-19 19:00 by serhiy.storchaka. This issue is now closed.

Messages (4)
msg332808 - (view) Author: bzip2 (bzip2) Date: 2018-12-31 10:48
The sections on list, set and dict comprehensions in the tutorial on data structures (ref. 1) state repeatedly that they are equivalent to for loops, but do not mention that this is not true in classes. In fact, the example used for nested list comprehensions (section 5.1.4) will work in a function, but not in a class. Similarly, there seems to be no mention of this scope "limitation" in the tutorial on classes (ref. 2), despite a section on scopes and namespaces (section 9.2) and another that mentions list comprehensions (section 9.10). The scope "limitation" is mentioned at the end of a section on resolution of names on a page about the execution model in the reference guide (ref. 3), and of course in various forums, where people may perhaps eventually find them after wasting time trying to figure out what they've done wrong.

If comprehensions are "equivalent" to for loops only under certain conditions (in a class, but only in a class, only one variable from outside the comprehension is accessible in the comprehension, and it must be the outermost iterable), they are not equivalent and should not be described as such. This "limitation" should be mentioned prominently wherever comprehensions are described, since both classes and comprehensions are presumably common constructs. When people read "is equivalent to" without a qualifier, they assume "is always equivalent to".

Returning to section 9.10 in ref. 2, the unique_words example is misleading because it strongly implies that nested for loops in a comprehension should work in a class. Since that's only true in some cases, the example should be qualified. More broadly, because that tutorial is about classes, the relevance of the last three sections should be revisited.

As an aside, I agree with the developers who consider this scope "limitation" a bug and not (paraphrasing) "just how the language works", since the exact same two lines of code, which depend on no other variables or functions, work in a function or module but not in a class.

1. https://docs.python.org/3/tutorial/datastructures.html
2. https://docs.python.org/3/tutorial/classes.html
3. https://docs.python.org/3/reference/executionmodel.html
msg333037 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-01-05 04:20
We should remove the "equivalent to for-loops" wording for list comprehensions.  That is a hold-over from 2.7 where it used to be true.  That said, the list comprehension section is too early in the tutorial to go into scopes and generators, so a full explanation will need to be deferred.

> As an aside, I agree with the developers who consider 
> this scope "limitation" a bug and not (paraphrasing) 
> "just how the language works", since the exact same two
> lines of code, which depend on no other variables or 
> functions, work in a function or module but not in a class.

If you view classes as just another nested scope, I can see why you might think the behavior is buggy, limited, or undesirable.  That however is not how the language is designed.  Think about why methods have to reference class variables using "self.classvar" rather than just "classvar".  When the methods run, they have access to their own locals() and to the module level globals().  To access the locals() for the class, they need use "self.classvar" or "classname.classvar".  This is central to how python works .  There are two separate lookup chains, one for variables (locals -> nested scopes -> globals -> __builtins__ -> NameError) and another for attributes (inst_dict -> class_dict -> parent_class_dict -> AttributeError).

Guido intentionally shifted list comprehensions to work like generator expressions.  In a class scope, they behave like methods in that they have access to the outer globals but no direct access to the locals() in the class.
msg333061 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-01-05 15:30
See also issue3692.
msg348193 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-19 19:00
As a documentation issue this is a duplicate of issue26951.
History
Date User Action Args
2019-07-19 19:00:22serhiy.storchakasetstatus: open -> closed
superseder: Unintuitive error when using generator expression in class property
messages: + msg348193

stage: resolved
2019-04-03 16:59:37Aaron Hallsetnosy: + Aaron Hall
2019-01-06 14:06:01r.david.murraysetfiles: - keyfile.png
2019-01-06 14:05:33r.david.murraysetnosy: - barry, r.david.murray
type: security -> behavior
components: - email
2019-01-05 21:39:22Seval Geyiksetfiles: + keyfile.png
nosy: + r.david.murray, barry
type: behavior -> security
components: + email
2019-01-05 15:30:26serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg333061
2019-01-05 04:20:45rhettingersetnosy: + rhettinger
messages: + msg333037
2019-01-04 19:31:39terry.reedysettitle: documentation of list, set & dict comprehension make no mention of buggy class scope behavior -> Comprehension doc doesn't mention buggy class scope behavior
versions: + Python 3.8, - Python 2.7, Python 3.4, Python 3.5, Python 3.6
2018-12-31 10:48:00bzip2create