This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Star expression in comprehension wrongly indicates to use or_expression after the star
Type: Stage: patch review
Components: Documentation Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arthur-Milchior, Dennis Sweeney, docs@python
Priority: normal Keywords: patch

Created on 2021-10-28 14:42 by Arthur-Milchior, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 29303 open Arthur-Milchior, 2021-10-29 04:26
Messages (6)
msg405187 - (view) Author: Arthur Milchior (Arthur-Milchior) * Date: 2021-10-28 14:42
According to the current documentation, I believe there is an error in
list_display, set_display and dict_display.
According to list display, this is a valid expression
[* l1 ^ l2]
parsed as
list_display:
  "["
  starred_list:
    starred_item:
      "*"
      or_expr:
        xor_expr:
          xor_expr: "l1"
          "^"
          and_expr: "l2"
  "]"  

I add the full derivation from and_expr to l2 at the end of the report, it's not important for this topic. 

The same issue is present for set. For dictionary it is also present, but instead the error is in key_datum.

I have no idea what would be the proper correction.
Indeed [*l1+l2] is valid (in 3.9) at least. So it's not like the element after a star is restricted to be an atom.

At the very least, I believe it should be clearly indicated that, contrary to what the name indicates, "or_expr" do not mean that we accept any or, nor any expression. It just mean that 
*some_complex_expression
is interpreted as
* (some_complex_expression)
and that it is valid iff some_complex_expression is valid



            shift_expr:
              a_expr:
                m_expr:
                  power:
                    primary:
                      atom:
                        identifier:
                          "l2"
msg405188 - (view) Author: Arthur Milchior (Arthur-Milchior) * Date: 2021-10-28 14:50
https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-expression is wrong
This line state:
`expression             ::=  conditional_expression | lambda_expr`
and it is a problem because, by the way sphinx understand reference, this mean that any people who use the token `expression` will be linked to this particular line.
For example in https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-key_datum
`key_datum          ::=  expression ":" expression | "**" or_expr`
while clearly a key_datum is not expecting only a conditional expression or a lambda expression.

Admittedly, you can derive, from  expression:
* conditional_expression
* or_test
* and_test
* not_test
* comparison
* or_expr
* xor_expr
* and_expr
* shift_expr
* a_expr
* m_expr
* u_expr
* power
* primary
* atom

in this order, so in practice you can probably derive most of the expressions. However, I doubt that this is what the documentation meant here, and clearly not practical.
msg405191 - (view) Author: Arthur Milchior (Arthur-Milchior) * Date: 2021-10-28 14:54
Message 405188 was supposed to be another bug report. My bad.
msg405288 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-10-29 06:33
Hi Arthur,

I agree that the line about "name ::= othername" could be more helpful.

I think an affirmative example would be better than a warning against depending on variable names, because the top of the page mentions that BNF is being used and IMHO the variable names are consistent with how BNF is typically used, so negative/warning language might be more confusing. I'm also not sure a note saying "read the documentation" in the documentation will help people read the documentation ;)

Maybe an example could be something like:

---------
In this and the following chapters, extended BNF notation will be used to describe syntax (but not tokenization/lexical analysis). Extended BNF uses syntax rules like the following:

    a_expr ::=  m_expr | a_expr "+" m_expr | a_expr "-" m_expr

This rule indicates that an a_expr ("a" is for addition, but subtraction has the same precedence) is one of the following:

    * Just an m_expr (the "m" is for multiplication-like operations)
    * A smaller a_expr, followed by a "+" sign, followed by an m_expr
    * A smaller a_expr, followed by a "-" sign, followed by an m_expr

For concreteness, given that `x * y`, `x / y`, `a[i]` and `foo.bar` are instances of `m_expr`, the rule implies that `x * y`, `x / y + a[i]`, and (recursively) `foo.bar + x * y - a[i]` are all instances of `a_expr`.

---------------

Maybe there could be a link to a more-detailed BNF tutorial/reference.
msg405571 - (view) Author: Arthur Milchior (Arthur-Milchior) * Date: 2021-11-03 00:37
Thank you very much Dennis. For two reasons.

First, because I like a lot what you wrote. I suspect that it would have saved me time and make things less confusing. 
That's not certain, because at the time I was reading the documentation of or_expr, I may have skipped or forgotten the part about BNF.
A solution - that would require a bigger change - is that the various blocks containing rules have a small link back to the text you wrote, so that if a reader reads the rule directly, they can find out the meta information they need to know about rules.

I guess that it's technically possible to do it with an extra sphinx rule, but may not be trivial to do, and it's not clear how such a link and text should look like in practice, as it must be noticeable the first time and easy to ignore next time.


Thanks also because I'm happy, and slightly relieved, that my concerns are taken seriously. I admit that I feared that I would get as answer something such as:
> actually, if you knew BNF, it should be clear that the actual meaning of or_expr is ....

And the fact that the answer I got tries to actually clarify the documentation for people wanting to learn or review Python without being expert in programming language theory is really great.

Do you open a PR with your suggested test or do I update mine?
msg405574 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-11-03 01:05
Feel free to use anything I wrote in a PR and make any revisions/edits you want.
History
Date User Action Args
2022-04-11 14:59:51adminsetgithub: 89809
2021-11-03 01:05:41Dennis Sweeneysetmessages: + msg405574
2021-11-03 00:37:10Arthur-Milchiorsetmessages: + msg405571
2021-11-02 16:13:39terry.reedysetversions: - Python 3.6, Python 3.7, Python 3.8
2021-10-29 06:33:40Dennis Sweeneysetnosy: + Dennis Sweeney
messages: + msg405288
2021-10-29 04:26:52Arthur-Milchiorsetkeywords: + patch
stage: patch review
pull_requests: + pull_request27571
2021-10-28 14:54:01Arthur-Milchiorsetmessages: + msg405191
2021-10-28 14:51:55Arthur-Milchiorsettitle: "expression" is erroneous in the doc -> Star expression in comprehension wrongly indicates to use or_expression after the star
2021-10-28 14:50:56Arthur-Milchiorsetmessages: + msg405188
title: The documentation wrongly uses or_expr for star expresion in displays -> "expression" is erroneous in the doc
2021-10-28 14:42:22Arthur-Milchiorcreate