This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test for token.py, and consistency tests for tokenize.py
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Jim Fasarakis-Hilliard, ammar2, berker.peksag, r.david.murray, serhiy.storchaka
Priority: normal Keywords: easy

Created on 2017-03-14 23:03 by r.david.murray, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 681 closed ammar2, 2017-03-16 02:19
Messages (7)
msg289628 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-03-14 23:03
http://bugs.python.org/issue24622 made reminded me that a while back we added tests for the keyword module that includes a test that if you run it, you get the result that is checked in.  The same thing could be done for the token.py module.  And then we could add a cross-check test that tokenize.py has all the symbols defined as well.
msg289702 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2017-03-16 02:19
This is my first real substantial testing change so I'd appreciate all feedback. The way I did the cross-check doesn't actually rely on any of the information from the test_keyword style regeneration test. 

I think this approach is a lot simpler, and will prevent mistakes like the one seen in http://bugs.python.org/issue24622
msg289704 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-03-16 03:17
The cross check test itself doesn't depend on a regeneration, but it does depend on the information in token.h.  Meanwhile the validity of *that* is checked by regeneration in your test_token tests.  This is exactly what I had in mind :).

So, I haven't done a full review, but the quick glance I took looked good.
msg297532 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-03 04:20
See also issue30455. Don't these issues conflict one other?
msg297533 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-07-03 04:47
Yes, they are. I like the idea of generating C files from Lib/token.py. Let's see what David thinks about issue 30455.
msg297575 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-07-03 13:07
I have no opinion on that issue.  I opened this issue to fix a hole in our validation, but if that hole no longer exists after that (subsequently opened :) issue is resolved, then this isn't needed (sorry, Ammar, and thank you for your work!)
msg298192 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2017-07-12 02:22
No problem, closing this in favor of issue 30455.
History
Date User Action Args
2022-04-11 14:58:44adminsetgithub: 73998
2017-07-12 02:22:57ammar2setstatus: open -> closed
resolution: out of date
messages: + msg298192

stage: patch review -> resolved
2017-07-03 13:07:22r.david.murraysetmessages: + msg297575
2017-07-03 04:47:04berker.peksagsetnosy: + berker.peksag
messages: + msg297533
2017-07-03 04:20:14serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg297532
2017-07-03 03:38:28berker.peksagsetstage: needs patch -> patch review
2017-03-16 03:17:31r.david.murraysetmessages: + msg289704
2017-03-16 02:19:46ammar2setnosy: + ammar2
messages: + msg289702
2017-03-16 02:19:29ammar2setpull_requests: + pull_request558
2017-03-15 09:50:53Jim Fasarakis-Hilliardsetnosy: + Jim Fasarakis-Hilliard
2017-03-14 23:03:41r.david.murraycreate