New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tokenize module w/ coding cookie #38293
Comments
The tokenize module should honor the coding cookie in a |
Logged In: YES I don't think I will do anything about this anytime soon, so |
This issue is currently causing test_tokenize failures in Python 3.0. Does this still seem like a good idea to everyone? |
In 3k, the tokenize module should definitely return strings, and, in For 2.6, I wouldn't mind if it were changed incompatibly so that it |
I've attached a patch to test_tokenizer.py and a bunch of text files The existing implementation defines roundup() in the doctest, then FWIW, the file that causes the exception is test_doctest2.py as it So, the approach this patch takes is to drop the 'pick ten random test I'll move on to a patch to tokenizer.py now, but this patch is ok to |
Hmm, I take it multiple file uploads aren't supported. I don't want to |
Made quite extensive changes to tokenize.py (with tests) for Py3k. This |
Michael, is the disappearance of the generate_tokens function in the new |
That was 'by discussion with wiser heads than I'. The existing module As 'tokenize' is really the ideal name for the main entry point for the |
Is it worth keeping generate_tokens as an alias for tokenize, just So if I understand correctly, the readline argument to tokenize |
Sorry---ignore the last comment; if readline() doesn't supply bytes I'll try thinking first and posting later next time. |
Tested patch on Win x86/x64 2k8, XP & FreeBSD 6.2, +1. |
With the patch, ./python.exe Lib/test/regrtest.py test_tokenize fails for me with the following output: Macintosh-2:py3k dickinsm$ ./python.exe Lib/test/regrtest.py test_tokenize *** lines 2-5 of actual output doesn't appear in expected output after line 1: 1 test failed: I get something similar on Linux. |
If you remove the following line from the tests (which generates print('testing: %s' % path, end='\n') |
*Full* patch (excluding the new dependent test text files) for Python 3. If it breaks anything blame Trent... ;-) |
All tests pass for me on OS X 10.5.2 and SuSE Linux 10.2 (32-bit)! |
Exactly so - that was the primary rationale for renaming it. It |
This was fixed in trunk in r61573, and merged to py3k in r61982. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: