New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Py3K cannot run as python -S
#45608
Comments
If Py3K is executed without importing site, it fails horribly. This is The setting of sys.stderr is especially bad as exception printing |
I've some code around which sets sys.stdin, out and err in C code. The |
I've carefully checked and tested the initstdio() method. I'm sure that I've also added a PyErr_Display() call to Py_FatalError(). It's still |
Very cool! Some final suggestions: + Py_FatalError("Py_Initialize: can't initialize sys standard" Break this differently:
Don't call open() with keyword arg for newline="\r"; open() takes I would change the error handling to avoid the 'finally' label, like this: if (0) {
error:
status = -1;
Py_XDECREF(args);
} I would add a comment "see initstdio() in pythonrun.c" to the Thanks for doing this!!! |
Guido van Rossum wrote:
I need a way to open a file with a specific encoding for PyObject * Some people may also miss the PyFile_FromString() function but that's a PyFile_FromFileEx(stdin, "<stdin>", "r", fclose, -1, NULL, "\n") I kinda like it :)
Wow, that's tricky. :)
That sounds like a good idea! The code for const char *PyTokenizer_FindEncoding(FILE *fp) and Christian |
The patch is a combined patch for the imp.find_module() problem and |
Changes since last patch:
|
Christian Heimes wrote:
I found another bug in Python/import.c:call_find_method. The function if (strchr(fdp->mode, 'b') == NULL) {
/* Python text file, get encoding from tokenizer */
encoding = PyTokenizer_FindEncoding(fp);
encoding = (encoding != NULL) ? encoding :
PyUnicode_GetDefaultEncoding();
} Christian |
Does this mean I should hold off reviewing the patch? On 10/16/07, Christian Heimes <report@bugs.python.org> wrote:
|
Update since last patch
Please review the patch. |
I will take this. |
Thanks, Guido. |
Committed revision 58553 (with minor tweaks only). Thanks! |
FWIW, on Mac I get six failures (all these pass on Linux): test_cmd_line test_inspect test_modulefinder test_pyclbr test_quopri test_runpy |
Yeah, I just noticed this as well. is it definitely this change? On 10/19/07, Guido van Rossum <report@bugs.python.org> wrote:
|
I don't have a Mac at my disposal any more. :( Can you attach the output of the failing tests to the bug report? Maybe |
runpy is failing because pkgutil is failing because it is giving Traceback (most recent call last):
File "/Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py", line 97,
in _run_module_as_main
loader, code, fname = _get_module_details(mod_name)
File "/Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py", line 82,
in _get_module_details
code = loader.get_code(mod_name)
File "/Users/drifty/Dev/python/3.x/pristine/Lib/pkgutil.py", line
275, in get_code
self.code = compile(source, self.filename, 'exec')
File "/Users/drifty/Dev/python/3.x/pristine/Lib/tokenize.py", line 2
"UR'''": single3prog, 'UR"""': double3prog,
^
IndentationError: unexpected indent That bad line is the first line in the 'source' variable being passed On 10/19/07, Christian Heimes <report@bugs.python.org> wrote:
|
It looks like the file object returned by imp.find_module() has its read Re-opening. |
> runpy is failing because pkgutil is failing because it is giving
> compile() part of a source file instead of the entire thing::
>
> Traceback (most recent call last):
> File "/Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py", line 97,
> in _run_module_as_main
> loader, code, fname = _get_module_details(mod_name)
> File "/Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py", line 82,
> in _get_module_details
> code = loader.get_code(mod_name)
> File "/Users/drifty/Dev/python/3.x/pristine/Lib/pkgutil.py", line
> 275, in get_code
> self.code = compile(source, self.filename, 'exec')
> File "/Users/drifty/Dev/python/3.x/pristine/Lib/tokenize.py", line 2
> "UR'''": single3prog, 'UR"""': double3prog,
> ^
> IndentationError: unexpected indent
>
> That bad line is the first line in the 'source' variable being passed
> to compile(). So somewhere the beginning of the source file is being
> chopped up. Could you please comment out the PyTokenizer_FindEncoding(fp) call in Christian |
Brett Cannon wrote:
That's strange. It should never read more than 2 lines of a file. I char *
PyTokenizer_FindEncoding(FILE *fp) {
struct tok_state *tok;
char *p_start=NULL, *p_end=NULL;
if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) {
rewind(fp);
return NULL;
}
while(((tok->lineno <= 2) && (tok->done == E_OK))) {
PyTokenizer_Get(tok, &p_start, &p_end);
}
rewind(fp);
return tok->encoding;
} |
PyTokenizer_FindEncoding() is the culprit. If I comment it out in |
It's unrelated to the problem but Parser/tokenizer.c:1619 has a minor bug.
(tok->lineno < 2) is sufficient. See line 516 Christian |
OK, for some reason, when PyTokenizer_FindEncoding() is called and the But I am not sure what the magical size is as creating a file of 4095, |
I've made a short unit tests which tests a large file with and w/o -- |
Brett Cannon wrote:
Maybe rewind() doesn't do what is should do? Could you replace the w/o the rewind() calls in PyTokenizer_FindEncoding I'm getting an error Traceback (most recent call last):
File "Lib/test/test_imp.py", line 66, in <module>
test_main()
File "Lib/test/test_imp.py", line 62, in test_main
ImportTests,
File "/home/heimes/dev/python/py3k/Lib/test/test_support.py", line
541, in run_unittest
_run_suite(suite)
File "/home/heimes/dev/python/py3k/Lib/test/test_support.py", line
524, in _run_suite
raise TestFailed(err)
test.test_support.TestFailed: Traceback (most recent call last):
File "Lib/test/test_imp.py", line 50, in test_issue1267
self.assertEqual(fd.tell(), 0)
AssertionError: 4096 != 0 The number is looking familiar, isn't it? :) Is it the default buffer Christian |
I checked in your |
Nope, didn't do it. I also checked using gdb a few minutes ago and |
OK, I think I might have a solution, and it is really stupid. I will |
Here's keeping my fingers crossed. I do note that the new code still leaks the encoding string which is BTW, the test_inspect failure I saw was shallow; I'd renamed the |
58557 as the fix. Yes, it was stupid on OS X's part and I was lucky to Basically OS X does not like it when you do stuff with a file pointer This suggests that perhaps we should standardize on file pointers or |
Brett Cannon wrote:
rewind() it used couple of times in the Python code. Have you checked if Thx for finding the bug :) Great work! Christian |
On 10/19/07, Christian Heimes <report@bugs.python.org> wrote:
No. I am still rather frustrated that was the problem. The only reason I feel safe with this solution is that the file
Thanks. I just wish this whole ordeal had not been necessary (filed a |
Brett Cannon wrote:
The bug is rather strange. I would have imagined that fseek() and IronPython might be fun but I've hacked PythonDotNet. You get the worst |
Thanks for persevering!!! The dangers of switching between fileno(fp) and fp are actually well I think there's still a bug or two lurking in this area: first, each You're right that a lot of this could be avoided if we used file Rewriting everywhere that uses PyFile_FromFile[Ex] to use file |
Sorry, I was wrong about the encoding leak; you fixed that. The |
Guido van Rossum wrote:
If I understand you right you want to alter the interface of PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (close)(FILE), to PyFile_FromFileEx(int fd, char *name, char *mode, int (close)(FILE), ? I could write a patch and remove int (close)(FILE), too. It's no Christian |
On 10/20/07, Christian Heimes <report@bugs.python.org> wrote:
Yes, I think you got it. -Brett |
Guido, Brett, how much of the code should I change? import.c is mainly Christian |
I think find_module() should return a file descriptor (fd), not a I recommend changing the name of the API used to turn a fd into file |
Guido van Rossum wrote:
K, I understand.
Is PyFile_FromFd and PyFile_FromFdEx fine with you or do you prefer a I've another question. The os.tmpfile method is using tmpfile() which static PyObject *
posix_tmpfile(PyObject *self, PyObject *noargs)
{
FILE *fp;
int fd; fp = tmpfile();
if (fp == NULL)
return posix_error();
fd = dup(fileno(fp));
if (fd == -1)
return posix_error();
fclose(fp);
return PyFile_FromFD(fd, "<tmpfile>", "w+b");
} |
FYI: http://www.gnu.org/software/libc/manual/html_mono/libc.html#Stream_002fDescriptor-Precautions Christian |
Let's have a single function PyFile_FromFd(fd, name, mode, buffering,
You should dup it; if you don't dup it, the fd will be closed when you |
I'd have to touch most functions in import.c and related files to change The new patch adds PyFile_FromFd and removes the other PyFile_FromFile* |
Do you have access to Windows? I believe it doesn't have dup(). :-( |
Guido van Rossum wrote:
I've an old laptop with Win2k at my disposal but it has no VS yet. Can |
Sorry, I'm as green as you when it comes to these. :-( But I believe I was mistaken about dup() not existing on Windows; Alas, my family won't let me use the computer for more than a minute |
OK, checked in. You might want to compare what I checked in to your patch; I did a few Committed revision 58587. |
Guido van Rossum wrote:
Ah I see that you prefer to keep assignment and check against NULL/-1 on I had the lseek() in PyTokenizer_FindEncoding() because I prefer By the way I got Windows, VS 2003 and several SDKs installed in VMWare |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: