classification
Title: logging.__init__ assumes that __file__ is always set
Type: Stage: resolved
Components: Interpreter Core, Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, lemburg, python-dev, vinay.sajip
Priority: normal Keywords:

Created on 2014-06-10 21:15 by lemburg, last changed 2014-06-15 17:35 by vinay.sajip. This issue is now closed.

Messages (16)
msg220196 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-10 21:15
It is not when freezing the logging package, so any use of the logging package fails:

>>> import logging
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<pyrun>/importlib/_bootstrap.py", line 2237, in _find_and_load
  File "<pyrun>/importlib/_bootstrap.py", line 2226, in _find_and_load_unlocked
  File "<pyrun>/importlib/_bootstrap.py", line 1200, in _load_unlocked
  File "<pyrun>/importlib/_bootstrap.py", line 1129, in _exec
  File "<pyrun>/importlib/_bootstrap.py", line 1359, in exec_module
  File "<pyrun>/logging/__init__.py", line 61, in <module>
NameError: name '__file__' is not defined

This is the code in question:

#
# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
if hasattr(sys, 'frozen'): #support for py2exe
    _srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
else:
    _srcfile = __file__
_srcfile = os.path.normcase(_srcfile)

Note the special case for py2exe. But this doesn't really help, since frozen modules don't have the __file__ attribute set - at least not when generated with Tools/freeze.

PS: This is with Python 3.4.1.
msg220201 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-10 22:05
The issue is similar to Issue20884.
msg220241 - (view) Author: Roundup Robot (python-dev) Date: 2014-06-11 07:04
New changeset 11a920a26f13 by Vinay Sajip in branch '3.4':
Issue #21709: Remove references to __file__ when part of a frozen application.
http://hg.python.org/cpython/rev/11a920a26f13

New changeset 149cc6364180 by Vinay Sajip in branch 'default':
Closes #21709: Merged fix from 3.4.
http://hg.python.org/cpython/rev/149cc6364180
msg220246 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-11 08:46
Hi Vinaj,

thanks for the patch, but it doesn't really help outside of py2exe. The sys.frozen flag is not an official Python API and it's unlikely to become one, since you can freeze the whole application or just parts of it, which sys.frozen would not be able to address.

Instead, the modules in the stdlib should be aware of the fact that __file__ is not always available and provide fallback solutions.

Could you please use a fix that works for Python tools in general ?

E.g. instead of doing an equal test inf .findCaller() it may be better to use a regular expression or you could patch the __init__ module's co_filename into the module as _srcfile (after it's fully initialized).

Thanks.
msg220247 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-06-11 09:25
> Could you please use a fix that works for Python tools in general?

I suggested an alternative implementation altogether in Issue #16778, but it was suggested that we wait for frame annotations. I'm not sure what the schedule for that is.

> The sys.frozen flag is not an official Python API and it's unlikely to become one

Would using imp.is_frozen('logging') rather than hasattr(sys, 'frozen') meet your requirement here? I'm not saying it's the ideal solution, but perhaps it will do until frame annotations arrive and we can avoid using filenames altogether?
msg220248 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-11 09:54
On 11.06.2014 11:25, Vinay Sajip wrote:
> 
> Vinay Sajip added the comment:
> 
>> Could you please use a fix that works for Python tools in general?
> 
> I suggested an alternative implementation altogether in Issue #16778, but it was suggested that we wait for frame annotations. I'm not sure what the schedule for that is.
> 
>> The sys.frozen flag is not an official Python API and it's unlikely to become one
> 
> Would using imp.is_frozen('logging') rather than hasattr(sys, 'frozen') meet your requirement here? I'm not saying it's the ideal solution, but perhaps it will do until frame annotations arrive and we can avoid using filenames altogether?

I don't think any of this is needed here. _srcfile is only used to
identify the caller's stack frame and needs to be set to the co_filename
of the stack frame used by the logging.__init__ module.

Here's a sketch of what I had hinted at in my last reply:

def _get_module_filename():
    return getLogger.func_code.co_filename

You simply use the .co_filename attribute of one of the module's functions
to get useable value for __file__.
msg220250 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-06-11 10:32
> _srcfile is only used to identify the caller's stack frame

Not quite. It's also used to indicate whether findCaller() should be called at all: setting it to None avoids calling findCaller(), which might be desirable in some performance-sensitive scenarios.

So if you mean "just call _get_module_filename() instead of accessing _srcFile", that won't do. If you mean "set _srcFile to the return value of _get_module_filename()", that might work, if I e.g. move the _srcFile definition to after addLevelName (say) and do just

_srcFile = addLevelName.__code__.co_filename

How does that sound?
msg220252 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-11 10:45
On 11.06.2014 12:32, Vinay Sajip wrote:
> 
> Vinay Sajip added the comment:
> 
>> _srcfile is only used to identify the caller's stack frame
> 
> Not quite. It's also used to indicate whether findCaller() should be called at all: setting it to None avoids calling findCaller(), which might be desirable in some performance-sensitive scenarios.
> 
> So if you mean "just call _get_module_filename() instead of accessing _srcFile", that won't do. If you mean "set _srcFile to the return value of _get_module_filename()", that might work, if I e.g. move the _srcFile definition to after addLevelName (say) and do just
> 
> _srcFile = addLevelName.__code__.co_filename
> 
> How does that sound?

That's what I meant, yes. Please also add some comment explaining
why this is done in this way.

FWIW: Given that __file__ is not always set, it may be worthwhile
introducing some generic helper to the stdlib which uses the
.co_filename attribute to get the compile time filename as fallback
in case __file__ is not set.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com
msg220254 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-06-11 11:22
> Please also add some comment explaining why this is done in this way.

Natürlich :-)

> it may be worthwhile introducing some generic helper to the stdlib

Wouldn't you have to pass in a function (or code object) from a specific module, though? It seems more logical to have __file__ set, even for frozen modules (after all, if it's there in a code object's co_filename, is there some reason it shouldn't be exposed as a module attribute? (Even though it isn't at the moment.)
msg220256 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-11 11:32
On 11.06.2014 13:22, Vinay Sajip wrote:
> 
> Vinay Sajip added the comment:
> 
>> Please also add some comment explaining why this is done in this way.
> 
> Natürlich :-)

Prima :-)

>> it may be worthwhile introducing some generic helper to the stdlib
> 
> Wouldn't you have to pass in a function (or code object) from a specific module, though? It seems more logical to have __file__ set, even for frozen modules (after all, if it's there in a code object's co_filename, is there some reason it shouldn't be exposed as a module attribute? (Even though it isn't at the moment.)

Well, I guess passing in a reference to the module would suffice. The
function could then look around for functions, methods, etc. to find
a usable code object.

I agree that having a __file__ attribute in frozen modules would
be helpful, since it's obviously not widely known that this
attribute does not always exist (just grep the stdlib as example,
in particular the test suite).

The module object's PyModule_GetFilenameObject() even reports a missing
__file__ attribute as a SystemError.

Perhaps something to discuss on python-dev.
msg220363 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-12 16:17
While the current patch does not resolve the issue, I'm leaving the issue closed and have instead opened a new Issue21736 which tracks the idea to add a __file__ attribute to frozen modules per default.
msg220370 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-06-12 16:46
> While the current patch does not resolve the issue, I'm leaving the issue closed

That's fine - I will implement the changes we discussed in this issue, even if it's something of a stop-gap.
msg220393 - (view) Author: Roundup Robot (python-dev) Date: 2014-06-12 22:38
New changeset bec6f18dd636 by Vinay Sajip in branch '3.4':
Issue #21709: Improved implementation to cover the frozen module case.
http://hg.python.org/cpython/rev/bec6f18dd636

New changeset bd44ad77013a by Vinay Sajip in branch 'default':
Issue #21709: Merged update from 3.4.
http://hg.python.org/cpython/rev/bd44ad77013a
msg220587 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-06-14 22:01
> addLevelName.__code__.co_filename

Isn't __code__ implementation-specific?
msg220601 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-06-15 01:04
> Isn't __code__ implementation-specific?

It is, but ISTM it's worth getting a resolution for #21736 before another patch for this issue is developed.
msg220655 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-06-15 17:35
> Isn't __code__ implementation-specific?

Further data point: the contents of __code__ might be implementation-specific (as there are no other Python 3 implementations), but the 2.x equivalent, func_code, is to be found in CPython, Jython, IronPython and PyPy, and in each case it has a co_filename attribute pointing to a source file. Other internal details no doubt differ.
History
Date User Action Args
2014-06-15 17:35:00vinay.sajipsetmessages: + msg220655
2014-06-15 01:04:54vinay.sajipsetmessages: + msg220601
2014-06-14 22:01:29eric.snowsetnosy: + eric.snow
messages: + msg220587
2014-06-12 22:38:32python-devsetmessages: + msg220393
2014-06-12 16:46:04vinay.sajipsetmessages: + msg220370
2014-06-12 16:17:06lemburgsetmessages: + msg220363
2014-06-11 11:33:00lemburgsetmessages: + msg220256
2014-06-11 11:22:49vinay.sajipsetmessages: + msg220254
2014-06-11 10:45:52lemburgsetmessages: + msg220252
2014-06-11 10:32:35vinay.sajipsetmessages: + msg220250
2014-06-11 09:54:50lemburgsetmessages: + msg220248
2014-06-11 09:25:20vinay.sajipsetmessages: + msg220247
2014-06-11 08:46:01lemburgsetmessages: + msg220246
2014-06-11 07:04:28python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg220241

resolution: fixed
stage: resolved
2014-06-11 00:50:34berker.peksagsetnosy: + vinay.sajip
2014-06-10 22:05:30lemburgsetmessages: + msg220201
2014-06-10 21:15:54lemburgcreate