Title: enable discovery of class source code in IPython interactively defined classes
Type: enhancement Stage: resolved
Components: Interpreter Core, Library (Lib) Versions: Python 3.8
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.Pozdeev, pablogsal, taleinat, xtreak
Priority: normal Keywords: patch

Created on 2018-06-11 06:16 by t-vi, last changed 2019-06-07 14:54 by t-vi. This issue is now closed.

File name Uploaded Description Edit
class-filename.diff t-vi, 2018-06-11 06:16
Pull Requests
URL Status Linked Edit
PR 13894 closed t-vi, 2019-06-07 14:30
Messages (14)
msg319274 - (view) Author: Thomas Viehmann (t-vi) * Date: 2018-06-11 06:16

thank you for Python!
In IPython (or Jupyter), `inspect.getsource` is able to retrieve the source code for functions but not classes.
The fundamental reason is that for functions, a "filename" (input reference rather) is available via fn.__code__.co_filename, while for classes, there is no equivalent (and __module__ is __main__, so no "filename" there).
This could be helped by providing a "__filename__" (or however named) attribute in classes.
Some digging in the Python code suggests that Python/bltinmodule.c might be a good place to resolve this, namely in function builtin___build_class__. In there, there is the func object and so ((PyCodeObject*) PyFunction_GET_CODE(func))->co_filename has the filename.
Then one would copy that to an appropriate item in the namespace ns.

I do have a patch (against python 3.6), but as this is my first attempt at digging in the internals, it is probably more than just a bit raw (I see test failures in test_dbm test_lib2to3 test_pydoc test_site, at least test_pydoc is due to the patch).

Best regards

msg319650 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-06-15 19:21
In [1]: import logging

In [2]: logging.Logger.__module__
Out[2]: 'logging'

In [4]: import sys

In [6]: sys.modules[logging.Logger.__module__].__file__
Out[6]: 'C:\\Program Files\\Python36\\lib\\logging\\'
msg319654 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-06-15 19:25
What do you mean by "class source code" anyway? A class doesn't actually contain any executable code (and there's no code object associated with it).
msg319659 - (view) Author: Thomas Viehmann (t-vi) * Date: 2018-06-15 19:46
Apologies for not being clear.
So I wrote:

In IPython (or Jupyter), `inspect.getsource` is able to retrieve the source code for functions but not classes.

By this I mean:

import inspect

class X:


Note that:
- it won't work in the vanilla python interpreter,
- it does work for functions in Jupyter/IPython currently,
- it works for classes (and functions) in modules.
What I would like is to have it work for classes in Jupyter/IPython.

By the way: Would a PR be more convenient than a patch?
msg319660 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-06-15 19:54
I got that part, thank you.

I can't get 1)what you're expecting as an output and 2)why this should work the way you're suggesting 'cuz functions and classes are very different.

In particular, classes can be modified dynamically, unlike functions. So if you just print out the lines from the file with the definition (which is what getsource() does), it may not be what the interpreter is actually using.
msg319664 - (view) Author: Thomas Viehmann (t-vi) * Date: 2018-06-15 20:14
So I want "inspect.showsource" to have the same output whether my class has been defined in a .py or in my Jupyter notebook.

I appreciate there are limitations to what "inspect.showsource" can do, but I am not so sure how that creates uncertainty my expectation of the output when it should just be the same as the current behaviour when the class is defined in a .py module.
msg319692 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-06-15 23:36
Oh, I see, you want to be able to get the source for code entered into the interactive session.

IPython does this by creating a separate fake "file name" for every input and adding corresponding entries for them into `linecache.cache'. This doesn't work for classes 'cuz for them, inspect.getfile() rather looks at __module__.__file__ , and there's no single __file__ for __main__.

Now, there are two concerns here:

* This will lead to linecache being polluted with old input, effectively a memory leak. Unlike vanilla's readline cache which has a maximum length. Not much concern for IPython which only needs to be good enough for its target audience and can be dropped if it doesn't work for a specific scenario. But not for vanilla which needs to work reliably in all cases.

* There indeed is a discrepancy between functions that link to source directly and classes that rely on heuristics for that (inspect.findsource() searches the file with a regex for the class' definition, how absurd is that?). Adding __file__ and __firstlineno__ _could_ help here, the `class' directive will need to add them.

    * In any case, the returned source is prone to "not necessarily what the interpreter uses", but that's the problem with how the returned source is generated, not where it's taken from.
msg328813 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-10-29 12:13
This seems to be a duplicate of issue12920. issue24491 was also a related issue closed as a duplicate of issue12920. So I propose closing this to continue the discussion in issue12920. Feel free to add in if I am missing something.
msg328815 - (view) Author: Thomas Viehmann (t-vi) * Date: 2018-10-29 12:24
Yeah, it's a shame no-one looked at the patch that seems to fix the underlying cause and now it's just a duplicate of a bug to improve error  messages.
On the up side, closing this bug will stop me from getting reminders about it.
msg328822 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-10-29 12:42
@Thomas Viehmann , as it's currently formulated, this is a duplicate because it strives to allow getting class source from past interactive input -- which, as I explained, is already possible without the patch and seems to be inappropriate for vanilla console anyway.

Your patch is rather a new class feature that provides a more robust link to its source, wherever it's located. If you reformulate your ticket to propose that instead, it will no longer be a duplicate. I'm not sure if the UI allows that, it may be better to create a new ticket.
msg328824 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-10-29 12:51
> which, as I explained, is already possible without the patch

Sorry, I myself explained in that it's not possible. The rest still stands though.
msg328831 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-10-29 14:06
Sorry about that I thought to redirect the discussion to the linked issue since it had a lot of people who might provide a better discussion about it. Feel free to reopen this if needed.
msg344945 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-06-07 14:43
Given that classes already have a '__module__' attribute, using which it is rather easy to find the filename, I don't see a good reason to add such an attribute to classes.

IMO it would be more appropriate to directly create enhancement proposal for IPython/Jupyter to enhance the capabilities of their source code retrieval.

I suggest rejecting this enhancement.
msg344947 - (view) Author: Thomas Viehmann (t-vi) * Date: 2019-06-07 14:50
Note that the module is less granular than the filename in IPython - i.e. it's useless for within notebooks.
Date User Action Args
2019-06-07 14:54:24t-visetstatus: open -> closed
stage: patch review -> resolved
2019-06-07 14:54:02t-visetnosy: - t-vi
resolution: rejected
2019-06-07 14:50:59t-visetnosy: + t-vi
messages: + msg344947
2019-06-07 14:43:58taleinatsetnosy: + taleinat
messages: + msg344945
2019-06-07 14:30:40t-visetstage: patch review
pull_requests: + pull_request13770
2019-06-07 06:18:44SilentGhostsetnosy: + pablogsal
2018-11-10 16:29:23xtreaksetstatus: closed -> open
stage: resolved -> (no value)
2018-11-10 16:18:07t-visetresolution: duplicate -> (no value)
2018-10-29 14:06:45xtreaksetmessages: + msg328831
2018-10-29 13:20:19t-visetnosy: - t-vi
2018-10-29 12:51:35Ivan.Pozdeevsetmessages: + msg328824
2018-10-29 12:42:47Ivan.Pozdeevsetmessages: + msg328822
2018-10-29 12:25:22t-visetstatus: open -> closed
stage: resolved
2018-10-29 12:24:27t-visetresolution: duplicate
messages: + msg328815
2018-10-29 12:13:16xtreaksetnosy: + xtreak
messages: + msg328813
2018-06-15 23:36:26Ivan.Pozdeevsetmessages: + msg319692
2018-06-15 20:14:02t-visetmessages: + msg319664
2018-06-15 19:54:20Ivan.Pozdeevsetmessages: + msg319660
2018-06-15 19:46:16t-visetmessages: + msg319659
2018-06-15 19:25:37Ivan.Pozdeevsetmessages: + msg319654
2018-06-15 19:21:20Ivan.Pozdeevsetnosy: + Ivan.Pozdeev
messages: + msg319650
2018-06-11 06:16:36t-vicreate