classification
Title: function annotation for builtin and C function
Type: enhancement Stage: needs patch
Components: Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ramchandra Apte, amaury.forgeotdarc, benjamin.peterson, bhy, haypo, larry, richardb
Priority: normal Keywords:

Created on 2008-06-26 13:16 by bhy, last changed 2013-12-04 01:45 by larry.

Messages (18)
msg68783 - (view) Author: Haoyu Bai (bhy) Date: 2008-06-26 13:16
It is better if the function annotation(PEP 3107) can be supported by
built-in function and C function writtin in extension module, just like
the __doc__ attribute.
msg69629 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-07-13 22:29
Extension modules can use PyFunction_GetAnnotations() to access and
modify the annotations dictionary. In addition,
PyFunction_SetAnnotations() can be used to add annotations.

I added some documentation for these functions in r64934.
msg69678 - (view) Author: Haoyu Bai (bhy) Date: 2008-07-15 10:52
Sorry I haven't state the issue clearly. For this issue I mean the
built-in function should able to define an __annotations__ attribute,
just like the __doc__ attribute, but not to access it in extension module.
msg70104 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-21 12:15
PyCFunctionObject has indeed no way to store annotations. This could be
useful for extension module writers. 

The PyMethodDef structure could grow a "ml_annotations" member. A patch
is welcome!
msg70167 - (view) Author: Haoyu Bai (bhy) Date: 2008-07-23 05:16
By considering the implementing, some problems emerged. 

First of all, as we know, all CFunctionObject and their attributes are 
imutable, but the __annotations__ attribute should be a dict, and dict 
is mutable. So how to solve this?

Secondly, the annotation value can be abitrary expression, and then, 
for extension module, would it be reasonable to restrict these value to 
string?

Thanks!
msg70170 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-23 07:13
- A immmutable object may contain mutable members. Try with a tuple
containing a list.
Then, I don't think that something says that CFunctionObjects are
immutable. They don't have any modifiable attribute, until today!

- (Did I say "string"?) The new PyMethodDef::ml_annotations would not be
a char*, but a PyObject* member. If it is not possible to set it in the
static array, one could update the array in the module init function.

Anyway, for a SWIG module I think the best is to set the __annotations__
in the shadow python file. It seems more practical to build the dict there.
msg70178 - (view) Author: Haoyu Bai (bhy) Date: 2008-07-23 17:34
I think there is reason that CFunctionObjects are immutable: single 
CFunctionObject is shared by mutiple Python interpreters, so any change 
of CFunctionObject would affect other Python interpreters. Is that 
right?

If it should be immutable, then we should use something like static 
array to assign annotations to CFunctionObject, and the value also 
should be immutable, that means the value couldn't be abitrary 
PyObject. (by value I mean the value of every __annotations__ dict 
items.)

For SWIG, there's a way to bypass the Python side proxy, eg. for a 
simple C function, in the shadow module we directly 
let 'func=_cmod.func', where _cmod is the C DLL module. So the 
annotation information would be lost if we can't directly assign 
annotation to C function.
msg70179 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-07-23 17:37
There never should be multiple Python interpreters running in the same
process, though.
msg70186 - (view) Author: Haoyu Bai (bhy) Date: 2008-07-24 01:44
As I understand, at least C extension modules, which built as shared 
library, would be shared among Python interpreter in different process 
space. Is that correct?
msg70192 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-07-24 02:48
On Wed, Jul 23, 2008 at 8:44 PM, Haoyu Bai <report@bugs.python.org> wrote:

>
> Haoyu Bai <divinekid@gmail.com> added the comment:
>
> As I understand, at least C extension modules, which built as shared
> library, would be shared among Python interpreter in different process
> space. Is that correct?

The operating system should provide memory protection between processes.

>
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue3208>
> _______________________________________
>
msg70194 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-24 06:31
Shared libraries share code, not memory.

But were you talking about sub-interpreters?
http://docs.python.org/dev/c-api/init.html#Py_NewInterpreter
mod_python uses them, but see the "Caveats" section of the doc.
msg70195 - (view) Author: Haoyu Bai (bhy) Date: 2008-07-24 06:46
I found the explanation of why buitl-ins are immutable:

For the curious: there are two reasons why changing built-in classes is 
disallowed. First, it would be too easy to break an invariant of a 
built-in type that is relied upon elsewhere, either by the standard 
library, or by the run-time code. Second, when Python is embedded in 
another application that creates multiple Python interpreters, the 
built-in class objects (being statically allocated data structures) are 
shared between all interpreters; thus, code running in one interpreter 
might wreak havoc on another interpreter, which is a no-no.

(From http://www.python.org/download/releases/2.2.3/descrintro/)

Is the statement still valid for current version of Python?
msg70196 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-24 07:00
The "First" argument does not apply here, we could just say "annotations
are not a function invariant", but the "Second" argument is valid to me. 

A solution would be a global (or interpreter-local if we really want to
support sub-interpreters) registry that stores annotations. The index
could not be the PyCFunctionObject (since it is different for every
bound method), but the address of the PyMethodDef entry.
msg70422 - (view) Author: Richard Boulton (richardb) Date: 2008-07-30 11:18
I don't think it's reasonable not to support multiple interpreters in a
single process - they're quite widely used by mod_python and mod_wsgi,
and probably by others.  I'm not sure whether that's a problem here or
not, though.

If we need to allow function annotations to be arbitrary PyObjects,
these PyObject pointers can't (in general) refer to statically allocated
python objects, so some extension modules will have to allocate them in
the module initialisation function (and presumably deallocate them again
when the module is unloaded).

I would have thought that any such PyObjects are going to be valid only
from within a single interpreter.  Perhaps I'm wrong.

Certainly it would be unpleasant if a change to one of the objects in
one interpreter was reflected in other interpreters, but if that didn't
risk causing a crash due to the memory allocation going wrong, or
something equally nasty, it might be acceptable.
msg163251 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-06-20 05:46
What is the status of this bug?
msg163252 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-06-20 05:49
Awaiting a patch.
msg205184 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-12-04 01:28
This issue has been addressed by the PEP 436 (Argument Clinic) which supports annotation per parameter and annotation on the return type. This PEP has been implemented in Python 3.4.

I suggest to close the issue, but I would prefer that Larry closes the issue instead of me, he wrote the PEP.
msg205186 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-12-04 01:45
Argument Clinic theoretically could support annotations for builtins, though it's never been tested.  I don't know if it makes sense to close this bug yet.
History
Date User Action Args
2013-12-04 01:45:05larrysetmessages: + msg205186
2013-12-04 01:28:22hayposetnosy: + larry, haypo

messages: + msg205184
versions: + Python 3.4, - Python 3.5
2013-12-01 21:38:10alexandre.vassalottisetnosy: - alexandre.vassalotti
stage: needs patch

versions: + Python 3.5, - Python 3.0
2012-06-20 05:49:22benjamin.petersonsetmessages: + msg163252
2012-06-20 05:46:55Ramchandra Aptesetnosy: + Ramchandra Apte
messages: + msg163251
2008-07-30 11:18:46richardbsetnosy: + richardb
messages: + msg70422
2008-07-24 07:00:01amaury.forgeotdarcsetmessages: + msg70196
2008-07-24 06:46:04bhysetmessages: + msg70195
2008-07-24 06:31:18amaury.forgeotdarcsetmessages: + msg70194
2008-07-24 06:21:59amaury.forgeotdarcsetfiles: - unnamed
2008-07-24 02:48:13benjamin.petersonsetfiles: + unnamed
messages: + msg70192
2008-07-24 01:44:16bhysetmessages: + msg70186
2008-07-23 17:37:23benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg70179
2008-07-23 17:35:00bhysetmessages: + msg70178
2008-07-23 07:13:59amaury.forgeotdarcsetmessages: + msg70170
2008-07-23 05:16:50bhysetmessages: + msg70167
2008-07-21 12:15:55amaury.forgeotdarcsetstatus: closed -> open
resolution: works for me ->
messages: + msg70104
nosy: + amaury.forgeotdarc
2008-07-15 10:52:13bhysetmessages: + msg69678
2008-07-13 22:29:28alexandre.vassalottisetstatus: open -> closed
resolution: works for me
messages: + msg69629
nosy: + alexandre.vassalotti
2008-06-26 13:16:17bhycreate