Attached patch fixes test_eval by dealing with unicode
input correctly.  I'm also compiling tokenizer.c
directly rather than tokenizer_pgen.c --
tokenizer_pgen.c simply defines PGEN and then includes
tokenizer.c, which disables some of the unicode
support.  I don't think tokenizer_pgen.c is needed.

Logged In: YES 
user_id=35752

Commited.  Thanks for the patch.

Having poked around a little bit, I found it's not
simple to get at the contents of a cell object from
Python. It's not the sort of thing that one needs to be
doing very frequently, but I've run into a few
situations recently where it would be really useful
from a debugging standpoint.

You can get at a cell object containing a given value
by making a quick closure and looking at the
func_closure attribute:

  (lambda x: lambda: x)(some_value).func_closure[0]

but there's not anything we can easily do with that
object to find out what it's pointing at. The str()
representation only tells us the id of the contained value.

This trivial patch creates a "value()" method for cell
objects that returns a new reference to the contained
object, for introspection purposes.

I should mention it's not the only way to accomplish
this; you can also get at the contents of a cell by
creating a new function from a code object and
manufacturing its func_closures tuple from the cell you
already have:

def get_cell_value(cell):
    return type(lambda: 0)(
        (lambda x: lambda: x)(0).func_code, {}, None,
None, (cell,)
    )()

..but that's non-obvious and not particularly convenient.

Logged In: YES 
user_id=222090

I should mention the patch applies against both 2.4 and the
latest 2.5 from CVS.

Logged In: YES 
user_id=4771

I guess this can be safely sneaked in 2.5, as cells are quite undocumented internal objects anyway.  I believe it would be more natural to access the value as an attribute "c.value", though.  (Possibly even a read-write attribute?)

Probably not in 2.4, though; there is a no-new-feature policy.

Logged In: YES 
user_id=849994

Added a "cell_contents" attribute to cell objects in rev. 43131.

Logged In: YES 
user_id=31435

Jeremy, I vaguely recall that you deliberately made cell
objects non-inspectable.  Is that right?  If so and you
still care about that, note that it's no longer true.

The py_compile.compile() function, when doraise=False
and a compilation error is encountered, simply prints
the message to sys.stderr and returns.  However, it
neglects to add a newline.  Furthermore, judging by the
definition of PyCompileError earlier in the file and
the fact that the message will always come from an
instance of PyCompileError, the message will never
include a newline.

Some shells issue a carraige return before the command
prompt, so that would hide the output from
py_compile.compile if it were the last message to the
console.

Checking all occurences of "py_compile" in the python
source indicates they all either use compile() with
doraise=True or expect normal newline-terminated output
on error.

This patch appends the newline.

Logged In: YES 
user_id=1188172

Thanks! Checked in as py_compile.py rev 1.27.

The python implementation of islice() in
itertools-functions.html
fails the following test, unlike itertools.islice():

def test_islice():
  it = iter(range(10))
  a = list(islice(it, 0,3))
  next = it.next()
  assert next == 3

test_islice()


A fix is given in the attached file.

Logged In: YES 
user_id=80475

Checked in an alternate version of the supplied patch. 
Also, added the given test.

See:
    Doc/lib/libitertools.tex 1.35 and 1.32.2.1
    Lib/test/test_itertools.py 1.39

base64.b32decode didn't decode some things properly. 
Seems to have only impacted characters with a value <
32.  Attached is a fix and a new test.

Logged In: YES 
user_id=699438

Updated patch to reflect another fix.

I originally thought the bug was related to cleanup of a
partial quanta.  It will trigger on any quanta where the
initial hex value is < 0x10.  When this happens part of the
hex code disappears in the initial conversion and is
mistakenly swallowed.  The hex function that was originally
used doesn't zero-pad, but the way the bit shifting works,
it doesn't need to when encoding normal alpha-numeric
characters, so it seemed to work correctly.

This will only cause problems if you are using a character
value of < 0x10 in the 0th, 4th, 8th, 12th, etc positions in
the encoded string.  Otherwise it'll work properly.  Using
"%010x" ensures that you'll get the appropriate zero-padding
if necessary.

Example of more broken code not covered in the original bug:

>>> base64.b32decode(base64.b32encode('\x00aaa'))
'aaa'

Updated patch fixes this case as well.

Logged In: YES 
user_id=11375

Applied; thanks!  I also added a test case that would have
caught the bug.

Darwin 8's headers are anal about POSIX compliance, and linking 
has changed (prebinding is now deprecated, and libcc_dynamic no 
longer exists).  This configure patch makes things right.

Logged In: YES 
user_id=139309

got the OK from Anthony, going to apply and backport to 2.4-maint 
tomorrow after sufficient testing.

Logged In: YES 
user_id=139309

This is an updated version of the patch.  When using gcc4, libcc_dynamic 
does not exist.  Previous versions of gcc require it.  So the linker flags are 
dependent on the gcc version.

This fixes compilation on Darwin >= 8 with gcc < 4.0 (gcc 4 is the default)

Logged In: YES 
user_id=139309

Apparently I don't have write access to this part of CVS yet, so I can't 
commit this.  Here's the commit message I would've used:

patch [1171735] - Darwin 8's headers disable functionality when POSIX
is enabled.  This prevents the toolbox glue, all of Carbon, and various
other non-POSIX features from compiling.  The POSIX symbols are 
stillused by default, so turning off the #define doesn't hurt.

Additionally, linker flags have changed for Darwin 8, and are different
for Darwin 8/gcc4 (default) and Darwin 8/gcc3.3.  

Approved by Anthony 

Logged In: YES 
user_id=139309

I did have write access, I was confused by the fact that my python24 tag 
was checked out anon.  Committed.

(this is the 2.4-maint backport of 1171735)

Darwin 8's headers are anal about POSIX compliance, and linking 
has changed (prebinding is now deprecated, and libcc_dynamic no 
longer exists). This configure patch makes things right.

Logged In: YES 
user_id=139309

... patch small enough to upload, using same autoconf version as what 
was used by CVS

Logged In: YES 
user_id=21627

In the comment as to why this disables platform specific
features beyond repair, please list specific examples (e.g.
the first one that is encountered); please also put your
name as the one to ask for further details into this comment.

Also, I would prefer this not to talk about versions whose
development has not started yet (ie. 8.*), unless there is
some really strong indication that none of the 8.* releases
will provide a solution.

Logged In: YES 
user_id=139309

I would prefer to not talk about versions that have not been released 
publicly yet too (it has been in development for quite some time), but it's 
going to be released with this bug, and we want Python to compile on this 
platform.

I need to make some updates to these patches, and will improve the 
comment at that time.

Logged In: YES 
user_id=21627

I'm fine with this patch applying to 8.0 (or whatever it is
going to be called); my concern is that it is unclear
(atleast to me) whether the problem will still exist in,
say, 8.5.

Logged In: YES 
user_id=139309

That's up to them.. I don't have access to that kind of information.  We'll 
know when a new set of headers are released.  In my experience with 
the platform, it is very rare that headers are modified in any way after 
release, until the next major version.  So my guess is that the problem 
will still exist in 8.5.

It doesn't matter so much either way, turning these defines off doesn't 
break anything.  On this platform, they are used almost exclusively to 
turn off behavior, not to enable or change behavior.  I'm perfectly willing 
to revise the patch when/if it's fixed in the headers, but I think that will 
be a while.

Logged In: YES 
user_id=139309

This is an updated version of the patch.  When using gcc4, libcc_dynamic 
does not exist.  Previous versions of gcc require it.  So the linker flags are 
dependent on the gcc version.

This fixes compilation on Darwin >= 8 with gcc < 4.0 (gcc 4 is the default)

Logged In: YES 
user_id=21627

Ok, this is fine then. The other option would have been to
explicitly state 8.0, instead of 8.*, and extend that to
8.[01] when 8.1 is released and still shows the problem, and
so on. I really wish Apple would provide a #define to turn
on additional features.

Logged In: YES 
user_id=139309

I wish Apple would have an additional #define too, but they don't.

The other option sounds bad.  On the first minor kernel update, Python 
won't compile any more.  In the manner I applied the patch, a minor kernel 
update will not break anything.  When they fix the headers, we can fix 
Python's configure on our own schedule, since this patch won't regress 
with fixed headers.

Logged In: YES 
user_id=21627

I'm concerned that this section in configure.in might
collect old cruft, without a chance of ever getting rid of
it. Therefore, in general, I would like to see each setting
be reconsidered every time a new OS release is made;
allowing for removal of settings for systems that are not
supported any longer.

In the specific case, this is probably fine as-is, since
"Darwin 8.*" will likely be superceded with Darwin 9.0
within a few years, at which point the issue then is
reconsidered.

Logged In: YES 
user_id=139309

I agree, and I will fix it when the headers allow such a fix.

I just don't want to see a minor OS update break all current versions of 
Python, such that we have to issue a patch and accelerate a release.  I 
also don't want it to be specific to Darwin 8.*, because chances are, 
Darwin 9 is going to be more like Darwin 8 than previous versions... so it 
makes sense to default to whatever worked most recently.

Logged In: YES 
user_id=139309

Apparently I don't have write access to this part of CVS yet, so I can't 
commit this.  Here's the commit message I would've used:

patch [1171767] - Darwin 8's headers disable functionality when POSIX
is enabled.  This prevents the toolbox glue, all of Carbon, and various
other non-POSIX features from compiling.  The POSIX symbols are 
stillused by default, so turning off the #define doesn't hurt.

Additionally, linker flags have changed for Darwin 8, and are different
for Darwin 8/gcc4 (default) and Darwin 8/gcc3.3.  

Approved by Anthony 

Logged In: YES 
user_id=21627

Hmm; I can't see anything wrong with your user settings, so
you should have write access in theory. Can you do checkouts
using your SSH key?

Logged In: YES 
user_id=139309

nice catch!  you're right, my release24-maint wasn't checked out with SSH.  
I did have access to commit.  Thanks.

Logged In: YES 
user_id=45365

Adding this information here for reference: rumour has it that Apple is 
actively thinking of ways to fix the POSIX_C_SOURCE problems (i.e. the 
fact that this define disables some features while enabling others), 
probably through a different define that you can use together with 
POSIX_C_SOURCE to re-enable apple-specific additions or something like 
that.

So, we should keep our eyes open for this.

This patch adds signed and unsigned long long support
to the array module. These types are already supported
by the struct module and use the same format characters
(q/Q).

Also corrects a minor bug in PyLong_AsUnsignedLongLong
which reports a BadInternalCall for arguments of
inappropriate type rather than a mere TypeError as
reported by the other conversion functions.

Logged In: YES 
user_id=4771

No two conversion function in longobject.c seem to have the same rules for what to do about non-long objects :-(  I'm afraid some clean-up would be useful, but also difficult for fear of breaking existing user C code :-(

In fact, your patch doesn't apply with today's CVS because someone already tried to add some magic in PyObject_AsLongLong().  It also fails on test_array.py and applies uncleanly on arraymodule.c.

Also, it needs to update the array module documentation.

Logged In: YES 
user_id=562624

My patch was against 2.4... (duh!)

The other bug is already fixed on 2.5.

Distutils have a script functionality that copy the 
corresponding file
into a common script folder (that could be on the 
path) and sligthly
modify the first line of the script to reflect the path 
of the python
executable.

It is very nice because it make new commands 
directly available but 
for the window platform I think it could be 
improved.

First the extention of the script could be .bat such 
that they will be
accessible as commands as soon as the script 
folder is in the path.

Second the first line adaptation could be slightly 
different and take
advantage of the -x option of the python executable

#!python foo bar

could become:

@C:\Python24\python.exe -x "%~f0" foo bar  & exit /
b

instead of:

#!C:\Python24\python.exe -x "%~f0"  foo bar & exit /
b

In attachement I add my modified version of the 
build_scripts.py file (Revision 1.25, the head of the 
30th of March)

Logged In: YES 
user_id=511447

 Here is a small summary of the remark made arround 
the she-bang modification of the windows platform.

1. Trent let us know that the she-bang modification 
works only for the win nt sons and that in general it is 
better to use the .bat extension instead of the .cmd one

2. I realize that we shouldn't change the extension of a 
script if it exist since the extention could be usefull to 
determine which version of python to run
 - pythonw for the .pyw extention
 - python for the .py extention

According to these remarks and in the purpose to be 
as conservative as possible I make this another 
proposition:

The change in the she bang will happend only if
 - the win32 platform is detected (os.name == "nt")
 - the platform is win NT son (os.environ["OS"] == 
"Windows_NT")
 - the original script had no extension (os.path.
splitext(outfile)[1] == "")

Furthermore I add the %* in the new nt she-bang to let 
the underlying script know about command line 
argument.

such that the unix she-bang:

    #!python foo bar

will become:

    @C:\Python24\Python.exe -x "%%~f0" foo bar %* & 
exit /b

I think that in these conditions the change could only 
improve the existing situation at least it improve for 
some them (Zope, Chetaah, PyCrust, IPython)

Tell me what you think about the proposed change.

Vivian.

Docstrings for unicodedata, extracted from the tex
documentation.

Logged In: YES 
user_id=55188

Committed in Modules/unicodedata.c 2.33.

Thank you for the patch!

Logged In: YES 
user_id=954779

I think the module's docstring is wrong (and so is the tex
documentation, and this page too
http://docs.python.org/lib/module-unicodedata.html).

The URL for UnicodeData File Format 3.2.0 is
http://www.unicode.org/Public/3.2-Update/UnicodeData-3.2.0.html
now.

Logged In: YES 
user_id=55188

URL is corrected in CVS. Thank you!

Doc/lib/libunicodedata.tex 1.9
Modules/unicodedata.c 2.34

This is a first, rough cut at allowing subclasses of variable length 
types to have __slots__ of all flavours, not just __dict__.

The motivation is trying to understand and document what's going on 
in typeobject.c, and the less special cases knocking around the 
better.

This patch also allows instances of such classes to be weakly 
referenced.

What is missing: tests, lots of tests, documentation.  Also, the code 
is a bit hackish at various points; a degree of clean up can certainly 
be acheived.

Also, I think my code probably fails to cope with code like:

class A(str):
 pass # implicitly adds __dict__, __weakref__
class B(A):
 __slots__ = ["a", "b"]

b = B()
b.c = 1

Hmm, yes.  Oh well, no time to fix today (I don't think it's that big a 
deal).

Logged In: YES 
user_id=4771

I'm confused: the rule for negative slot offsets appear to be different to the one for tp_dictoffset, which only increases the amount of obscurity around here.

tp_dictoffset counts relative to the end of the object, whereas in your patch negative slot offsets are a different trick to mean "relative to the start but skipping the varsized part".  The difference shows up when subclassing increases tp_basicsize.  This should be resolved one way or the other -- and I think that a clear picture of the various parts of the object and how they are measured would be a good start.

That's also related to your proposed change to extra_ivars(), which would become slightly more permissive; I strongly suspect that it would allow more strange segfaulting cases to sneak in undetected...

Logged In: YES 
user_id=6656

> I'm confused: the rule for negative slot offsets appear to be
> different to the one for tp_dictoffset

Yes.  I think this is actually necessary.

Consider:

class S(str):
    __slots__ = ['a']

you'd except S.__dict__['a'].__offset__ (well, if the attribute existed) to be 
-4.

Then

class T(S):
    __slots__ = ['b']

then using the 'from the end of the object' rule for  T().a would actually find 
T.b.  (IOW, T.__dict__['b'].__offset__ == T.__dict__['a'].__offset__ == -4).  
The alternative would be to somehow override all the slots in S when T is 
defined -- and this doesn't seem wise.

__dict__ indeed works differently, because 
instance.__class__.__dictoffset__ is updated on subclassing.  You could 
make __dict__ work like the slots mentioned above, but then you'd have to 
find the '__dict__' descriptor every time you wanted to access an 
instance's dictionary, and that would be slow (and might even not really 
work, but I don't want to risk brain-explosion by thinking about it too hard)

> which only increases the amount of obscurity around here.

Yeah, sorry about that.

I think something I've realised over the past few days is that __dict__ 
really is special.  I'm not sure __weakref__ is (though I guess it's special in 
that you want to be able to access it from C without any risk of executing 
Python level code, i.e. replacing Py_GETWEAKREFLIST(ob) with 
PyOjbect_GetAttrString(ob, "__weakref__") would be unfortunate).

> This should be resolved one way or the other 

See above -- don't think you can.

> -- and I think that
> a clear picture of the various parts of the object and how they
> are measured would be a good start.

No kidding here!

> That's also related to your proposed change to extra_ivars(),
> which would become slightly more permissive; I strongly suspect
> that it would allow more strange segfaulting cases to sneak in
> undetected...

Almost certainly!

Logged In: YES 
user_id=4771

I think it's still possible to give slot.offset the same meaning as tp_dictoffset, even given the additional constrain that it can't change upon subclassing.  In your example classes S and T, we can put 'b' before 'a' in memory, so that a.offset==-4 (for both S and T) and b.offset==-8.

Logged In: YES 
user_id=6656

Heh, yes that works, and completely hadn't occurred to me.

If a malformed XML file (containing non unicode
characters) is parsed with pyexpat, python crashes.

Most details on request.

Logged In: YES 
user_id=1210681

Maybe security related, as it can lead to denial of service:
it crashes a Zope server using the ParsedXML product simply
by uploading a malformed XML file.