unicodedata.normalize cannot convert turkish letter "ı" into "i":


import unicodedata
s = u"üfürükçü ağaç ve ıslıkçı çeşme"
print(shoehorn_unicode_into_ascii(s))
print unicodedata.normalize('NFKD', s).encode('ascii','ignore')

>> ufurukcu agac ve slkc cesme

but the result should be
>> ufurukcu agac ve islikci cesme

unicodedata.normalize cannot convert turkish letter "ı" into "i":


import unicodedata
s = u"üfürükçü ağaç ve ıslıkçı çeşme"

print unicodedata.normalize('NFKD', s).encode('ascii','ignore')

>> ufurukcu agac ve slkc cesme

but the result should be
>> ufurukcu agac ve islikci cesme

You are doing two different things to the original string: normalizing and encoding to ascii with errors ignored. Each should be tested separately.
On 3.2:
import unicodedata
s1 = "üfürükçü ağaç ve ıslıkçı çeşme"
s2 =  unicodedata.normalize('NFKD', s1)
print(s2)
print(s2.encode('ascii','ignore'))

#prints
üfürükçü ağaç ve ıslıkçı çeşme
b'ufurukcu agac ve slkc cesme'

The dotless i (==  '\u0131') in s2 does not encode to ascii and is properly dropped when the error is ignored.

I believe you are mistaken to think that unicodedata.normalize *should* turn turkish letter "ı" == "\u131" into "i". Unicodedata.decomposition("ı") returns an empty string, as it should (see below) because that character has no decomposition normalization in Unicode 6. So I am closing this issue as invalid.

Here is the entry from
http://www.unicode.org/Public/6.0.0/ucd/UnicodeData.txt
0131;LATIN SMALL LETTER DOTLESS I;Ll;0;L;;;;;N;;;0049;;0049
That is explained here
http://www.unicode.org/reports/tr44/tr44-6.html#UnicodeData.txt
The blank after 'L' (bidi class - left to right) is for decomposition type and mapping. There is none, so unicodedata.decomposition is correct. The last three entries are for uppercase, lowercase, and titlecase conversions. Those are different from normalizations.

To reinforce this,
http://www.unicode.org/Public/6.0.0/ucd/NormalizationTest.txt
says explicitly
"@Part1 # Character by character test
# All characters not explicitly occurring in c1 of Part 1 have identical NFC, D, KC, KD forms."
'c1' is column 1, starting from 1.
In this list, 0130 is followed by 0132, omitting 0131, so the line above applies.

After writing this, I discovered that Lib/test/test_normalization.py runs the complete test specified in NormalizationTest.txt for code points that have and do not have normalization forms.

Side note" Python 2.6 is in security-fix-only mode.