This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Expand unicodedata module documentation
Type: enhancement Stage: patch review
Components: Documentation, Unicode Versions: Python 3.2, Python 3.3, Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: belopolsky, docs@python, ezio.melotti, georg.brandl, lemburg, loewis, vstinner
Priority: normal Keywords: needs review, patch

Created on 2010-12-09 21:31 by belopolsky, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
unicodedata-doc.diff belopolsky, 2010-12-09 22:26
issue10665.diff belopolsky, 2010-12-10 21:23 review
Messages (21)
msg123700 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-09 21:30
unicodedata module documentation has not been updated to reflect transition to 6.0.  Attached patch fixes the version and unicode.org links and starts making the documentation rely less on the unicode.org pages for basic understanding of the provided functionality.

I am posting work in progress to solicit feedback on how much of the Unicode Standard information we would want to present here.

On of the goals of this patch is to provide a standard reference that can be used throughout the library manual for basic Unicode concepts without sending the reader over to unicode.org.
msg123704 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-09 22:26
Added more tables semi-automatically produced from http://www.unicode.org/Public/UNIDATA/PropertyValueAliases.txt
msg123708 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-12-09 23:10
Please, one issue per report and checkin, and no work-in-progress on the tracker. The issue of factually correcting claims about the unicodedata module and elaborations on how it works are unrelated issues.
msg123740 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-10 18:31
On Thu, Dec 9, 2010 at 6:10 PM, Martin v. Löwis <report@bugs.python.org> wrote:
..
> Please, one issue per report and checkin,

The s/5.2/6.0/ issue is hardly worth a tracker ticket.   I've
committed these changes in r87159.  (Sorry for the unrelated changes -
reverted in the next checkin.)

> and no work-in-progress on the tracker.

Why?  I thought "release early, release often" was a good thing.   I
wanted to get an early feedback because we certainly don't want to
replicate the Unicode Standard in the Python documentation, but I
think at least for the category() method that returns cryptic 2-letter
codes, we should include a table explaining them.   I am not so sure
about bidirectional() or asian_width().

> The issue of factually correcting claims about the unicodedata module and elaborations on how it works are unrelated issues.

I am changing the title of the issue to make it cover only the latter.
>
> ----------
> nosy: +loewis
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue10665>
> _______________________________________
>
msg123753 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-10 21:23
In issue10665.diff, I completed the character examples in the general categories table.
msg123756 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-12-10 23:04
> Why?  I thought "release early, release often" was a good thing.

Create a branch for that, or post an issue on Rietveld. W-I-P IMO
confuses people reviewing the patches, running into the same ones
over-and-over again, only to find out every time "it's not ready yet".
So the natural reaction is to close it as rejected, for it being
incomplete.

Regards,
Martin
msg123890 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-13 18:48
On Fri, Dec 10, 2010 at 6:04 PM, Martin v. Löwis <report@bugs.python.org> wrote:
..
>> Why?  I thought "release early, release often" was a good thing.
>
> Create a branch for that, or post an issue on Rietveld.

Martin,

This is a documentation patch affecting a single HTML page.  An svn
branch for something like this is certainly an overkill.  Maybe when
branches become more user-friendly with Hg, it will make sense to do
something like this in a branch.   Rietveld is great for code reviews,
but for doc patches its is sometimes desirable to post a rendered page
for a review.  In this particular case, however the reST in diff is
quite readable.

> W-I-P IMO
> confuses people reviewing the patches, running into the same ones
> over-and-over again, only to find out every time "it's not ready yet".

I posted this patch with a specific question: "how much of the Unicode
Standard information we would want to present?"   The patch included
several tables similar to what a determined reader can find at
unicode.org.   I think it is useful to present this information in the
Python docs for several reasons:

1. It makes the information more readily accessible to someone who
just want to figure out what the code returned by
unicodedata.category() means.

2. We can present examples using Python notation and focus on what is
relevant to Python users.  For example, what are the digits other than
0-9, or what is the difference between a digit and a decimal.

3. Other parts of the documentation can refer to this information more
easily.  For example, str.isdigit() can refer to 'Nd' general
category.

The downside is that we may need to update this info when Unicode
Standard changes.  Given the pace of change for this info, I don't
think this is serious burden and most of the data can be
auto-generated from UCD files.

> So the natural reaction is to close it as rejected, for it being
> incomplete.

Are you going to reject say issue2636 on this basis? :-)   Has *any*
patch ever been rejected as incomplete?

Seriously, I had a specific reason to post an incomplete patch for
review: formatting reST tables is tedious and if others think we
should not include this info in Python docs, I don't want to spend
more time polishing the patch.  On the other hand, an incomplete patch
is helpful because it demonstrates how much information I am proposing
to include.  If I just posted a request without a patch, a natural
reaction would be: do you have a patch?
msg123905 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-12-13 22:26
> Are you going to reject say issue2636 on this basis? :-)   Has *any*
> patch ever been rejected as incomplete?

I certainly did close patches for that reason. Before that, I keep
asking people not to post W-I-P. As for issue2636, I have been really
tempted to, see msg83277, msg83411. In one case, the conversation ended
with the user being banned from the tracker. I personally feel fairly
strongly that it is abuse to post things that you know are incomplete.
I won't review any such code.
msg130536 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-03-11 00:26
The patch contains non-ascii chars that should be avoided (they break `make pdf`).
msg130646 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-03-11 23:43
> The patch contains non-ascii chars that should be avoided (they break `make pdf`).

What is the error? Can't you fix the PDF generator instead?
msg130699 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-03-12 22:26
I think Georg said that non-ascii chars shouldn't be used directly in the rst files, and one of the reasons is that they break `make pdf` (I haven't tried though).
I added him to the nosy.
msg130864 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-03-14 17:39
The "PDF generator" is PDFLaTeX, whose range of Unicode characters is very limited, so no, I can't fix it.
msg132236 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-03-26 13:40
FWIW latin-1 chars are fine (e.g. ¶ or è).  The right command to build the pdfs is `make latex` and then `make all-pdf` in build/latex/.

Alexander, now you could also make a remote hg repo, and use it to generate up-to-date patches that can be reviewed directly.

I left a few more comments on rietveld about the last patch.
msg133845 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-04-15 16:28
Alexander suggested on IRC to use the 'unicode' directive[0], but even if that works in the HTML (only outside code blocks), it still breaks the PDF.
Another alternative that might work is the 'raw' role[1].

[0]: http://docutils.sourceforge.net/docs/ref/rst/directives.html#unicode-character-codes
[1]: http://docutils.sourceforge.net/docs/ref/rst/roles.html#specialized-roles
msg133848 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-04-15 17:09
> The "PDF generator" is PDFLaTeX, whose range of Unicode characters
> is very limited, so no, I can't fix it.

My search for pdflatex and unicode has quickly revealed this 4-year old howto:

http://tclab.kaist.ac.kr/ipe/pdftex_2.html

I'll experiment with some recent LaTeX distributions before making further effort to work around current unicode limitations.  For example, XeTeX appears to have good unicode support:

http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=xetex
msg133849 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-04-15 17:10
Ezio Melotti wrote:
> 
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
> 
> Alexander suggested on IRC to use the 'unicode' directive[0], but even if that works in the HTML (only outside code blocks), it still breaks the PDF.
> Another alternative that might work is the 'raw' role[1].
> 
> [0]: http://docutils.sourceforge.net/docs/ref/rst/directives.html#unicode-character-codes
> [1]: http://docutils.sourceforge.net/docs/ref/rst/roles.html#specialized-roles

I don't think we should include Unicode code points as literals
in Python source code examples, for much the same reason we
don't want them in the stdlib source code.

Why don't you use the standard literal escapes for the examples
and annotate the code points with the code point names ?
msg133852 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-04-15 17:15
One reason is that unicodedata.lookup actually returns a unicode char, so if we want to show a code snippet that uses unicodedata.lookup we either have to use a unicode literal or limit the chars in the examples to latin1 to make sure it works nice with the PDF generator.

Using escape sequences elsewhere might work, but in some examples it's better to use the actual chars IMHO (except that they don't work with the PDF).
msg133853 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-04-15 17:23
Ezio Melotti wrote:
> 
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
> 
> One reason is that unicodedata.lookup actually returns a unicode char, so if we want to show a code snippet that uses unicodedata.lookup we either have to use a unicode literal or limit the chars in the examples to latin1 to make sure it works nice with the PDF generator.

Why not wrap the calls with a repr() ?

> Using escape sequences elsewhere might work, but in some examples it's better to use the actual chars IMHO (except that they don't work with the PDF).

Sure, it'll look nicer, but it will also make comparing the examples
with the actual output users see on the screen error-prone (e.g. if
the fonts don't have the necessary glyphs).

Copy&paste will also often fail.

I think it's more useful to show examples that more or less always
work, than ones which display all available goodies.
msg133854 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-04-15 17:25
On Fri, Apr 15, 2011 at 1:10 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>..
> Why don't you use the standard literal escapes for the examples
> and annotate the code points with the code point names ?

A am neutral on how to enter unicode characters in source reST.   In
the previous discussions most people seemed to prefer WISIWYG.  If
literal escapes solved the PDF issue, I would use it even at the
expense of loosing testability of the output displays.

Code point names as usually very long for exotic characters that
illustrate UCD features.  I like presenting them, but in tables I'd
rather present more examples and still keep column width reasonable.
msg133857 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-04-15 17:38
On Fri, Apr 15, 2011 at 1:23 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>..
> Why not wrap the calls with a repr() ?
>

Won't help:

"'Ӝ'"

I think you meant ascii(), but that's ugly IMO:

"'\\u04dc'"

Maybe

'\u04dc'

but that's too much of scaffolding.

..
> I think it's more useful to show examples that more or less always
> work, than ones which display all available goodies.

I disagree.  Users that are advanced enough to be interested in
reading unicodedata reference documentation should be capable of
either fixing their environment or understanding its limitations.
msg133884 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-04-16 06:46
Can y'all please accept the limitation to latin-1?  Like Marc-Andre says, the more "exotic" the examples, the less likely it is that users reading the HTML can display it anyway.
History
Date User Action Args
2022-04-11 14:57:10adminsetgithub: 54874
2012-09-26 18:46:07ezio.melottisetversions: + Python 3.4
2012-04-30 16:43:21ezio.melottisetstage: patch review
type: enhancement
versions: + Python 3.3
2011-04-16 06:46:11georg.brandlsetmessages: + msg133884
2011-04-15 17:38:38belopolskysetmessages: + msg133857
2011-04-15 17:25:00belopolskysetmessages: + msg133854
2011-04-15 17:23:35lemburgsetmessages: + msg133853
2011-04-15 17:15:45ezio.melottisetmessages: + msg133852
2011-04-15 17:10:33lemburgsetmessages: + msg133849
2011-04-15 17:09:20belopolskysetmessages: + msg133848
2011-04-15 16:28:38ezio.melottisetkeywords: + needs review

messages: + msg133845
2011-03-26 13:40:39ezio.melottisetmessages: + msg132236
2011-03-14 17:39:41georg.brandlsetnosy: lemburg, loewis, georg.brandl, belopolsky, vstinner, ezio.melotti, docs@python
messages: + msg130864
2011-03-12 22:26:14ezio.melottisetnosy: + georg.brandl
messages: + msg130699
2011-03-11 23:43:37vstinnersetnosy: lemburg, loewis, belopolsky, vstinner, ezio.melotti, docs@python
messages: + msg130646
2011-03-11 00:26:57ezio.melottisetnosy: lemburg, loewis, belopolsky, vstinner, ezio.melotti, docs@python
messages: + msg130536
2010-12-13 22:26:55loewissetmessages: + msg123905
2010-12-13 18:48:15belopolskysetmessages: + msg123890
2010-12-10 23:04:39loewissetmessages: + msg123756
2010-12-10 21:23:08belopolskysetfiles: + issue10665.diff

messages: + msg123753
2010-12-10 18:42:27belopolskysetnosy: + lemburg, vstinner, ezio.melotti
2010-12-10 18:31:15belopolskysetmessages: + msg123740
title: Update and expand unicodedata module documentation -> Expand unicodedata module documentation
2010-12-09 23:10:04loewissetnosy: + loewis
messages: + msg123708
2010-12-09 22:27:07belopolskysetfiles: - unicodedata-doc.diff
2010-12-09 22:27:00belopolskysetfiles: + unicodedata-doc.diff

messages: + msg123704
2010-12-09 21:34:32belopolskysetnosy: belopolsky, docs@python
components: + Unicode
2010-12-09 21:31:00belopolskycreate