classification
Title: Documentation i18n: Added trans tags in sphinx templates
Type: Stage: resolved
Components: Documentation Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: abarry, docs@python, ezio.melotti, inada.naoki, mdk, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2015-12-18 16:30 by mdk, last changed 2016-03-05 02:23 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
i18n.patch mdk, 2015-12-18 16:30 Patch to add translation tags to documentation index template review
i18n-tags-in-sidebar.patch mdk, 2016-01-22 16:47 review
i18n-tags-in-layout.patch mdk, 2016-01-22 16:47 review
trans-tags-in-2.7.patch mdk, 2016-01-23 11:11
Messages (15)
msg256691 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2015-12-18 16:30
o/

# Context

I'm trying to resuscitate the translation of Python doc in french, [python_doc_fr](https://github.com/AFPy/python_doc_fr).

I finished the translation of `tutorial.po` (that was partially translated by the AFPY team around 2012) and now working on `glossary.po` (~60%). :-)

# Problem

The homepage (`src/Doc/tools/templates/indexcontent.html`) is not translatable as is, as it misses the translation markers.

# Solution

I attached a patch to this issue adding them, it works well so I translated the generated `sphinx.po` and rebuilt the doc you can enjoy the result here:
 
http://www.afpy.org/doc/python/3.4/
msg256729 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2015-12-19 14:06
For emacs users, this `query-replace-regex` comes handy to add trans tags around strings, if one want to apply them to another file / version:

>\([^>^J]*[a-zA-z][^>^J]*\)< -> >{% trans %}\1{% endtrans %}

Note that ^J is obtained typing C-q C-j.
msg258743 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-21 08:20
The patch only changes Doc/tools/templates/indexcontent.html.

(Simplified) example of change on Doc/tools/templates/indexcontent.html:

   "<p>What's new in Python {{ version }}?</p>"

becomes

   "<p>{% trans %}What's new in Python {{ version }}?{% endtrans %}</p>"

I'm not a big fan of this option. Where does {% trans %}...{% endtrans %} format come from?

Is there an existing format to translate HTML templates?

I found html_translator_class config option in Sphinx:

http://www.sphinx-doc.org/en/stable/config.html?highlight=trans#confval-html_translator_class

Can't you avoid the new tags and use a small HTML parser to extract all strings?
msg258749 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-01-21 09:56
@haypo I used the ``{trans}`` ``{endtrans}`` syntax because [Sphinx uses Jinja](http://www.sphinx-doc.org/en/stable/templating.html) and [``{trans}`` is the Jinja syntax for internationalisation](http://jinja.pocoo.org/docs/dev/templates/#i18n)

So ``{trans}`` is the standard syntax in this context and as you see I did not implemented a small script to parse them, Sphinx does it for me and add them in ``po`` files by itself, so Sphinx understand them, without need of some code.

It is possible to parse HTML to extract the strings to translate, but it would also fetch some false positives, and probably won't cut sentences at the right place, typically:

    <h1>The Python Documentation<h1>

is easy to parse, extract the sentence, translate it, and, at compile-time, search and replace the sentence by its translation, nice.

    Here are <a href="...">more details</a> about it.

Would be hard to parse: we'll need to introduce an heuristic to tell if we keep the HTML tag in the translation, or split it in three sentences "Here are", "more details", "about it". Sadly, splitting it in three sentences is not possible, because in some languages the order or those blocks may be swapped. So we need to get the <a href> and </a> inside the sentence to allow translator to move the three part around freely.

Also, the Jinja xgettext parser handle variables inside translation, to start from a non-standard syntax ``{{ variable_name }}`` to a more standard one (normally ``%s`` so the string can be used on almost every languages, but here) ``%(variable)s``.

Those variables are obligatory, without them we'll get *code* inside translations strings, like ``pathto("whatsnew/index")``, which is even uglier than ``%(variable)s``.

Here it is:

    <span class="linkdescr"> {% trans whatsnew_index=pathto("whatsnew/index") %}or <a href="{{ whatsnew_index }}">all "What's new" documents</a> since 2.0{% endtrans %}</span></p>

Which yiels, in po files:

    "or <a href=\"%(whatsnew_index)s\">all \"What's new\" documents</a> since 2.0"

Which is better than:

    "or <a href=\"{{ pathto("whatsnew/index") }}\">all \"What's new\" documents</a> since 2.0"

Which itself is better (for the re-ordering problem in other languages) than:

    "or"
    "all \"What's new documents"
    "since 2.0"
msg258750 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-01-21 10:05
That's also the actual syntax used by existing Sphinx templates:

    mandark@windhowl$ grep -r '{% trans'  ~/.local/lib/python3.4/site-packages/sphinx_rtd_theme/
    footer.html:        {% trans path=pathto('copyright'), copyright=copyright|e %}&copy; <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
    footer.html:        {% trans copyright=copyright|e %}&copy; Copyright {{ copyright }}.{% endtrans %}
    ...
    ...
msg258752 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-21 10:19
> @haypo I used the ``{trans}`` ``{endtrans}`` syntax because [Sphinx uses Jinja](http://www.sphinx-doc.org/en/stable/templating.html) and [``{trans}`` is the Jinja syntax for internationalisation](http://jinja.pocoo.org/docs/dev/templates/#i18n)

Oh ok, fair enough :-)

Doc/tools/templates/ contains 3 HTML files, why not updating the 2 other HTML files?
msg258827 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-01-22 16:47
And here are the i18n tags for the sidebar, thanks @haypo it was a good idea to translate them too ^-^
msg258828 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-01-22 16:47
And tags for the layout template.
msg258830 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-22 17:01
New changeset b1615165fa4a by Victor Stinner in branch '3.5':
doc: i18n HTML templates
https://hg.python.org/cpython/rev/b1615165fa4a

New changeset 5af5b36c197e by Victor Stinner in branch 'default':
Merge 3.5 (i18n doc, issue #25907)
https://hg.python.org/cpython/rev/5af5b36c197e
msg258831 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-22 17:02
I applied 3 patches to branches 3.5 and default (3.6), but the patches don't apply cleanly to Python 2.7.

Can you please cook patches for Python 2.7?

(I suggest to focus on translation of the Python 3 documentation, but for me, it's cheap to apply such patch to Python 2.7.)
msg258857 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2016-01-23 10:00
O/T

Hi, Julien.

I'm maintainer of Python Document Japanese translation project.  (http://docs.python.jp/ )
We use Transifex to ease many volunteers working on translating.

https://www.transifex.com/python-doc-ja/python-35/dashboard/

Repository of Python 3.5 Japanese document is here: https://github.com/python-doc-ja/cpython-doc-intl
msg258860 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-01-23 11:11
@haypo: And here is the patch for 2.7, and here is the result of the patch applied and new msgids translated: http://www.afpy.org/doc/python/2.7/
msg258861 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-01-23 11:23
@naoki Not sure, but with those patches you'll may be able to translate the documentation without forking it. Like we're doing here: https://github.com/afpy/python_doc_fr

Also did you contacted the upstream to ask them if they want to cross-link your versions ? Sphinx don't natively support it, but it still doable and can be a nice way to advertise our efforts (I mean, I'm sure a lot of french people don't know we're doing it and stuck to the english version).

I heard we used transfiex too like 6 years ago, long before me being on the project, I may drop an eye on it, thanks.

It may be a bit out of the scope of this issue, so let's continue elsewhere, like by email if you want I'm julien at palard dot fr.

BTW, nice to hear we're not alone to translate the Python doc :-))).
msg258863 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-23 11:55
New changeset 72edb81e456b by Victor Stinner in branch '2.7':
doc: i18n HTML templates
https://hg.python.org/cpython/rev/72edb81e456b
msg258864 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-23 11:57
> @haypo: And here is the patch for 2.7, and here is the result of the patch applied and new msgids translated: http://www.afpy.org/doc/python/2.7/

Thanks. I also pushed this change.

@INADA Naoki, @Julien: Don't hesitate to bug me if you want to push other changes to make your work easier.
History
Date User Action Args
2016-03-05 02:23:34berker.peksagsetresolution: fixed
stage: resolved
2016-03-05 00:19:45mdksetstatus: open -> closed
2016-01-23 11:57:15vstinnersetmessages: + msg258864
2016-01-23 11:55:34python-devsetmessages: + msg258863
2016-01-23 11:23:31mdksetmessages: + msg258861
2016-01-23 11:12:00mdksetfiles: + trans-tags-in-2.7.patch

messages: + msg258860
2016-01-23 10:00:11inada.naokisetnosy: + inada.naoki
messages: + msg258857
2016-01-22 17:02:17vstinnersetmessages: + msg258831
versions: + Python 2.7, Python 3.5, Python 3.6, - Python 3.4
2016-01-22 17:01:04python-devsetnosy: + python-dev
messages: + msg258830
2016-01-22 16:47:50mdksetfiles: + i18n-tags-in-layout.patch

messages: + msg258828
2016-01-22 16:47:21mdksetfiles: + i18n-tags-in-sidebar.patch

messages: + msg258827
2016-01-21 10:19:23vstinnersetmessages: + msg258752
2016-01-21 10:05:18mdksetmessages: + msg258750
2016-01-21 09:56:51mdksetmessages: + msg258749
2016-01-21 08:20:44vstinnersetnosy: + ezio.melotti, vstinner
messages: + msg258743
2015-12-19 14:06:29mdksetmessages: + msg256729
2015-12-18 16:45:14abarrysetnosy: + abarry
2015-12-18 16:30:35mdkcreate