classification
Title: Links for French documentation PDF is broken: LaTeX issue with non-ASCII characters?
Type: behavior Stage: patch review
Components: Documentation Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: benjamin.peterson, docs@python, ezio.melotti, fabrice, haypo, linkid, mdk
Priority: normal Keywords: patch

Created on 2017-09-26 08:59 by fabrice, last changed 2017-10-17 11:58 by mdk.

Pull Requests
URL Status Linked Edit
PR 3940 open python-dev, 2017-10-10 06:14
Messages (13)
msg303024 - (view) Author: fabrice (fabrice) Date: 2017-09-26 08:59
Hi,

In this page : https://docs.python.org/fr/2/download.html, all the documentation links aren't  available, for example : 

- https://docs.python.org/fr/2/archives/python-2.7.14-docs-pdf-a4.zip
- https://docs.python.org/fr/2/archives/python-2.7.14-docs-pdf-a4.tar.bz2

So, I can't read the documentation in French :)

Regards,
Fabrice
msg303028 - (view) Author: Julien Palard (mdk) * Date: 2017-09-26 09:58
Hi Fabrice,

Thanks for reporting.

The whole archives/ directory is completly missing for french and japanese, I'll take a look.

Bug is probably really near https://github.com/python/docsbuild-scripts/blob/master/build_docs.py#L222
msg303498 - (view) Author: Julien Palard (mdk) * Date: 2017-10-01 21:52
Problem happen during pdflatex, I tried a local build and got: 

! Package hyperref Error: Wrong DVI mode driver option `dvipdfmx',
(hyperref)                because pdfTeX or LuaTeX is running in PDF mode.

See the hyperref package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.4362 \ProcessKeyvalOptions{Hyp}
                                 
? 
! Emergency stop.
msg303501 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2017-10-01 22:36
FWIW most of the errors I met while trying to build the pdfs of the main docs were caused by the presence of non-latin1 characters.  French should be limited to the latin1 range and the error you pasted doesn't seem to be related, however that might explain while the Japanese docs are also missing (unless this issue got fixed in the meanwhile -- I haven't built the pdfs in a while).
msg303502 - (view) Author: Julien Palard (mdk) * Date: 2017-10-01 23:11
After an upgrade of my venv, the error is now:

Latexmk: applying rule 'pdflatex'...
Rule 'pdflatex': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'faq.aux'
      'faq.out'
      'faq.toc'
Latexmk: Maximum runs of pdflatex reached without getting stable files
Latexmk: All targets (faq.pdf) are up-to-date
Latexmk: Did not finish processing file 'faq.tex':
   'pdflatex' needed too many passes
msg303503 - (view) Author: Julien Palard (mdk) * Date: 2017-10-01 23:32
One difference I see in the logs of sucessfully building the faq.tex and failing to build it is:

   Package hyperref Warning: Token not allowed in a PDF string (Unicode):

Looks like if I remove all non-ascii characters from titles, it builds again.
msg303562 - (view) Author: Julien Palard (mdk) * Date: 2017-10-02 19:43
Problem looks like the utf8x package is not friend with tableofcontent:

- https://tex.stackexchange.com/questions/240801/utf8x-character-fails-in-the-table-of-contents-every-second-time-i-compile
- https://tex.stackexchange.com/questions/164458/pleaseinsertintopreamble-in-toc-and-header
msg303586 - (view) Author: Julien Palard (mdk) * Date: 2017-10-03 08:55
For the record, I can reproduce the issue with this minimal test file:

mdk@windhowl$ ls -lah
total 108K
drwxr-xr-x  2 mdk  mdk  4.0K Oct  2 21:15 .
drwxrwxrwt 18 root root  96K Oct  2 21:15 ..
-rw-r--r--  1 mdk  mdk   196 Oct  2 21:13 faq.tex

mdk@windhowl$ cat faq.tex 
\documentclass[a4,10pt,french]{report}

\usepackage[utf8x]{inputenc}
\usepackage[T1,T2A]{fontenc}
\usepackage{babel}

\begin{document}
\tableofcontents
\chapter{FAQ sur Python éh}
\end{document}

mdk@windhowl$ latexmk faq 2>&1 | tail -n 15
(/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-0.def) [2] (./faq.aux) )
Output written on faq.dvi (2 pages, 608 bytes).
Transcript written on faq.log.
Latexmk: Log file says output to 'faq.dvi'
Rule 'latex': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'faq.aux'
      'faq.toc'
Latexmk: Maximum runs of latex reached without getting stable files
Latexmk: Did not finish processing file 'faq':
   'latex' needed too many passes
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs of latex/pdflatex.
Latexmk: applying rule 'latex'...
Latexmk: All targets (faq.dvi) are up-to-date


Also, according to matrixise, it works with xelatex.
msg303587 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-03 09:34
Julien: Is there any reason why you want to use utf8x in specific?

https://tex.stackexchange.com/questions/13067/utf8x-vs-utf8-inputenc

"""
utf8x vs. utf8 (inputenc)
(...)
The simple answer is that utf8x is to be avoided if possible. It loads the ucs package, which for a long time was unmaintained (although there is now a new maintainer) and breaks various other things.
"""
msg303590 - (view) Author: Julien Palard (mdk) * Date: 2017-10-03 09:47
I personally do not care about using utf8x, it has been introduced in:

  r74549 | benjamin.peterson | 2009-08-24 12:42:36 -0500 (Mon, 24 Aug 2009) | 1 line
  fix pdf building by teaching latex the right encoding package

  # Get LaTeX to handle Unicode correctly
  latex_elements = {'inputenc': r'\usepackage[utf8x]{inputenc}'}

I tried with utf8 instead and it yielded a different set of errors, so I did not tried much. But utf8-induced bugs may be easier to fix that utf8x-induced bugs, I don't know.
msg303592 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-03 09:54
(Oh sorry, I misunderstand what you wrote. I understood that Python didn't use utf8x yet and you proposed to start using it.)
msg304016 - (view) Author: Julien Palard (mdk) * Date: 2017-10-10 06:10
Today I tried without utf8x and an up-to-date version of texlive, errors looked less mystical, so I was able to open a readable / easy to reproduce issue:

https://github.com/sphinx-doc/sphinx/issues/4136

I then followed the idea of trying xetex and I was able to build english and french PDF (still not had time to try japanese).
msg304499 - (view) Author: Julien Palard (mdk) * Date: 2017-10-17 11:58
Finally have a combination of latex engines that work for every cases, did a PR on docsbuild-scripts:

https://github.com/python/docsbuild-scripts/pull/34/

I already pulled xelatex on docs.iad1.psf.io via https://github.com/python/psf-salt/commit/989a7715c4a452b5af13baf9a33535bab0af822b#diff-6fb01fe8bbc22a54d234a57ad58e291e

Also opened a few related issue sphinx-doc side to track my thoughts:

- https://github.com/sphinx-doc/sphinx/issues/4150 (Ask if there is a better way to set the latex engine than giving it from docsbuild-scripts)
- https://github.com/sphinx-doc/sphinx/issues/4159 (Choose unicode-enabled default latex engines)
- https://github.com/sphinx-doc/sphinx/issues/4149 (Be more explicit on documentation about how to choose a latex engine)
History
Date User Action Args
2017-10-17 11:58:28mdksetmessages: + msg304499
2017-10-10 06:14:15python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request3913
2017-10-10 06:10:12mdksetmessages: + msg304016
2017-10-03 09:54:44hayposetmessages: + msg303592
2017-10-03 09:47:48mdksetmessages: + msg303590
2017-10-03 09:35:08hayposettitle: Links for French documentation pdf is broken -> Links for French documentation PDF is broken: LaTeX issue with non-ASCII characters?
2017-10-03 09:34:49hayposetnosy: + haypo
messages: + msg303587
2017-10-03 08:55:57mdksetmessages: + msg303586
2017-10-02 19:43:57mdksetnosy: + benjamin.peterson
messages: + msg303562
2017-10-01 23:32:44mdksetmessages: + msg303503
2017-10-01 23:11:49mdksetmessages: + msg303502
2017-10-01 22:36:48ezio.melottisettype: behavior

messages: + msg303501
nosy: + ezio.melotti
2017-10-01 21:54:28mdksetnosy: + linkid
2017-10-01 21:52:17mdksetmessages: + msg303498
2017-09-26 09:58:20mdksetmessages: + msg303028
2017-09-26 09:18:47hayposetnosy: + mdk
2017-09-26 08:59:09fabricecreate