classification
Title: Full docs build of 3.6 and 3.7 failing since 2017-10-15
Type: Stage: resolved
Components: Documentation Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Mariatta, docs@python, jfbu, mdk, ned.deily, willingc
Priority: normal Keywords: patch

Created on 2017-12-02 15:25 by willingc, last changed 2017-12-03 18:18 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4683 merged python-dev, 2017-12-02 22:24
Messages (13)
msg307433 - (view) Author: Carol Willing (willingc) * (Python committer) Date: 2017-12-02 15:25
Currently, the 3.7.0a2 doc links are giving 404 errors. 3.6 docs are working just fine.

https://docs.python.org/3.7/archives/python-3.7.0a2-docs-pdf-letter.tar.bz2 results in a 404.

I wasn't sure if I should report here or in python.org issue tracker.
msg307436 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-02 15:44
It looks like the 3.6 and 3.7 full doc build has been failing since 2017-10-15.  Seen in /var/log/docsbuild/python37-en.log:

[...]
[3] [4] [5] [6] [7] (/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-1.def
) (/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uninames.dat)
(/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-1.def)

! Package ucs Error: Unknown Unicode character 383 = U+017F,
(ucs)                possibly declared in uni-1.def.
(ucs)                Type H to see if it is available with options.

See the ucs package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.781 Latin small letter dotless i), ‘ſ
                                          ’ (U+017F, Latin small letter lo...

? 
! Emergency stop.
 ...                                              
                                                  
l.781 Latin small letter dotless i), ‘ſ
                                          ’ (U+017F, Latin small letter lo...

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on howto-regex.log.
Latexmk: Index file 'howto-regex.idx' was written
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 256
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs of latex/pdflatex.
Latexmk: Errors, so I did not complete making targets
make[2]: *** [howto-regex.pdf] Error 12
make[2]: Leaving directory `/srv/docsbuild/python37/Doc/build/latex'
make[1]: *** [dist] Error 2
make[1]: Leaving directory `/srv/docsbuild/python37/Doc'
make: *** [autobuild-dev] Error 2

Julien, can you take a look please?
msg307442 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2017-12-02 17:46
Yes, it's related to:
- https://github.com/python/cpython/pull/3940
- https://github.com/python/cpython/pull/4069

Idea is: Unicode characters are not well supported by pdflatex which what's we're using to build our PDF files.

There's two solutions:

- Use xelatex which just works but does not use the same fonts, and the output style is slightly different
- Use hard handcrafted latex macros to make pdflatex work

I'm personally in favor of the xelatex solution (The simple one: the patch even remove some lines from the sphinx conf).

But the handcrafted latex macro have some advantages: they only "whitelist" the actually used characters, so typos are easily spotted, by the cost of adding a new macro each time we use a new out of the recognized set unicode character.

My point of view is: Documentation people should not have to care about Latex, they should not even know we're using latex in the background to build PDFs, so the xelatex looks the right solution to me.

I'm currently running a test build of an un-to-date Python 3.6 with PR 3940 applied to check if it succeed as expected. (I'm personally unable to type the macro to allow "Latin small letter dotless i" to work, so I won't test PR 31589).

It it succeed, I'll also provide screenshot showing the differences between the current PDF (pdflatex) and the xelatex one.
msg307445 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2017-12-02 18:15
It works with xelatex (https://github.com/python/cpython/pull/3940), here's the difference:

https://screenshots.firefox.com/wSn4a8WyFMNzXgKm/null
https://screenshots.firefox.com/WmUNljwcO8mImchP/null

The "bold" one is the xelatex one.

In case we want this to be merged, xelatex have already been installed on the build server (https://github.com/python/psf-salt/pull/122).
msg307451 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-02 19:35
Thanks, Julien.  To be clear, are you recommending applying just PR 3940 or also the unique part of PR 4069?  At this point, I am not concerned about the apparent minor difference in appearance; I would just like the downloadable docs to build and be usable.
msg307462 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2017-12-02 21:36
I recommand PR 3940, as it's simple, and as I'm still unable to fix issue32200 with PR 4069 (I still don't understand how to write those latex macros).

I'm open to follow the discussion about PR 4069, if one find a nice documentable reproductible protocol to update the list of latex macros to allow for a new character to be added.

Please not the french and japanese documentation may still fail (French should succeed with xelatex, japanese will still fail), which is expected, it's the origigin of PR 3940 and PR 4069, I'm still working on it, also see: https://github.com/python/docsbuild-scripts/pull/34.
msg307470 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-02 22:24
New changeset 7324b5ce8e7c031a0a3832a6a8d7c639111ae0ff by Ned Deily (Julien Palard) in branch 'master':
bpo-31589 : Build PDF using xelatex for better UTF8 support. (#3940)
https://github.com/python/cpython/commit/7324b5ce8e7c031a0a3832a6a8d7c639111ae0ff
msg307473 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-02 22:35
New changeset 2ad350a713360e89ae6d264924cd28f519b8b22c by Ned Deily (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31589 : Build PDF using xelatex for better UTF8 support. (GH-3940) (#4683)
https://github.com/python/cpython/commit/2ad350a713360e89ae6d264924cd28f519b8b22c
msg307474 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-02 22:39
OK, thanks.  I've pushed PR 3940 to master and backported it to 3.6.  I'll keep this open until we've seen the next full doc build complete successfully.
msg307484 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-12-03 03:21
I verified that the download links for Python 3.7.0a2  (English) are now working.

The French and Japanese docs downloads are still 404.
msg307489 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2017-12-03 08:28
Japanese is failing expectedly (may work with platex but not xelatex), but I hoped french to build. But the build server is giving:

    ! Improper discretionary list.
    <recently read> }
                
    l.359 ...{PyObject} \PYG{o}{*}\PYG{n}{t}\PYG{p}{;}
                                                  
    ? 
    ! Emergency stop.

I suspect it's because we're using an old version of xetex (2013.20140215-1ubuntu0.1) and I tested with a recent one (2017.20171128-1). I'm trying to reproduce the issue locally, and to understand the message.

In any cases, issue 32200 was about build failing since the introduction of the dotless i, which is now fixed, french and japanese problems are tracked here: https://bugs.python.org/issue31589.

I'd close issue 32200.
msg307491 - (view) Author: jfbu (jfbu) * Date: 2017-12-03 08:53
For info, the xetex problem "Improper discretionary list" is presumably the one seen at  https://github.com/sphinx-doc/sphinx/issues/3546. I asked on xetex mailing list at http://tug.org/pipermail/xetex/2017-March/027056.html to which xetex bug it was related but it appears I got no reply. Hence I don't know the precise xetex release which fixed it, but it is ok with `XeTeX 0.99996`. No action was taken on Sphinx side as this appeared to be a XeTeX bug only.
msg307520 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-03 18:18
OK, I'm going to close this.  We can discuss further steps elsewhere.  Thanks for reporting the problem, Carol, and thanks for the quick response, Julien!
History
Date User Action Args
2017-12-03 18:18:27ned.deilysetstatus: open -> closed
type: behavior ->
messages: + msg307520

resolution: fixed
stage: patch review -> resolved
2017-12-03 08:53:21jfbusetmessages: + msg307491
2017-12-03 08:28:03mdksetmessages: + msg307489
2017-12-03 03:21:15Mariattasetnosy: + Mariatta
messages: + msg307484
2017-12-02 22:39:26ned.deilysetmessages: + msg307474
2017-12-02 22:35:11ned.deilysetmessages: + msg307473
2017-12-02 22:24:55python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request4596
2017-12-02 22:24:41ned.deilysetmessages: + msg307470
2017-12-02 21:36:30mdksetmessages: + msg307462
2017-12-02 19:35:19ned.deilysetmessages: + msg307451
2017-12-02 18:15:13mdksetmessages: + msg307445
2017-12-02 17:46:15mdksetmessages: + msg307442
2017-12-02 17:35:52mdksetnosy: + jfbu
2017-12-02 15:44:19ned.deilysetnosy: + ned.deily, mdk
title: Download of 3.7.0a2 docs giving a 404 error -> Full docs build of 3.6 and 3.7 failing since 2017-10-15
messages: + msg307436

versions: + Python 3.6
2017-12-02 15:25:56willingccreate