Currently, the 3.7.0a2 doc links are giving 404 errors. 3.6 docs are working just fine.

https://docs.python.org/3.7/archives/python-3.7.0a2-docs-pdf-letter.tar.bz2 results in a 404.

I wasn't sure if I should report here or in python.org issue tracker.

It looks like the 3.6 and 3.7 full doc build has been failing since 2017-10-15.  Seen in /var/log/docsbuild/python37-en.log:

[...]
[3] [4] [5] [6] [7] (/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-1.def
) (/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uninames.dat)
(/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-1.def)

! Package ucs Error: Unknown Unicode character 383 = U+017F,
(ucs)                possibly declared in uni-1.def.
(ucs)                Type H to see if it is available with options.

See the ucs package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.781 Latin small letter dotless i), ‘ſ
                                          ’ (U+017F, Latin small letter lo...

? 
! Emergency stop.
 ...                                              
                                                  
l.781 Latin small letter dotless i), ‘ſ
                                          ’ (U+017F, Latin small letter lo...

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on howto-regex.log.
Latexmk: Index file 'howto-regex.idx' was written
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 256
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs of latex/pdflatex.
Latexmk: Errors, so I did not complete making targets
make[2]: *** [howto-regex.pdf] Error 12
make[2]: Leaving directory `/srv/docsbuild/python37/Doc/build/latex'
make[1]: *** [dist] Error 2
make[1]: Leaving directory `/srv/docsbuild/python37/Doc'
make: *** [autobuild-dev] Error 2

Julien, can you take a look please?

Yes, it's related to:
- https://github.com/python/cpython/pull/3940
- https://github.com/python/cpython/pull/4069

Idea is: Unicode characters are not well supported by pdflatex which what's we're using to build our PDF files.

There's two solutions:

- Use xelatex which just works but does not use the same fonts, and the output style is slightly different
- Use hard handcrafted latex macros to make pdflatex work

I'm personally in favor of the xelatex solution (The simple one: the patch even remove some lines from the sphinx conf).

But the handcrafted latex macro have some advantages: they only "whitelist" the actually used characters, so typos are easily spotted, by the cost of adding a new macro each time we use a new out of the recognized set unicode character.

My point of view is: Documentation people should not have to care about Latex, they should not even know we're using latex in the background to build PDFs, so the xelatex looks the right solution to me.

I'm currently running a test build of an un-to-date Python 3.6 with PR 3940 applied to check if it succeed as expected. (I'm personally unable to type the macro to allow "Latin small letter dotless i" to work, so I won't test PR 31589).

It it succeed, I'll also provide screenshot showing the differences between the current PDF (pdflatex) and the xelatex one.

It works with xelatex (https://github.com/python/cpython/pull/3940), here's the difference:

https://screenshots.firefox.com/wSn4a8WyFMNzXgKm/null
https://screenshots.firefox.com/WmUNljwcO8mImchP/null

The "bold" one is the xelatex one.

In case we want this to be merged, xelatex have already been installed on the build server (https://github.com/python/psf-salt/pull/122).

Thanks, Julien.  To be clear, are you recommending applying just PR 3940 or also the unique part of PR 4069?  At this point, I am not concerned about the apparent minor difference in appearance; I would just like the downloadable docs to build and be usable.

I recommand PR 3940, as it's simple, and as I'm still unable to fix issue32200 with PR 4069 (I still don't understand how to write those latex macros).

I'm open to follow the discussion about PR 4069, if one find a nice documentable reproductible protocol to update the list of latex macros to allow for a new character to be added.

Please not the french and japanese documentation may still fail (French should succeed with xelatex, japanese will still fail), which is expected, it's the origigin of PR 3940 and PR 4069, I'm still working on it, also see: https://github.com/python/docsbuild-scripts/pull/34.

New changeset 7324b5ce8e7c031a0a3832a6a8d7c639111ae0ff by Ned Deily (Julien Palard) in branch 'master':
bpo-31589 : Build PDF using xelatex for better UTF8 support. (#3940)
https://github.com/python/cpython/commit/7324b5ce8e7c031a0a3832a6a8d7c639111ae0ff

New changeset 2ad350a713360e89ae6d264924cd28f519b8b22c by Ned Deily (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31589 : Build PDF using xelatex for better UTF8 support. (GH-3940) (#4683)
https://github.com/python/cpython/commit/2ad350a713360e89ae6d264924cd28f519b8b22c

OK, thanks.  I've pushed PR 3940 to master and backported it to 3.6.  I'll keep this open until we've seen the next full doc build complete successfully.

I verified that the download links for Python 3.7.0a2  (English) are now working.

The French and Japanese docs downloads are still 404.

Japanese is failing expectedly (may work with platex but not xelatex), but I hoped french to build. But the build server is giving:

    ! Improper discretionary list.
    <recently read> }
                
    l.359 ...{PyObject} \PYG{o}{*}\PYG{n}{t}\PYG{p}{;}
                                                  
    ? 
    ! Emergency stop.

I suspect it's because we're using an old version of xetex (2013.20140215-1ubuntu0.1) and I tested with a recent one (2017.20171128-1). I'm trying to reproduce the issue locally, and to understand the message.

In any cases, issue 32200 was about build failing since the introduction of the dotless i, which is now fixed, french and japanese problems are tracked here: https://bugs.python.org/issue31589.

I'd close issue 32200.

For info, the xetex problem "Improper discretionary list" is presumably the one seen at  https://github.com/sphinx-doc/sphinx/issues/3546. I asked on xetex mailing list at http://tug.org/pipermail/xetex/2017-March/027056.html to which xetex bug it was related but it appears I got no reply. Hence I don't know the precise xetex release which fixed it, but it is ok with `XeTeX 0.99996`. No action was taken on Sphinx side as this appeared to be a XeTeX bug only.

OK, I'm going to close this.  We can discuss further steps elsewhere.  Thanks for reporting the problem, Carol, and thanks for the quick response, Julien!

New changeset d9b321692b6770c0fe7cb7ba05f9bd5c6082a880 by Miss Islington (bot) in branch '2.7':
bpo-31589 : Build PDF using xelatex for better UTF8 support. (GH-3940)
https://github.com/python/cpython/commit/d9b321692b6770c0fe7cb7ba05f9bd5c6082a880