classification
Title: Build 3.x documentation using python3.x
Type: enhancement Stage: needs patch
Components: Build, Documentation Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: out of date
Dependencies: 16471 Superseder:
Assigned To: Nosy List: asvetlov, belopolsky, brett.cannon, cben, chris.jerdonek, daniel.urban, docs@python, eric.araujo, eric.snow, ezio.melotti, georg.brandl, loewis, ned.deily, r.david.murray, syeberman, terry.reedy, tshepang, zach.ware, zbysz
Priority: normal Keywords: patch

Created on 2010-10-28 22:16 by belopolsky, last changed 2014-07-02 03:49 by ezio.melotti. This issue is now closed.

Files
File name Uploaded Description Edit
Issue10224Warnings.patch syeberman, 2011-05-23 13:15 Patch to add warnings of Python 2.x dependence to build, docs (does NOT fix the issue) review
output.txt cben, 2013-03-21 03:17 build/doctest/output.txt with locally built python
Messages (31)
msg119819 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-28 22:16
Opening a ticket to track the progress of getting python 3.x build its own documentation.

So far I have installed Sphinx from http://bitbucket.org/birkenfeld/sphinx, but I get the following error:

$ sphinx-build -b html -d Doc/build3/doctrees -D latex_paper_size=  Doc  Doc/build3/html


There is a syntax error in your configuration file: invalid syntax (patchlevel.py, line 71)
Did you change the syntax from 2.x to 3.x?
msg119821 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-28 22:39
I had some success after running 2to3 on Doc/conf.py and Doc/tools.  The build succeeded, but I have not compared the output yet.  Running doctest, however, still reported lots of errors and hang in turtle.rst. 

$ sphinx-build -b doctest -d Doc/build3/doctrees -D latex_paper_size=  Doc  Doc/build3/html
msg119851 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-29 04:07
Incremental build was not that successful.  After editing Doc/faq/programming.rst, I get the following in the error log:


# Sphinx version: 1.1pre
# Python version: 3.2a3+.0
# Docutils version: 0.7 release
# Jinja2 version: 2.5.5
Traceback (most recent call last):
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/cmdline.py", line 173, in main
    app.build(force_all, filenames)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/application.py", line 203, in build
    self.builder.build_update()
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/__init__.py", line 193, in build_update
    'out of date' % len(to_build))
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/__init__.py", line 249, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/__init__.py", line 279, in write
    self.prepare_writing(docnames)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/html.py", line 231, in prepare_writing
    self.load_indexer(docnames)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/html.py", line 619, in load_indexer
    self.indexer.load(f, self.indexer_format)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 131, in load
    frozen = format.load(stream)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 64, in load
    return self.loads(f.read())
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 55, in loads
    if not data or not s.startswith(self.PREFIX) or not \
TypeError: expected an object with the buffer interface
msg119854 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-29 05:19
I fixed that bug in Sphinx rev 49747f5b0c70 (which I will push as soon as bitbucket is up again); incremental build now works for me.
msg119856 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-29 05:31
In r85910, I ported the Python-specific modules in tools/ so that they don't need to be 2to3-converted.

Now you can basically easy_install Sphinx on 3.1 and run its sphinx-build without touching the Doc/ tree.
msg119861 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-10-29 06:40
Did you know break building the 3.x documentation with Python 2? If so, please revert that change.
msg119863 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-29 06:55
Nope, these files run just as fine in Python 2.
msg119864 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-29 06:56
(The usual build process via Makefile still uses Python 2, and that won't change for 3.2.)
msg119910 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-29 15:50
> The usual build process via Makefile still uses Python 2,
> and that won't change for 3.2.

Would you consider changing "doctest" make target to check out sphinx trunk (or 3.x compatible release) and run it with the py3k python?

This should not affect regular doc builds and running doctest under anything other than the current version of python makes very little sense.

Note that I am still targeting documentation fixes for 3.2.  See issue 10225.
msg120765 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-08 16:12
As discussed on #python-dev, building  3.x documentation using python3.x will require Sphinx 1.1 which will not be released in time for 3.2.

I am adding #10225 as a dependency because I feel that fixing actual errors in ReST doctests is more important than upgrading documentation tool chain.  Hopefully this will bring more eyes to that issue.

This said, I believe it is important for python-dev to "eat their own dogfood" and make 3.2 self-hosting.  In other words, if we want to convince users that 3.x is ready, we should not require 2.x in the build process.

Fortunately, Doc/tools is not shipped with python releases, so the transition can be made at any time regardless of python release schedule.
msg120766 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-11-08 16:20
I agree that 3.2 should eventually be independent of a 2.x Python.  Since the "port to 3.x" part of Sphinx 1.1 is basically done, I might just do an early alpha release and use that for Doc/tools before 3.2 final.
msg136629 - (view) Author: Sye van der Veen (syeberman) * Date: 2011-05-23 13:15
I ran smack into this while setting up my Doc build for the first time.  In trying to fix my build problems, it took me about an hour to find that this was a known issue.

I've attached a patch to warn others of the issue, in the documentation and the build itself.  Also, I've updated the version of Sphinx in building.rst, and made a note reminding others to do the same. I've only tested this patch on Windows 7.
msg136637 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-05-23 13:57
I don’t see how the comment “As such, you will need to specify PYTHON explicitly” logically follows from the fact that Python 2.x is required.  IMO, building the docs require Python 2.x, so you’ll need a “python” executable on your PATH that is 2.x.
msg136814 - (view) Author: Sye van der Veen (syeberman) * Date: 2011-05-25 00:24
I had trouble wording that sentence.  Under Unix, Makefile is used, in which you specify "PYTHON=<path>" on the command line else the default "python" (from the PATH) is used.  Under Windows, it's make.bat, where PYTHON needs to be set in the environment else "..\pcbuild\python" is used (which ignores the PATH).  In either case, there's a high likelihood the PYTHON default will resolve to 3.3a0 for any Python developer.

building.rst doesn't mention any of this, though; README.txt does.  I wanted to keep the patch small, as I suspect this issue will be properly fixed soon, but I could create a new patch that better explains how to force that 2.x be used.
msg137971 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-09 14:33
> In either case, there's a high likelihood the PYTHON default will
> resolve to 3.3a0 for any Python developer.

Ah, I understand the reason for the phrasing in your patch: UNIX systems typically install 3.x as python3, not python, so the high likelihood is only for Windows and a few UNIX systems like Arch.
msg138297 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-06-14 06:26
Actually, on Windows, PYTHON is typically not set at all. So the likelihood of it being set to Python 3 is very low, unless you are trying to build Python documentation from time to time.

Sye: I fail to see the point of your patch. sphinx-build.py already issues an error message - why do you need more error messages?
msg140252 - (view) Author: Sye van der Veen (syeberman) * Date: 2011-07-13 14:14
When I built the documentation on Win7, it failed with a SyntaxError, and it took some digging to find the reason why.  I was hoping that issuing this warning would save others the trouble.

I agree: on Windows, PYTHON is rarely set.  However, look in make.bat: you'll see that when PYTHON is not already set, it's set to ..\pcbuild\python, which is Python 3.
msg175768 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-11-17 16:21
What exactly is blocking this? From looking on PyPI it seems Jinja2, Pygments, and Sphinx have all been ported to Python 3 at this point. Is it a matter of porting any custom code in Doc/ to Python 3 and then updating the Makefile to grab Python 3 versions of the required libraries?
msg183706 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-07 21:54
Doc/make.bat assumes that this issue has already been resolved:
  if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python
Even with '_d' appended, as it should be, that does not work for 3.x.
I think that line should be @rem'ed out until it does work.
Does anyone mind if I do so?

I already suggested on #15605 that a workaround be given until it is no longer needed.
msg183838 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-03-09 19:06
> Doc/make.bat assumes that this issue has already been resolved:
>   if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python
> Even with '_d' appended, as it should be, that does not work for 3.x.
> I think that line should be @rem'ed out until it does work.
> Does anyone mind if I do so?

I have offered an alternative fix for that issue as a part of issue17386.
msg184838 - (view) Author: Cherniavsky Beni (cben) * Date: 2013-03-21 03:17
After a lot of experimentation, this worked for me to build AND doctest using the local python:

cd Doc/
../python -m venv toolsenv
curl http://python-distribute.org/distribute_setup.py | toolsenv/bin/python
toolsenv/bin/easy_install pip
toolsenv/bin/pip install Sphinx
# EDIT conf.py: add 'toolsenv' to exclude_trees.
toolsenv/bin/sphinx-build -b html -d build/doctrees -D latex_paper_size=  . build/html
toolsenv/bin/sphinx-build -b doctest -d build/doctrees . build/doctest

The tool versions I got:

$ toolsenv/bin/pip list
distribute (0.6.35)
docutils (0.10)
Jinja2 (2.6)
Pygments (1.6)
Sphinx (1.1.3)

HTML: http://dl.dropbox.com/u/132894668/python3-Doc-build/html/index.html
The HTML has no significant differences vs the output of "make html".
It's mostly sphinx html tweaks.  The only user-visible diff I noticed is s/v3.4.0a0/3.4.0a0/.

The doctests are pretty horrible - 559 failed out of 1954!  (attached)
But it's better than 900+ failed with python 2 ("make doctest") and 700+ with python 3.2 that I originally tried via "apt-get install python3-sphinx"...

So let's update the tools ASAP, and start using PYTHON=../python.
The doctests won't get any love unless "make doctest" does the right thing out of the box.

Should I write a Makefile patch using the venv/install from pypi approach?
Or could somebody upload new versions to http://svn.python.org/projects/external/?
(Lack of access there was the reason I went the venv way.
IMHO the ease of future upgrades outweighs the concern of not depending on external sites, but I don't feel strongly about it.)
msg184872 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-21 14:36
> Should I write a Makefile patch using the venv/install from pypi approach?
> Or could somebody upload new versions to http://svn.python.org/projects/external/?

I looked at this issue over the past few days and discussed briefly with Georg about it.  The 3 main issues are:
  1) Sphinx and its dependency have a 2to3-able source that needs to be converted to Python 3 before being used.  Should this be done every time or should we provide the already converted source somewhere?
  2) If we go for the latter, should it be on externals/ (which still uses SVN) or on a new hg repo?
  3) since we don't actually need all the history, should we still just clone these repos or is it better to download the source only?

IMHO for 1) it would be better to provide Python3-ready code.  For 2) it would also be nice to get rid of the SVN dependency for the docs so if we are going to use a repo it might be better to make a new one on HG.  For 3), I'm not sure it makes sense to use repos at all if we don't want the history.  If we don't, we would have to put the files somewhere and write some code to download and extract the zip/tarball.  OTOH, if we do, we will be able to use hg to clone and update the repo easily, and, if we still decide not to download the history, we can just use the zip/tarballs already provided by hg.python.org.
msg184874 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-03-21 15:13
A problem with using automatic tarballs from hg.python.org is that they take server load to generate, every time someone builds the docs from a new checkout.
msg184884 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-21 16:30
Apparently they are cached, and unless the cache expires often, the server doesn't need to regenerate the tarball too often.  However I don't know the details of the caching, and if they can be tweaked.
msg184892 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-03-21 17:29
Ok, that sounds good then.

BTW, for 3.3+ would it make sense instead to create a virtualenv and directly install the required versions from pypi?
msg184893 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-21 17:38
I guess if we are willing to write a mini-pip to download the latest release directly from PyPI we could. Since there are no C extensions we don't have to care about the lack of header files in Include if we are going to build from the repo build directly. Or are you more thinking that we just require 3.3 externally to build and not rely on the repo build unless absolutely necessary? Just wondering about when the venv environment copies instead of symlinks the binary.
msg190396 - (view) Author: Cherniavsky Beni (cben) * Date: 2013-05-31 09:41
I was only thinking of 3.4, which will have venv and a pip bootstrapper.
Is changing the doc build / doctest in scope for minor releases of 3.3 (or even earlier)?

The commands I listed (using setup_distribute.py) also work with 3.3.
(But they're unsecure -- https://bitbucket.org/tarek/distribute/issue/374/)
3.2 is harder as it doesn't even have builtin venv.
msg216161 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-04-14 18:34
For 3.4.1 and 3.5, the Doc/Makefile no longer explicitly vendors sphinx and its dependencies (via svn), rather it assumes that an externally-supplied sphinx-build is available, for example, via a venv and pip.  And the current versions of sphinx and its dependencies are all supported on current Py3's.  So can this issue be closed, Georg?
msg216185 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-04-14 19:24
Yep, thanks for the reminder.
msg216187 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-04-14 19:36
Do we need to change Doc/Makefile to have “PYTHON = python3” or even “PYTHON = ./python” (so doctests run with the development version)?
msg222066 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-07-02 03:49
Éric, I think you should open a separate issue for this.
History
Date User Action Args
2014-07-02 03:49:21ezio.melottisetmessages: + msg222066
2014-04-14 19:36:13eric.araujosetmessages: + msg216187
2014-04-14 19:24:45georg.brandlsetstatus: open -> closed
resolution: out of date
messages: + msg216185
2014-04-14 18:34:37ned.deilysetversions: + Python 3.5
nosy: + ned.deily

messages: + msg216161

assignee: docs@python ->
2013-10-13 19:11:21ezio.melottilinkissue19248 superseder
2013-05-31 09:41:45cbensetmessages: + msg190396
2013-03-21 17:38:24brett.cannonsetmessages: + msg184893
2013-03-21 17:29:55georg.brandlsetmessages: + msg184892
2013-03-21 16:48:04r.david.murraysetnosy: + r.david.murray
2013-03-21 16:30:49ezio.melottisetmessages: + msg184884
2013-03-21 15:13:23georg.brandlsetmessages: + msg184874
2013-03-21 14:36:01ezio.melottisetmessages: + msg184872
2013-03-21 03:17:12cbensetfiles: + output.txt
nosy: + cben
messages: + msg184838

2013-03-15 01:10:23terry.reedylinkissue16471 superseder
2013-03-09 19:06:02zach.waresetnosy: + zach.ware
messages: + msg183838
2013-03-07 22:39:56tshepangsetnosy: + tshepang
2013-03-07 21:54:09terry.reedysetnosy: + terry.reedy
messages: + msg183706
2012-11-17 17:42:38ezio.melottisetdependencies: + upgrade to sphinx 1.1
2012-11-17 16:21:11brett.cannonsetdependencies: - Fix doctest runable examples in python manual
messages: + msg175768
2012-09-12 21:47:54chris.jerdoneksetnosy: + chris.jerdonek
2012-09-12 21:36:20brett.cannonsetversions: + Python 3.4, - Python 3.3
2012-09-08 16:27:04brett.cannonsetnosy: + brett.cannon
2012-08-13 13:54:55asvetlovsetnosy: + asvetlov
2011-07-13 16:06:26eric.snowsetnosy: + eric.snow
2011-07-13 14:14:49syebermansetmessages: + msg140252
2011-06-14 06:26:11loewissetmessages: + msg138297
2011-06-09 14:33:27eric.araujosetmessages: + msg137971
2011-05-28 10:05:53ezio.melottisetnosy: + ezio.melotti
2011-05-25 00:24:32syebermansetmessages: + msg136814
2011-05-23 13:57:44eric.araujosetmessages: + msg136637
2011-05-23 13:15:46syebermansetfiles: + Issue10224Warnings.patch

nosy: + syeberman
messages: + msg136629

keywords: + patch
2011-01-20 23:34:37brett.cannonlinkissue10965 dependencies
2010-11-08 16:20:13georg.brandlsetmessages: + msg120766
2010-11-08 16:12:44belopolskysettype: enhancement
dependencies: + Fix doctest runable examples in python manual
messages: + msg120765
stage: needs patch
2010-10-30 09:37:21zbyszsetnosy: + zbysz
2010-10-29 16:50:14daniel.urbansetnosy: + daniel.urban
2010-10-29 15:50:45belopolskysetmessages: + msg119910
2010-10-29 06:56:25georg.brandlsetmessages: + msg119864
2010-10-29 06:55:35georg.brandlsetmessages: + msg119863
2010-10-29 06:40:29loewissetnosy: + loewis
messages: + msg119861
2010-10-29 05:31:25georg.brandlsetmessages: + msg119856
2010-10-29 05:20:00georg.brandlsetmessages: + msg119854
versions: + Python 3.3, - Python 3.2
2010-10-29 04:07:12belopolskysetmessages: + msg119851
2010-10-29 02:19:06belopolskylinkissue10225 dependencies
2010-10-28 22:39:05belopolskysetmessages: + msg119821
2010-10-28 22:29:53belopolskysetassignee: docs@python

nosy: + docs@python
components: + Build, Documentation
versions: + Python 3.2
2010-10-28 22:29:14belopolskysetnosy: + georg.brandl, eric.araujo
2010-10-28 22:16:37belopolskycreate