Opening a ticket to track the progress of getting python 3.x build its own documentation.

So far I have installed Sphinx from http://bitbucket.org/birkenfeld/sphinx, but I get the following error:

$ sphinx-build -b html -d Doc/build3/doctrees -D latex_paper_size=  Doc  Doc/build3/html


There is a syntax error in your configuration file: invalid syntax (patchlevel.py, line 71)
Did you change the syntax from 2.x to 3.x?

I had some success after running 2to3 on Doc/conf.py and Doc/tools.  The build succeeded, but I have not compared the output yet.  Running doctest, however, still reported lots of errors and hang in turtle.rst. 

$ sphinx-build -b doctest -d Doc/build3/doctrees -D latex_paper_size=  Doc  Doc/build3/html

Incremental build was not that successful.  After editing Doc/faq/programming.rst, I get the following in the error log:


# Sphinx version: 1.1pre
# Python version: 3.2a3+.0
# Docutils version: 0.7 release
# Jinja2 version: 2.5.5
Traceback (most recent call last):
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/cmdline.py", line 173, in main
    app.build(force_all, filenames)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/application.py", line 203, in build
    self.builder.build_update()
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/__init__.py", line 193, in build_update
    'out of date' % len(to_build))
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/__init__.py", line 249, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/__init__.py", line 279, in write
    self.prepare_writing(docnames)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/html.py", line 231, in prepare_writing
    self.load_indexer(docnames)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/html.py", line 619, in load_indexer
    self.indexer.load(f, self.indexer_format)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 131, in load
    frozen = format.load(stream)
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 64, in load
    return self.loads(f.read())
  File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 55, in loads
    if not data or not s.startswith(self.PREFIX) or not \
TypeError: expected an object with the buffer interface

I fixed that bug in Sphinx rev 49747f5b0c70 (which I will push as soon as bitbucket is up again); incremental build now works for me.

In r85910, I ported the Python-specific modules in tools/ so that they don't need to be 2to3-converted.

Now you can basically easy_install Sphinx on 3.1 and run its sphinx-build without touching the Doc/ tree.

Did you know break building the 3.x documentation with Python 2? If so, please revert that change.

Nope, these files run just as fine in Python 2.

(The usual build process via Makefile still uses Python 2, and that won't change for 3.2.)

> The usual build process via Makefile still uses Python 2,
> and that won't change for 3.2.

Would you consider changing "doctest" make target to check out sphinx trunk (or 3.x compatible release) and run it with the py3k python?

This should not affect regular doc builds and running doctest under anything other than the current version of python makes very little sense.

Note that I am still targeting documentation fixes for 3.2.  See issue 10225.

As discussed on #python-dev, building  3.x documentation using python3.x will require Sphinx 1.1 which will not be released in time for 3.2.

I am adding #10225 as a dependency because I feel that fixing actual errors in ReST doctests is more important than upgrading documentation tool chain.  Hopefully this will bring more eyes to that issue.

This said, I believe it is important for python-dev to "eat their own dogfood" and make 3.2 self-hosting.  In other words, if we want to convince users that 3.x is ready, we should not require 2.x in the build process.

Fortunately, Doc/tools is not shipped with python releases, so the transition can be made at any time regardless of python release schedule.

I agree that 3.2 should eventually be independent of a 2.x Python.  Since the "port to 3.x" part of Sphinx 1.1 is basically done, I might just do an early alpha release and use that for Doc/tools before 3.2 final.

I ran smack into this while setting up my Doc build for the first time.  In trying to fix my build problems, it took me about an hour to find that this was a known issue.

I've attached a patch to warn others of the issue, in the documentation and the build itself.  Also, I've updated the version of Sphinx in building.rst, and made a note reminding others to do the same. I've only tested this patch on Windows 7.

I don’t see how the comment “As such, you will need to specify PYTHON explicitly” logically follows from the fact that Python 2.x is required.  IMO, building the docs require Python 2.x, so you’ll need a “python” executable on your PATH that is 2.x.

I had trouble wording that sentence.  Under Unix, Makefile is used, in which you specify "PYTHON=<path>" on the command line else the default "python" (from the PATH) is used.  Under Windows, it's make.bat, where PYTHON needs to be set in the environment else "..\pcbuild\python" is used (which ignores the PATH).  In either case, there's a high likelihood the PYTHON default will resolve to 3.3a0 for any Python developer.

building.rst doesn't mention any of this, though; README.txt does.  I wanted to keep the patch small, as I suspect this issue will be properly fixed soon, but I could create a new patch that better explains how to force that 2.x be used.

> In either case, there's a high likelihood the PYTHON default will
> resolve to 3.3a0 for any Python developer.

Ah, I understand the reason for the phrasing in your patch: UNIX systems typically install 3.x as python3, not python, so the high likelihood is only for Windows and a few UNIX systems like Arch.

Actually, on Windows, PYTHON is typically not set at all. So the likelihood of it being set to Python 3 is very low, unless you are trying to build Python documentation from time to time.

Sye: I fail to see the point of your patch. sphinx-build.py already issues an error message - why do you need more error messages?

When I built the documentation on Win7, it failed with a SyntaxError, and it took some digging to find the reason why.  I was hoping that issuing this warning would save others the trouble.

I agree: on Windows, PYTHON is rarely set.  However, look in make.bat: you'll see that when PYTHON is not already set, it's set to ..\pcbuild\python, which is Python 3.

What exactly is blocking this? From looking on PyPI it seems Jinja2, Pygments, and Sphinx have all been ported to Python 3 at this point. Is it a matter of porting any custom code in Doc/ to Python 3 and then updating the Makefile to grab Python 3 versions of the required libraries?

Doc/make.bat assumes that this issue has already been resolved:
  if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python
Even with '_d' appended, as it should be, that does not work for 3.x.
I think that line should be @rem'ed out until it does work.
Does anyone mind if I do so?

I already suggested on #15605 that a workaround be given until it is no longer needed.

> Doc/make.bat assumes that this issue has already been resolved:
>   if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python
> Even with '_d' appended, as it should be, that does not work for 3.x.
> I think that line should be @rem'ed out until it does work.
> Does anyone mind if I do so?

I have offered an alternative fix for that issue as a part of issue17386.

After a lot of experimentation, this worked for me to build AND doctest using the local python:

cd Doc/
../python -m venv toolsenv
curl http://python-distribute.org/distribute_setup.py | toolsenv/bin/python
toolsenv/bin/easy_install pip
toolsenv/bin/pip install Sphinx
# EDIT conf.py: add 'toolsenv' to exclude_trees.
toolsenv/bin/sphinx-build -b html -d build/doctrees -D latex_paper_size=  . build/html
toolsenv/bin/sphinx-build -b doctest -d build/doctrees . build/doctest

The tool versions I got:

$ toolsenv/bin/pip list
distribute (0.6.35)
docutils (0.10)
Jinja2 (2.6)
Pygments (1.6)
Sphinx (1.1.3)

HTML: http://dl.dropbox.com/u/132894668/python3-Doc-build/html/index.html
The HTML has no significant differences vs the output of "make html".
It's mostly sphinx html tweaks.  The only user-visible diff I noticed is s/v3.4.0a0/3.4.0a0/.

The doctests are pretty horrible - 559 failed out of 1954!  (attached)
But it's better than 900+ failed with python 2 ("make doctest") and 700+ with python 3.2 that I originally tried via "apt-get install python3-sphinx"...

So let's update the tools ASAP, and start using PYTHON=../python.
The doctests won't get any love unless "make doctest" does the right thing out of the box.

Should I write a Makefile patch using the venv/install from pypi approach?
Or could somebody upload new versions to http://svn.python.org/projects/external/?
(Lack of access there was the reason I went the venv way.
IMHO the ease of future upgrades outweighs the concern of not depending on external sites, but I don't feel strongly about it.)

> Should I write a Makefile patch using the venv/install from pypi approach?
> Or could somebody upload new versions to http://svn.python.org/projects/external/?

I looked at this issue over the past few days and discussed briefly with Georg about it.  The 3 main issues are:
  1) Sphinx and its dependency have a 2to3-able source that needs to be converted to Python 3 before being used.  Should this be done every time or should we provide the already converted source somewhere?
  2) If we go for the latter, should it be on externals/ (which still uses SVN) or on a new hg repo?
  3) since we don't actually need all the history, should we still just clone these repos or is it better to download the source only?

IMHO for 1) it would be better to provide Python3-ready code.  For 2) it would also be nice to get rid of the SVN dependency for the docs so if we are going to use a repo it might be better to make a new one on HG.  For 3), I'm not sure it makes sense to use repos at all if we don't want the history.  If we don't, we would have to put the files somewhere and write some code to download and extract the zip/tarball.  OTOH, if we do, we will be able to use hg to clone and update the repo easily, and, if we still decide not to download the history, we can just use the zip/tarballs already provided by hg.python.org.

A problem with using automatic tarballs from hg.python.org is that they take server load to generate, every time someone builds the docs from a new checkout.

Apparently they are cached, and unless the cache expires often, the server doesn't need to regenerate the tarball too often.  However I don't know the details of the caching, and if they can be tweaked.

Ok, that sounds good then.

BTW, for 3.3+ would it make sense instead to create a virtualenv and directly install the required versions from pypi?

I guess if we are willing to write a mini-pip to download the latest release directly from PyPI we could. Since there are no C extensions we don't have to care about the lack of header files in Include if we are going to build from the repo build directly. Or are you more thinking that we just require 3.3 externally to build and not rely on the repo build unless absolutely necessary? Just wondering about when the venv environment copies instead of symlinks the binary.

I was only thinking of 3.4, which will have venv and a pip bootstrapper.
Is changing the doc build / doctest in scope for minor releases of 3.3 (or even earlier)?

The commands I listed (using setup_distribute.py) also work with 3.3.
(But they're unsecure -- https://bitbucket.org/tarek/distribute/issue/374/)
3.2 is harder as it doesn't even have builtin venv.

For 3.4.1 and 3.5, the Doc/Makefile no longer explicitly vendors sphinx and its dependencies (via svn), rather it assumes that an externally-supplied sphinx-build is available, for example, via a venv and pip.  And the current versions of sphinx and its dependencies are all supported on current Py3's.  So can this issue be closed, Georg?

Yep, thanks for the reminder.

Do we need to change Doc/Makefile to have “PYTHON = python3” or even “PYTHON = ./python” (so doctests run with the development version)?

Éric, I think you should open a separate issue for this.