The following error is caused in Docs build for a 3.8 backport since sphinx is ran with warnings. Sphinx released 3.0 on April 6. The last successful build on master uses Sphinx 2.2.0 [0]. My guess is sphinx new version possibly breaking the build on Python 3.8 where it's not pinned to use 2.2.0 pulling the latest version. The changelog for Sphinx has below note :

https://www.sphinx-doc.org/en/master/changes.html#release-3-0-0-released-apr-06-2020 

The C domain has been rewritten, with additional directives and roles. The existing ones are now more strict, resulting in new warnings.

Python 3.8 and Python 3.7 doesn't have Sphinx pinned to 2.2.0 while master does.

Python 3.8 Docs makefile : https://github.com/python/cpython/blob/f7b0259d0d243a71d79a3fda9ec7aad4306513eb/Doc/Makefile#L146

Failed build :

https://github.com/python/cpython/pull/19388/checks?check_run_id=563053793#step:7:46

Error :

Warning, treated as error:
/home/runner/work/cpython/cpython/Doc/c-api/buffer.rst:92:Error in declarator or parameters
Invalid C declaration: Expected identifier in nested name. [error at 5]
  void \*buf
  -----^
Makefile:49: recipe for target 'build' failed
make[1]: *** [build] Error 2


[0] https://github.com/python/cpython/runs/564123625#step:6:24

It sounds dangerous to not pin the Sphinx version in our CI :-/ Another issue caused by CI configuration stored at the same place than the code:
https://mail.python.org/archives/list/python-committers@python.org/thread/WEU5CQKIA4LIHWHT53YA7HHNUY5H2FUT/

Maybe copy the code for deprecated and removed features to Doc/tools/extensions?

The sphinx version is not pinned in 3.7, 3.6, 3.5 and 2.7 branches too for Doc/Makefile that can cause error on someone trying it out locally. They are pinned in .travis.yml and .azure-pipelines configurations.

> Python 3.8 and Python 3.7 doesn't have Sphinx pinned to 2.2.0 while master does.

In 3.8, .azure-pipelines/docs-steps.yml contains:

- script: python -m pip install sphinx==1.8.2 blurb python-docs-theme

and .travis.yml contains:

- python -m pip install sphinx==1.8.2 blurb python-docs-theme

For example, the Sphinx version was changed from 1.8.1 to 1.8.2 in the CI configuration by:

commit 7f4ba4afd47f21f61de9035544809fc67d136f35
Author: Julien Palard <julien@palard.fr>
Date:   Sat Nov 24 11:35:21 2018 +0100

    Doc: Bump sphinx. (GH-10676)

--

I guess that you're talking about Doc/Makefile which uses "Sphinx" in 3.8 but "Sphinx==2.2.0" in master. Code in 3.8:

venv:
	$(PYTHON) -m venv $(VENVDIR)
	$(VENVDIR)/bin/python3 -m pip install -U pip setuptools
	$(VENVDIR)/bin/python3 -m pip install -U Sphinx blurb python-docs-theme
	@echo "The venv has been created in the $(VENVDIR) directory"

> I guess that you're talking about Doc/Makefile which uses "Sphinx" in 3.8 but "Sphinx==2.2.0" in master.

I wrote PR 19442 to pin Sphinx version to 1.8.2 in Doc/Makefile.

New changeset 37a257c0ae0d4ba746397ae7584db887b175ab24 by Victor Stinner in branch '3.8':
bpo-40204: Pin Sphinx version to 1.8.2 in Doc/Makefile (GH-19442)
https://github.com/python/cpython/commit/37a257c0ae0d4ba746397ae7584db887b175ab24

I have filed an issue upstream and it's fixed. The release of 3.0.1 is planned in few days and could help for other branches. But it would be nice to see the version pinned to avoid these problems in future. Upstream report : https://github.com/sphinx-doc/sphinx/issues/7423#event-3218761694

New changeset 9e5f159d3279a6b476d588010d529cfdbb8c8803 by Victor Stinner in branch '3.7':
bpo-40204: Pin Sphinx version to 1.8.2 in Doc/Makefile (GH-19442) (GH-19443)
https://github.com/python/cpython/commit/9e5f159d3279a6b476d588010d529cfdbb8c8803

Why are we pinning to 1.8.2 when the official docs builds for all releases are currently using 2.3.1?  (see, for example, https://docs.python.org/3.8/ at bottom right corner)

> Why are we pinning to 1.8.2 when the official docs builds for all releases are currently using 2.3.1?  (see, for example, https://docs.python.org/3.8/ at bottom right corner)

First of all, to repair the CI :-) Before my change, it was no longer possible to merge any change in 3.8! (I didn't check for 3.7, but I expect that it wasn't possible neither).

Second, 1.8.2 version comes from the CI configuration:

$ grep -i sphinx== .travis.yml .azure-pipelines/* Doc/Makefile 
.travis.yml:        - python -m pip install sphinx==1.8.2 blurb
.azure-pipelines/docs-steps.yml:- script: python -m pip install sphinx==1.8.2 blurb python-docs-theme
Doc/Makefile:	$(VENVDIR)/bin/python3 -m pip install -U Sphinx==1.8.2 blurb

If Sphinx version is changed in Doc/Makefile, I would prefer to also update the version in the CI configuration.

I also vaguely recall discussions about issues with newer Sphinx. I don't recall if it was 2.0 or another version. Julien Palard may recall that better than me! Was it bpo-35472?

--

I don't really care of the Sphinx version, I only care about working CI and consistency between all pinned versions ;-)

--

By the way, maybe Doc/Makefile could use the latest Sphinx version by default, and the Python CI could use a pinned version. The problem is that the Sphinx version cannot be configured currently when running "make venv".

Oh, .travis.yml is differecen between 3.7 and 3.8.

3.8 and master use "make -C Doc/ PYTHON=../python venv".

3.7 doesn't use "make venv" but:

  before_script:
    - cd Doc
    # Sphinx is pinned so that new versions that introduce new warnings won't suddenly cause build failures.
    # (Updating the version is fine as long as no warnings are raised by doing so.)
    - python -m pip install sphinx==1.8.2 blurb
  script:
    - make check suspicious html SPHINXOPTS="-q -W -j4"


Maybe 3.7 should mimick what is done in 3.8 and master to ease maintenance. I don't know.

I agree that it should be easier to keep them all in sync.  But my point is that the on-going official doc builds (some multi[ple times per day) for all of the active versions (2.7 and 3.6 through 3.9) are using 2.3.1 so we should be doing CI against that version as well.  The docs builds are configured in https://github.com/python/docsbuild-scripts.  So whenever requirements.in is update there, the CI versions should be updated as well. Even better would be someway to auto sync them.

Julien, any ideas on this?

A few days ago, 3.8 backports failed while 3.7 backports merged. Thank you for the fix, even if temporary.

I just noticed that the doc main page ends with "Created using Sphinx 2.3.1."  I plan to switch my local sphinx to that version.

New changeset 0d70a98f449d58311aff3374ab5f203a401fc3bf by Ned Deily in branch '3.7':
bpo-40204: Pin Sphinx version to 2.3.1 in Doc/Makefile
https://github.com/python/cpython/commit/0d70a98f449d58311aff3374ab5f203a401fc3bf

I look into this issue. It's quite complicated :-(

The Sphinx 3.0.0b1 changelog says:

* The C domain has been rewritten, with additional directives and roles. The existing ones are now more strict, resulting in new warnings.
* The C domain has been rewritten adding for example:

  * Cross-referencing respecting the current scope.
  * Possible to document anonymous entities.
  * More specific directives and roles for each type of entitiy, e.g., handling scoping of enumerators.
  * New role c:expr for rendering expressions and types in text.

The Python documentation has many issues:

* Doc/howto/instrumentation.rst: ".. c:function:: function__entry(str filename, str funcname, int lineno)" is not valid
  => replace "str" with "char*" and explain that "char*" must be read as "str"... The documentation is about DTrace functions, it's not C code.

* Doc/library/configparser.rst: "unique" is declared twice
  => the second declaration must have :noindex: marker

* ".. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ..., NULL)": the "..., NULL" part is invalid in C => remove ", NULL"

* .. c:var:: PY_VECTORCALL_ARGUMENTS_OFFSET: a variable must be declared with its type, but PY_VECTORCALL_ARGUMENTS_OFFSET is macro
  => use ".. c:macro::" instead

* ".. c:var:: Py_BytesWarningFlag": the type is missing
  => ".. c:var:: int Py_BytesWarningFlag"

* Doc/c-api/call.rst: vectorcallfunc documentation is not properly indented.
  => fix indentation

* Doc/c-api/buffer.rst: the table with ".. c:macro:: PyBUF_INDIRECT" is wrong. Sphinx fails because PyBUF_SIMPLE is declared twice.
  => use ":c:macro:`PyBUF_INDIRECT`" instead

* :c:type:`PyObject\*` is invalid
  => use  :c:type:`PyObject*`

* ... there are many other errors :-(

PR 19397 fix some errors, but not all of them.

Sphinx documentation of the C domain:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#the-c-domain

Don't hesitate to send requests back to the Sphinx tracker if any changes appear to be too restrictive to you. It is a documentation tool, not a compiler.

"""
* Doc/c-api/buffer.rst: the table with ".. c:macro:: PyBUF_INDIRECT" is wrong. Sphinx fails because PyBUF_SIMPLE is declared twice.
  => use ":c:macro:`PyBUF_INDIRECT`" instead
"""

Hmm, grep shows:

c-api/buffer.rst:123:      most cases such a request will be :c:macro:`PyBUF_SIMPLE` or :c:macro:`PyBUF_WRITABLE`.
c-api/buffer.rst:144:      If :c:member:`~Py_buffer.shape` is ``NULL`` as a result of a :c:macro:`PyBUF_SIMPLE`
c-api/buffer.rst:255:Since :c:macro:`PyBUF_SIMPLE` is defined as 0, :c:macro:`PyBUF_WRITABLE`
c-api/buffer.rst:258::c:macro:`PyBUF_FORMAT` can be \|'d to any of the flags except :c:macro:`PyBUF_SIMPLE`.
c-api/buffer.rst:280:| .. c:macro:: PyBUF_SIMPLE   |  NULL |   NULL  |    NULL    |



So I see a single definition ans several references. Whether the definition may occur in a table is more a question for Georg. :)

It worked up to now and is quite useful, because you can jump directly into the correct row.


Escaping the "\*" was necessary at the time for vim syntax highlighting. I think it is no longer needed.  So everything has a reason :)

New changeset 589e8fe07934a8585d6c5c31d12ae5f766b8fec7 by Ned Deily in branch 'master':
bpo-40204: Pin Sphinx version to 2.3.1 in ``Doc/Makefile``. (GH-21141)
https://github.com/python/cpython/commit/589e8fe07934a8585d6c5c31d12ae5f766b8fec7

New changeset 16e79a47ca3b4efb2300f24aa084c20bb5e8ae0d by Miss Islington (bot) in branch '3.9':
bpo-40204: Pin Sphinx version to 2.3.1 in ``Doc/Makefile``. (GH-21141)
https://github.com/python/cpython/commit/16e79a47ca3b4efb2300f24aa084c20bb5e8ae0d

New changeset 7318f0ab234e1c60a19eeea4e5968fb5d4fd70af by Ned Deily in branch '3.8':
bpo-40204: Pin Sphinx version to 2.3.1 in ``Doc/Makefile``. (GH-21141) (GH-21147)
https://github.com/python/cpython/commit/7318f0ab234e1c60a19eeea4e5968fb5d4fd70af

This issue prevents to upgrade Sphinx to Sphinx 3 in Fedora Rawhide, at least it breaks the python3-docs package which is Python 3.9 documentation:
https://bugzilla.redhat.com/show_bug.cgi?id=1823898

Making the Python documentation compatible with Sphinx 3.0 is causing multiple blocker issues.


(*) The first major problem is that new C domain markup like :c:struct: and :expr: are not recognized by Sphinx 2.4.4, but Sphinx 3.0 requires them. For example, a ".. c:struct:" section is completely hidden: treated as a comment.

I don't think that we can afford to drop Sphinx < 3 support since no Linux distribution uses Sphinx 3 yet. I see two options:

* Stay at Sphinx forever
* Backport the ":c:struct:" and ":c:expr:" markups somehow in Sphinx 2 using our CPython Sphinx extensions
* Update Sphinx 3 to tolerate our "incorrect" documentation using :c:type: and :c:type:`PyObject\*` (invalid C syntax), maybe as an opt-in option
* Update Sphinx 2 to support Sphinx 3 new syntax: something like Python 2.7 which backported a few Python 3 features like b"bytes string" syntax


(*) The second major question is: should we backport these changes to Python 3.8 and 3.9 branches? I consider that "supporting Sphinx 3" is a fix in the Python build system, and usually we do backport such changes, even if they can be seen as "new features". For me, it falls into the same category than supporting a newer version of C compiler.

PR 19397 makes tons of small changes. If we don't backport it, it will be very likely very painful to backport documentation changes. Julien Palard (who works on the documentation) suggests to backport the change. Usually, documentation changes are applied to all branches, not only master.

Again, right now, if we backport these changes, we would drop Sphinx 2 support.

In short, the current status is that IMHO it is not acceptable to support Sphinx 3. We cannot afford to break support with Sphinx 2.

I reported the issue to Sphinx:
"C domain changes of Sphinx 3 prevent to write doc compatible with Sphinx 2 and Sphinx 3"
https://github.com/sphinx-doc/sphinx/issues/7899

Using https://github.com/sphinx-doc/sphinx/pull/7905 but without -W, I'm able to build the unmodified Python documentation with Sphinx 3!

I tried c_allow_pre_v3=1.

The PR was updated to add a second c_warn_on_allowed_pre_v3=0 option so we can keep -W.

please note that pinning usually is not a solution for Linux distributions.  Yes, the most wanted fix would be to fix sphinx 3.x not to break compatibility with 2.x.  Or limit 3.9 to 2.x features for now.

https://github.com/sphinx-doc/sphinx/pull/7905 has been merged and will be part of the next Sphinx 3.2 release.

3.9.0 rc1 fails to build the docs with Sphinx 3.2.0, even with setting

c_allow_pre_v3 = True
c_warn_on_allowed_pre_v3 = False

Warning, treated as error:
/packages/python/3.9/python3.9-3.9.0~rc1/Doc/c-api/buffer.rst:92:Error in declarator or parameters
Invalid C declaration: Expected identifier in nested name. [error at 5]
  void \*buf
  -----^
make[1]: *** [Makefile:52: build] Error 2


With 3.2.0, you still need to build without -W

New changeset 423e77d6de497931585d1883805a9e3fa4096b0b by Victor Stinner in branch 'master':
bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844)
https://github.com/python/cpython/commit/423e77d6de497931585d1883805a9e3fa4096b0b

New changeset 43577c01a2ab49122db696e9eaec6cb31d11cc81 by Victor Stinner in branch 'master':
bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst (GH-21858)
https://github.com/python/cpython/commit/43577c01a2ab49122db696e9eaec6cb31d11cc81

New changeset 46d10b1237c67ff8347f533eda6a5468d098f7eb by Victor Stinner in branch 'master':
bpo-40204: Fix duplicates in the documentation (GH-21857)
https://github.com/python/cpython/commit/46d10b1237c67ff8347f533eda6a5468d098f7eb

New changeset d3ded080482beae578faa704b13534a62d066f9f by Victor Stinner in branch 'master':
bpo-40204: Add :noindex: in the documentation (GH-21859)
https://github.com/python/cpython/commit/d3ded080482beae578faa704b13534a62d066f9f

New changeset 474652fe9346382dbf793f20b671eb74668bebde by Victor Stinner in branch 'master':
bpo-40204, doc: Fix syntax of C variables (GH-21846)
https://github.com/python/cpython/commit/474652fe9346382dbf793f20b671eb74668bebde

With PR 21865, there are only two remaining warnings:

Doc/library/string.rst:311: WARNING: duplicate token description of sf:format_spec, other instance in library/string
Doc/reference/introduction.rst:96: WARNING: duplicate token description of *:name, other instance in reference/expressions

See bpo-35293 for RemovedInSphinx40Warning (which are already prevent with Sphinx 2).

New changeset bb0b08540cc93e56f3f1bde1b39ce086d9e35fe1 by Victor Stinner in branch 'master':
bpo-40204: Fix reference to terms in the doc (GH-21865)
https://github.com/python/cpython/commit/bb0b08540cc93e56f3f1bde1b39ce086d9e35fe1

New changeset 1abeda80f760134b4233608e2c288790f955b95a by Victor Stinner in branch 'master':
bpo-40204: Fix duplicated productionlist names in the doc (GH-21900)
https://github.com/python/cpython/commit/1abeda80f760134b4233608e2c288790f955b95a

I'm now able to build the Python documentation with Sphinx 3.2.1 without modifying the Doc/Makefile, so using -W option (treat warnings as errors): there are no more Sphinx 3 warnings.

When Sphinx 3 will be more widely available (ex: in Linux distributions), we will be able to consider removing c_allow_pre_v3=True and c_warn_on_allowed_pre_v3=False in doc/conf.py. For that, we should update the documentation to use the Sphinx 3 syntax, see PR 19397 written by Jakob Lykke Andersen.

For now, I prefer to keep Sphinx 2 support and so keep Sphinx 2 syntax in the C domain, since it allows supporting Sphinx 2 and Sphinx 3.

New changeset 8f88190af529543c84d5dc78f19abbfd73335cf4 by Victor Stinner in branch '3.9':
[3.9] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901)
https://github.com/python/cpython/commit/8f88190af529543c84d5dc78f19abbfd73335cf4

New changeset 7d0fef56d8eaac6309a66cb8c6ba6fd96f8c8a94 by Victor Stinner in branch '3.8':
bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901) (GH-21928)
https://github.com/python/cpython/commit/7d0fef56d8eaac6309a66cb8c6ba6fd96f8c8a94

I merged my changes to add Sphinx 3.2 and newer support into 3.8, 3.9 and master branches.

I close the issue.

Sadly, it seems like it's not possible to keep Sphinx 2 support if we add Sphinx 3.0 and Sphinx 3.1 support. For now, only Sphinx 2 and Sphinx 3.2+ are supported: 3.0 and 3.1 are not supported.

New changeset c0d5c131d9c1ac743058fae99f913b455fc90ac7 by Victor Stinner in branch 'master':
bpo-40204: Update Sphinx to version 3.2.1 in Doc/Makefile (GH-22043)
https://github.com/python/cpython/commit/c0d5c131d9c1ac743058fae99f913b455fc90ac7