[Ned asked to submit that as a release blocker]

the build of the documentation fails with at least the 3.5.4rc1.  It adds a new build dependency (blurb), which is inconvenient to build on stable environments, or when pip is not available. Please could you consider including the blurb module itself in python for the stable branches?

Also the build fails with the release tarballs.

We should be able to fix the Makefile recipe to handle either case, i.e. when Misc/NEWS.d is not present (as in the release tarballs), don't try to run blurb.

blurb should be only used when there is a release of python, and in this case, only by the release manager.

> blurb should be only used when there is a release of python, and in this case, only by the release manager.

No, that's not the case.  Blurb needs to be used whenever the docs are built because the NEWS file is used in the Docs build to produce the changelog.html file.

It will be easier to decide if Misc/NEWS was generated or not when... Misc/NEWS will be removed from Git:
https://github.com/python/cpython/pull/2719

No?

ok, I understand.

but blurb is an external tool (github.com/python/core-workflow).
There will be a new dependency, excepted if we store it into the Tools/ directory.

and if we add it into Tools, we have to add it to the old branches where blurb is supposed to be used, and what can we do with the only 'security fix' and 'bug fix' branches ?

Thread on python-committers:
https://mail.python.org/pipermail/python-committers/2017-July/004763.html

Unfortunately, updates to the Doc build process doc have not yet been made but the intended process is to provide blurb via the "make venv" Doc build recipe which now also pip-installs blurb from PyPI.  It is the necessary to use that venv to build the docs:

make venv
make html PYTHON=venv/bin/python3

Or you can just install blurb somewhere into your normal shell PATH.  Again, this will be documented.

> There will be a new dependency, excepted if we store it into the Tools/ directory.

Please don't do that. I will be a mess to update our copy. It's already painful enough to embed our copy of libexpat, zlib and indirectly of OpenSSL (on Windows and macOS).

> I will be a mess to update our copy.

typo, you should read: "It will be..." ;-)

(It will be a mess to update our copy of blurb.)

@haypo I don't want to add a new dependency, but it's an option.

I prefer the suggestion of Ned.

How do you imagine the patch ?

Please let Larry have a chance to look at this before we spend a lot of energy here rehashing discussions.  We'll fix it, it's not hard!

* I was told that blurb needed to be in the core-workflow repo, that this was better than checking it in to CPython itself.  Okay.  If you want to debate the point, loop in Brett Cannon, he's the one with the strong opinions.

* Ned has taken over the 3.3 branch, and he's asked that we not bother to blurbify it (convert it to use blurb).  My expectation is that we will blurbify all other active branches: 2.7 3.4 3.5 3.6 master.

* As part of the release process this weekend, I blurbified the 3.4 and 3.5 branches.  There were already NEWS items stored in the Misc/NEWS.d/next directory, so I had to do *something*, and Brett has been encouraging me to move a little faster, so... I figured it was best to start there, as people don't care so much about 3.4 and 3.5 anymore, and they're "my" branches (as RM) so I wouldn't be inconveniencing anyone else.  I assume we'll get all the problems ironed out before the next 3.6 release.

* When a branch is converted over to blurb, you'll have to use blurb to build the docs from a git checkout of that branch.  

* When a branch is converted over to blurb, Misc/NEWS will no longer be a checked-in file.  It will only be generated, by the release manager when making a release, or for fun whenever somebody feels like generating it I guess.  It's important to remove it, because it's an output file, and if we leave it checked in people will modify it by hand, and then those changes will be lost the next time it's generated.  In the 3.4 and 3.5 branches Misc/NEWS has been deleted.

* As Ned points out, blurb isn't necessary for tarballs released by the RM.  My makefile-fu isn't up to the task, but essentially Doc/Makefile should run this logic (assuming cwd is Doc):

    if os.path.exists("../Misc/NEWS"):
        shutil.copy("../Misc/NEWS", "build/NEWS")
    else:
        os.system("blurb merge build/NEWS")

* I honestly don't understand the disposition of the venv in Doc/Makefile.  Is it used automatically?  Isn't it?  It seems to be used under 3.5--when I do a docs build it installs a bunch of packages--so why didn't blurb get installed for doko?  What about 3.4, which predates the venv in Doc/Makefile?  TBH I'm a big dummy when it comes to venv stuff.  I'm hoping someone smart like Ned will figure it all out and make it sane, regular, and predictable.  (Hint hint!)

* Building the docs already requires a dozen packages (Sphinx, which in turn depends on markupsafe et al).  I assumed requiring one more package, also installed from pip, was not the worst thing in the world.  Is it?

Working on it.

> so why didn't blurb get installed for doko?

it's not packaged, and I'm (and I think any other provider of distro packages) not using network access during the build.  Sorry to say, but the assumption that pip should be used to be able to build core python seems to be the wrong direction. Packaging blurb sure is an option, but what kind of value has this as a distro package? only used for building cpython, so why not include it in cpython itself?

I have two use cases:

 - building a distro package from a (pre)release tarball with a
   included NEWS file.  I can handle that by just not calling
   blurb.

 - building a distro package from a release tarball + a diff
   for a branch. I haven't looked yet how I could do that,
   including a current NEWS file.

For the included external C libraries we do have the option to either use the included version, or an external one.  Why not do the same for blurb?

>  Sorry to say, but the assumption that pip should be used to be able to build core python seems to be the wrong direction.

I'm not sure I understand your position.  External tools are only permissible if Debian packages exist for them?

And, it's not "core python", it's the Python documentation, which has always depended on external tools.  The dependencies for CPython itself (the "python" executable) have not changed.

The good news is that the released tarballs starting now and going forward will already have a generated Misc/NEWS just like they always have had.  So blurb is not needed if you are building docs from such tarballs (like 3.5.4rc1).  We did forget about building docs directly from the tarball so we'll fix the Doc Makefile to use an existing Misc/NEWS file and not use blurb at all in that case. I think that should cover your first use case.

(If you are building from a git repo, you will need to use blurb as Misc/NEWS is no longer a checked-in file.  As I've tried to explain before, just running "make venv" and "make html" currently isn't sufficient; one needs to either activate the virtual env or fiddle with Doc Makefile variables PYTHON, SPHINXBUILD, and/or now BLURB.  I'm trying to see if I can make that smarter.)

Your second use case is more complicated, if I understand it correctly.  If you are generating the diff from a repo, going forward there won't be a unified Misc/NEWS file in the repo, rather there will be individual files added to Misc/NEWS.d for each checkin with a NEWS entry and those individual files have metadata meaningful to blurb.  When the release tarball is made, Misc/NEWS.d/* is removed and replaced by a blurb-generated Misc/NEWS file.  So if you are adding upstream pre-release diffs, you would have to run blurb as part of the process of building the docs. But I don't think blurb currently is designed to to handle the case of combining an existing generated Misc/NEWS file (from the tarball) with additional NEWS.d files. So I guess one solution would be you need to build the docs from the Git repo or at least copy the Misc/NEWS.d from the repo at the release tag and add the NEWS.d files for the cherrypicked pre-release fixes to it and then run blurb on it.  But I may have misunderstand and I should let Larry speak for blurb.

(Of course the above also applies to installed copies of Misc/NEWS, like Debian's /usr/share/doc/pythonx.x/NEWS.gz.  The vanilla main Python Makefile doesn't install NEWS.)

I guess it depends on what the upstream patches look like.  If they're always coded against release tarballs and not against git repos, then they'll assume (correctly) that Misc/NEWS exists.  In that case you shouldn't have to do anything.

If the upstream patches are extracted from GitHub PRs, then they'd presumably have files in Misc/NEWS.d.  You could just ignore them.

blurb doesn't support combining Misc/NEWS.d files with an existing Misc/NEWS file in a single operation.  But you could run "blurb split", breaking Misc/NEWS into constituent files in Misc/NEWS.d, and then "blurb merge", rebuilding Misc/NEWS from those constituent files in Misc/NEWS.d.  Naturally this would pick up the new files from Misc/NEWS.d too.

Currently blurb is designed for the CPython developer workflow, so operations like "blurb split" and "blurb merge" also automatically stage the files in git.  I can add a flag to suppress this feature, if it's going to become common that "blurb split" and "blurb merge" are run on tarballs.

I just pushed PR 2874 (for 3.5) for review.  I think it will solve the tarball problem.  Review comments welcome.  I'm still thinking about how to make blurb more easily "discoverable".  That would be a separate PR.

If we provide a prebuilt Misc/NEWS in tarball, if Makefile doesn't need blurb if Misc/NEWS already exists, doko's issue is fixed: Debian package builder will not need blurb anymore (PR 2874).


> If the upstream patches are extracted from GitHub PRs, then they'd presumably have files in Misc/NEWS.d.  You could just ignore them.

If a vendor don't want to rely on blurb, I think that it's acceptable tradeoff.

Usually, each backported change is documented in a vendor specific changelog anyway.

Maybe tomorrow, some Linux vendors will start using blurb, but I suggest to work step by step, and first focus on the Python 3.5 "regression" fixed by PR 2874.

--

> But you could run "blurb split", breaking Misc/NEWS into constituent files in Misc/NEWS.d, and then "blurb merge", rebuilding Misc/NEWS from those constituent files in Misc/NEWS.d.

I'm not following you. If you want to rebuild Misc/NEWS, why not just removing Misc/NEWS and then run blurb merge?

I expect that in a tarball, Misc/NEWS and Misc/NEWS.d contain exactly the same entries.

> I expect that in a tarball, Misc/NEWS and Misc/NEWS.d contain exactly the same entries.

They don't.  See for yourself :)

https://www.python.org/ftp/python/3.5.4/Python-3.5.4rc1.tgz

What you're missing is that Misc/NEWS.d is removed from tarballs.  We ship Misc/NEWS in tarballs but *not* Misc/NEWS.d.

If you start with a CPython tarball (e.g. 3.5.4rc1), and then apply vendor-specific patches on top of that, and those patches include news entries in Misc/NEWS.d, and you want to ship a unified Misc/NEWS file that contains those entries, then you'll have to use "blurb split" to repopulate Misc/NEWS.d from Misc/NEWS followed by "blurb merge" to reconstitute Misc/NEWS from Misc/NEWS.d.

FWIW, right now any entries under "Misc/NEWS.d/next" would appear in Misc/NEWS under the heading "What's new in Python next?", rather than e.g. "What's new in Python 3.5.4 release candidate 1?".  This would be a little curious.  blurb is only set up to understand "next" and CPython version numbers there; convincing it to put something else there would be a non-trivial amount of work.

On a related subject, I should mention something about blurb aggregated version files, e.g. "Misc/NEWS.d/3.5.4rc1.rst".  blurb doesn't know how to append / prepend to those, nor does it know how to combine two of them together.  It just isn't a feature I needed for CPython.  My guess is adding support for this wouldn't be too bad, and it may yet be helpful for doko's use case.  Something to keep in mind.

> I'm not sure I understand your position.  External tools are only
> permissible if Debian packages exist for them?

for a package build to use them, yes.

> And, it's not "core python", it's the Python documentation, which has
> always depended on external tools.  The dependencies for CPython itself
> (the "python" executable) have not changed.

afaict, the tools needed to build the documentation didn't change within subminor version updates.

We changed *revision control systems* "within subminor version updates".  If your argument is "you mustn't add a new tool used to build the documentation until 3.7", you're not going to win that argument, you should try another tactic.

And, in the past, I'm not aware that the availability of tools in the Debian ecosystem has been a strong determining factor in the choice of tooling for CPython.  In particular, I'm not aware of any tool in the Debian package ecosystem that would solve the problem blurb solves, much less in exactly the way blurb does it.  In fact, there was no existing tool anywhere that solved the *exact* problem we had.  So we wrote our own.  And obviously a newly written tool isn't going to have a Debian package.  So your argument "you should only choose external dependencies that already have Debian packages" isn't going to work either, as that is not and has never been a rule in CPython development.

I'm pretty sure blurb is going to remain a part of the CPython toolchain.  The slight inconvenience you have registered here is so far not enough of a negative vote to outweigh the groundswell of positive support.  I suggest you redirect your energies to solving the problem rather than complaining about it.

New changeset 3de144890ad3bc50694368a1b33be6d7f3a780b3 by larryhastings (Ned Deily) in branch '3.5':
[3.5] bpo-31036: use an existing Misc/NEWS rather than trying to use blurb (#2874)
https://github.com/python/cpython/commit/3de144890ad3bc50694368a1b33be6d7f3a780b3

New changeset 362e9fb0de4321bf265dbca290f7dc1f383a4a47 by Ned Deily in branch '3.4':
[3.5] bpo-31036: use an existing Misc/NEWS rather than trying to use blurb (#2874) (#2926)
https://github.com/python/cpython/commit/362e9fb0de4321bf265dbca290f7dc1f383a4a47

I added blurb to the docs build script (https://github.com/python/docsbuild-scripts/commit/06375b6d74f6d5613fcd48030b3af8032854c87d), so 3.5 should build soon. The requirements looks automatically pulled from psf-salt https://github.com/python/psf-salt/blob/master/salt/docs/init.sls#L42 but I don't know at which "frequency".

Still unfixed on docs.iad1.psf.io, the venv I modified is only used for sphinx and uses Python 2. If we can easily upgrade it to Python 3 we'll be able to pass it as the PYTHON parameter for make in docsbuild-script, allowing blurb to be found.

see: https://github.com/python/psf-salt/issues/109

We upgraded Python on docs.python.org to 3.6 (from 3.4), so blurb can run. I pushed https://github.com/python/docsbuild-scripts/pull/22 to give the right Python in the PYTHON Makefile parameter. I'm currently running a test build locally and will merge soon.

This is marked as a release blocker for 3.4 and 3.5; however, Ned's Doc/Makefile change has been merged in both those versions.  And no other branches have been "blurbified" yet.  While we may improve the solution in the future, the fundamental problem (you can't build the docs without blurb from a tarball) is fixed.  So I'm marking this closed.  If I'm mistaken for some reason please reopen.

New changeset abe7b60b80f5cb23f84a49f95ef6ce23521eed14 by Larry Hastings (Ned Deily) in branch '3.5':
[3.5] bpo-31036: use an existing Misc/NEWS rather than trying to use blurb (#2874)
https://github.com/python/cpython/commit/abe7b60b80f5cb23f84a49f95ef6ce23521eed14

New changeset 590665c399fc4aa3c4a9f8e7104d43a02e9f3a0c by Ned Deily in branch 'master':
bpo-31036: Allow sphinx and blurb to be found automatically (#3440)
https://github.com/python/cpython/commit/590665c399fc4aa3c4a9f8e7104d43a02e9f3a0c

New changeset 645c1e421348790d49eab863279a41cb2b6a007a by Ned Deily (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31036: Allow sphinx and blurb to be found automatically (GH-3440) (#3441)
https://github.com/python/cpython/commit/645c1e421348790d49eab863279a41cb2b6a007a

For the 3.6 and master branches, I've now enhanced Doc/Makefile to better find its tools.  If the doc venv is present, e.g. make venv was run, the Doc build recipe will use sphinx-build and blurb from there, otherwise the process PATH will be search for installed versions.  Hopefully, that covers nearly all use cases.  But, if necessary, explicit locations for the tools can be set with the SPHINXBUILD and BLURB variables.  3.4 and 3.5 are now in security-fix mode: this change can be backported at their release manager's discretion.  The 2.7 Doc/Makefile does not need to use blurb because its docset does not build a changelog html file like the 3.x docs do.

New changeset 9cc332094c5f8cbaa47400633ab3ba372da61c9d by Ned Deily in branch '3.5':
[3.5] bpo-31036: Allow sphinx and blurb to be found automatically (GH-3440)
https://github.com/python/cpython/commit/9cc332094c5f8cbaa47400633ab3ba372da61c9d

P.S. I just noticed that I forgot to credit Brett here as the source (from superseded PR 2953) of a good portion of the last round of PRs.  Thanks!

New changeset 0bd59d655280ad4c4aff2d327789815b89caee18 by Benjamin Peterson in branch '2.7':
[2.7] bpo-31036: Allow sphinx and blurb to be found automatically (GH-16638)
https://github.com/python/cpython/commit/0bd59d655280ad4c4aff2d327789815b89caee18