We keep muttering about coming up with a less conflict-prone approach to Misc/NEWS updates, without ever settling on a concrete solution.

For the last big discussion on this, see the subthread starting at https://mail.python.org/pipermail/python-committers/2013-May/002554.html

For what it is worth, I haven't had a non-trivial merge conflict on Misc/NEWS for quite some time now. It seems to me that they are much rarer than they were with svn. (By 'trivial' I mean that the conflicts I see is that there's a new entry on 3.4 that wasn't in 3.3, and the conflict is just the addition of the new entry in the merge, which is trivial to fix.)

Then, again, I pretty much only do Library fixes. Maybe 'Core and builtins' fixes get messier because of the proximity of the date and version-specific header? That should be easy enough to fix....

The one I ran into earlier today tried to merge the entirety of the
3.3.x RC NEWS into the 3.4 NEWS :P

Were you changing something in core/builtins? I'm wondering if just making some same-between-versions space between the version header/date and the beginning of that section would be enough to solve 99% of the problems.

No, way down in Tests.

Couple of strategies I thought for tacking this problem.

1. Documenting the resolution of NEWS specific merge conflict. Observing/recording when those arises and when it was there was no-conflict will be helpful. This could help us in organizing NEWS file in a manner that less conflicts occur.

2. Automatic generation of NEWS file from commit logs. We can tag commit logs like #NEWS-Lib-2.7, #NEWS-Core-3.4 and NEWS file can be auto generated from the log. This could be interesting, it could also be error prone. That's probably the reason why we (or any other OSS project) has not adopted this method already.

Automatic generation of NEWS file from commit logs.

Commit messages can't be modified after being published, this would be annoying to fix typos etc.

From what I record of the ML discussions, the proposed solutions actually sounded more annoying than the occasional merge annoyance.

I agree with Antoine, but then as I said I don't run into huge merge conflicts very often.

IMHO the status quo is fine.
Most of the time HG gets the merge right on its own. Some times kdiff3 pops up and asks me to solve the conflict [0]. Rarely something goes wrong so I just revert Misc/NEWS and re-add the entry manually.

It seems to me that they are much rarer than they were with svn.

I agree -- I think many of us still see merges as something "bad" that should be avoided, while this is generally no longer true with HG (it was with SVN).
A similar argument is "whitespace/PEP8 changes on 3.4 should be avoided because it makes merging changes from 3.3 more difficult". While it's true that it makes 3.4 diverge from 3.3, IME this doesn't really bother HG and merges usually goes smoothly (there are other arguments against that kind of changes, but I think this is not a very strong one).

[0]: this usually happens when there is an entry in 3.4 only and I'm merging from 3.3. kdiff3 asks me if it should use the entry I added in 3.3 or the one that is in 3.4, and all I have to do is telling it to keep them both (ctrl+3/ctrl+2).

I don't think I've *ever* had NEWS successfully merge when working on
a bug fix (even after the move to hg). It's one of the reasons I
prefer working on new features, or bugs where the fix is too intrusive
for a maintenance release :P

Attached is my rough attempt at a script to auto-generate Misc/NEWS from commit logs. Key points:

• Works from the hg log in XML format as dumped to a file
• Considers a commit newsworthy if it isn't a merge and either has an issue number or has an "interesting" path (e.g. not an HTML file) and details (i.e. the stuff after a \n\n)
• Uses paths in the commit to determine what category the commit belongs to

Decisions to make:

• How to exactly flag a commit should not be in NEWS (I say it doesn't contain "#\d+:" to signify an issue and lacks any details after a \n\n)
• How often to run (e.g. at release time, anyone, post-commit hook, etc.)

I like the script name :-)... Other than that, can you attach a NEWS file such as generated by this script?

Example file at Antoine's request. Just don't forget, very rough and not at all finished. =) We probably can't use something like this until Python 3.5 in order to make sure everyone uses commit messages that make sense, but you can get a sense of whether it has a chance of working.

I should also mention that it would be easy to have multiple outputs, e.g. a reST one that links to the actual commit as well as the issue in order to have a pretty version for the website, etc.

Decisions to make:

Another important question: how do we manage manual edits to Misc/NEWS? Is the script smart enough to recognize those edits and not override them?

I am opposed to generating NEWS from the commit messages. I thought Antoine was too.

I am opposed to generating NEWS from the commit messages. I thought Antoine was too.

I am *a priori* skeptical that auto-generating NEWS can give good
enough results, but I don't mind being proven wrong :-)

Antoine: Script isn't far enough along to care about manual edits.

David: That's fine, but I'm opposed to having to manually edit Misc/NEWS. =)

Just to be clear: the reason I am opposed is that the audience for Misc/NEWS is different from the audience for commit messages, and I would prefer that the text reflect that. Mine do :)

If the goal is avoiding merging conflicts, a Mercurial hook might solve this problem. This has been already suggested and discussed on python-dev a while ago.

Anything more complex than that would likely cause more problems that it solves.

I get the argument against, but from my experience the difference between the opening line of a commit and what goes into Misc/NEWS is negligible at best. All of the commit-specific stuff for me goes "below the fold" after \n\n.

And while the goal is partially to avoid merge conflicts, it's also to avoid writing the same thing essentially twice along with controlling the format of Misc/NEWS.

On 13.09.2013 22:23, R. David Murray wrote:

R. David Murray added the comment:

Just to be clear: the reason I am opposed is that the audience for Misc/NEWS is different from the audience for commit messages, and I would prefer that the text reflect that. Mine do :)

I'm with David and -1 on auto-generating documentation that
is meant for Python users. Here's my take on the intentions
behind the various forms of documentation we have:

• Commit messages only provide a very terse hint at what a
  particular patch set was meant for. The target audience
  is other core developers.

• News entries explain these patches (there may be more than one
  for a particular issue or project) to Python programmers who
  need to know what to watch out for when upgrading to a new
  version.

• The "What's New" document explains the reasoning and effects
  of major changes and gives a high level overview of what
  to expect in a new version.

• Finally, the Python documentation, with it's "Version added"
  markers, provides details to application programmers who need
  to know how the new techniques work and when they were added
  (in order to determine which APIs they can use if they want to
  support multiple Python versions).

it's also to avoid writing the same thing essentially twice

Before committing I always make an "hg di" to double-check, and then copy (part of) the Misc/NEWS entry (that usually happens to be conveniently placed at the end of the diff) in the commit message.

Automating the generation of Misc/NEWS would likely make whatever modification we might want to do to the content more difficult.

I would argue, MAL, that what you want Misc/NEWS to be should instead be merged into the end of What's New. If something changed which could potentially break a user it should be listed in the most visible upgrade doc we have. Otherwise pouring through Misc/NEWS is a trudge and you might as well read the commit logs.

On 13.09.2013 22:51, Brett Cannon wrote:

Brett Cannon added the comment:

I would argue, MAL, that what you want Misc/NEWS to be should instead be merged into the end of What's New. If something changed which could potentially break a user it should be listed in the most visible upgrade doc we have. Otherwise pouring through Misc/NEWS is a trudge and you might as well read the commit logs.

There are often lots of small changes between releases that don't
end up in the What's New, but which can still make changes necessary
in user applications. Also, "What' New" is often not updated for
patch level releases, so the NEWS file is the only source to
track patch level changes.

And again, no, going through commit messages does not give the
same kind of information as in the NEWS file :-)

As you can see in your auto-generated file, there are multiple
entries for the same ticket, some entries don't reference a ticket,
there's no structure in the file, it includes everything (including
changes that were reverted again before the release), etc.

The auto generation would only result in the same kind of NEWS
file if everyone were really stringent in the way they write commit
messages. With hg not supporting editing commit messages after
the fact (apart from perhaps the last one), this is bound to
cause extra overhead to e.g. fix typos, add more infos, remove/hide
unnecessary entries, etc. We're all humans after all ...

[MAL]

• Commit messages only provide a very terse hint at what a
  particular patch set was meant for. The target audience
  is other core developers.

• News entries explain these patches (there may be more than one
  for a particular issue or project) to Python programmers who
  need to know what to watch out for when upgrading to a new
  version.

I agree. But if you have to write two different versions, putting them
both in the commit message has two advantages:

• it's arguably easier not to have to touch another file
• the extended explanation can be useful for core developers too :)

In any case, I would very much like to see a unified style for commit messages a la Mercurial: a very terse, but self-sufficient first line, and more explanation below.

Not to speak of the non-arguable advantage of getting rid of merge conflicts in Misc/NEWS.

So if the script had a mode to "set off" the intended NEWS entry from both a header (the terse 1-line commit message) and an optional footer (explanation of technical details only of interest to committers), I would say +1.

I agree the "different audiences" problem can be addressed by using appropriate commit message formatting to say "this bit goes in NEWS" (perhaps with some metadata to say which section).

However, that doesn't solve the question of how we fix inevitable mistakes. One thought that occurs to me is a "fixNEWS" subdir where we can put text files named after commit hashes. If there is a file in fixNEWS, then the script would ignore the corresponding commit and use the file contents instead.

On 17.09.2013 11:09, Nick Coghlan wrote:

Nick Coghlan added the comment:

I agree the "different audiences" problem can be addressed by using appropriate commit message formatting to say "this bit goes in NEWS" (perhaps with some metadata to say which section).

However, that doesn't solve the question of how we fix inevitable mistakes. One thought that occurs to me is a "fixNEWS" subdir where we can put text files named after commit hashes. If there is a file in fixNEWS, then the script would ignore the corresponding commit and use the file contents instead.

I don't think commit messages are the right place for such text.
They cannot be corrected after the fact, which makes them less
than ideal for the intended purposes: to produce an edited news
compilation.

I'm still not convinced we actually have a problem that needs to
be solved. Please consider the time it takes to resolve a merge
conflict vs. the time it takes to correct mistakes in commit
message formatting, educating committers and tweaking the script
to handle the inevitable mistakes.

Aside: It may be better to add support for news entries to the
tracker. At eGenix we're using a custom field called "News" which
is then listed in a report to build the change log. These can be
edited, there's only one entry per ticket and the report can be
sorted by priority/component/etc, putting the more important
news item at the top.

An alternative would be to have separate files NEWS-3.2, NEWS-3.3, NEWS-3.4 etc. If a fix is added to 3.2 and will be merged to 3.3 and 3.4 then you add an entry to NEWS-3.2 and append some sort of tags to indicate merges:

- Issue bpo-1234: Fix something that was not working properly.
  #Merge-3.3 #Merge-3.4

Then NEWS can be auto-generated from NEWS-3.*.

One of the Mercurial devs convinced me to pursue what I had initially proposed in msg197645 and write a merge script (attached).

The script is still a proof of concept, it makes a few assumptions and doesn't handle all the cases, but I did a few tests and it seems to work for at least some cases. If people like it it can be improved.

In short it parses Misc/NEWS, see what news entries have been added in the changeset(s) that it's being merged, and adds them at the top of the corresponding section.

To enable it download the file, set it as executable (chmod +x newsmerge.py), and add the following to .hg/hgrc:

[merge-tools]
newsmerge.executable = /path/to/newsmerge.py
newsmerge.priority = 100
newsmerge.premerge = True
newsmerge.args = $base $local $other -o $output

[merge-patterns]
Misc/NEWS = newsmerge

This will kick in only if the regular merge results in a conflict (so if you don't see any of the debug output from newsmerge it means that mercurial managed to merge Misc/NEWS on its own without conflicts).

So Twisted is actually in the process of pulling out their tooling they use for the separate files technique and making it a stand alone project. Seems like it'd make sense to reuse/contribute to that?

It's at https://github.com/twisted/newsbuilder

I would like fixing NEWS to be the top infrastructure improvement project, as it is my biggest time waster, and certainly the most obnoxious. It definite makes working on Idle less pleasant.

Since an empty 3.4.2a1 section was added to 3.4 NEWS, after 3.4.1 was released, Idle news items always fail to merge *even when there is no real conflict*. I think the problem is the offset of hundred of lines (now a thousand) between the position in the 3.4 file and the position in the 3.5 file. The hg merge jams the Idle entry somewhere in the middle of the Library section, creating spurious and bizarre-looking merge 'conflicts', instead of looking far enough down in 3.5 NEWS to find the proper context for successful insertion.

I am not the only person this happens to, as the same should be true for all the other sections that come after the Library section. A couple of months ago, someone pushed a post-push patch to remove artifacts not properly removed before the original push. Today I just removed what looked like an artifact from a Demo news item.

Le 02/12/2014 00:11, Terry J. Reedy a écrit :

Terry J. Reedy added the comment:

I would like fixing NEWS to be the top infrastructure improvement
project, as it is my biggest time waster, and certainly the most
obnoxious. It definite makes working on Idle less pleasant.

You could pair with Pierre-Yves David:
https://mail.python.org/pipermail/python-dev/2014-December/137393.html

Good idea. I will send him a note.

As an interim step, should we add Ezio's "newsmerge.py" to Tools/scripts and instructions for enabling it to the devguide?

That seems straightforward enough, and doesn't require any global workflow changes.

Thanks for the reminder/suggestion. Re-reading Ezio's two patches (newsmerge.py and .hg/hgrc additions in msg217079), they appear to jointly automate what I now do by hand (revert to local and dump merge artifacts, replace with edited file with new entries pasted in, and mark as resolved) . I also checked that the current Windows installers make .py files executable, so hg should be able to run newsmerge on Windows. I have downloaded the file and made the additions to my 3.5 hgrc to test next time I have a news entry.

It should be mentioned in this issue that the core-workflow mailing list decided on having NEWS entries being attached to the related issue in the issue tracker. This was then presented to python-committers and no one objected to the idea.

Is that something that might happen 'soon' or only when the whole workflow is redone?

As soon as someone finds the time to make the change we can switch. While this is a necessary requirement to change the workflow it isn't gated by it either.

Okay. I got tired of the constant Misc/NEWS merge conflicts, so I wrote a tool to fix the problem. It's checked in to Bitbucket here:

https://bitbucket.org/larry/mergenews/

There's a readme, which you'll see rendered on that page.

I don't know what you mean by "attach Misc/NEWS entries to the issue tracker". I figure, the Misc/NEWS entry is used as the checkin comment, so Roundup Robot already adds it for you, so maybe that's good enough.

Can we maybe switch to this for 3.6? Should we consider switching all 3.x branches under active development to use this approach?

In case you're too lazy to go visit the link to my "mergenews" repository and read the readme...

mergenews has three tools:

• splitnews, which splits the existing Misc/NEWS file into hundreds of individual files,
• mergenews, which merges the hundreds of individual files back into a single coherent Misc/NEWS file, and
• pyci, a front-end for "hg ci" which automatically writes out the checkin comment as a new news entry into the right spot so mergenews will see it.

It's all basically working already. pyci limits you in what command-line options it supports, other than that I think it all does what you want. Can we use it please? ;-)

What I mean by "attach Misc/NEWS entries to the issue tracker" is there will literally be a field in the issue tracker to enter the entry for the change log and Misc/NEWS will simply be auto-generated from the issue tracker. You can't use the commit message for what to put in the change log because too many people are against that solution (trust me, I tried to convince them otherwise).

Larry: commit messages are for other developers, which news entries are ultimately for users. Some developers want (insist on) the freedom to make the two different, with different details. I am using that for Idle patches since NEWS entries become 'What New for Idle' in each release.

Brett, I presume that you mean a new NEWS box like this Comment box, maybe with 3 lines and a new Section: box like Components. Question: Is it impossible to put labels above a box rather than to the right?

We could allow that news section to be (initially) filled out by someone other than the committer. Latter could (optionally) copy and paste as (initial draft of) commit message. Having message visible on issue might result in typos being corrected sooner, and would make it much easier to edit news message later (But editing after close should be limited to core devs).

Larry, a commit hook would be better that a separate pyci since TortoiseHG [Commit] button would still work. With Brett's solution, a commit hook could email commit message to tracker as NEWS message (if empty). And yes, change should apply to all versions. Otherwise, NEWS commits to older versions would have to be null-merged.

Yes, a text box or something along those lines. I assume the selection label question is to specify a section of the change log for something to go into. That's actually not necessary as the proper section is implied by the Components label.

And we could allow people to initially fill it in if desired, but I think worrying about whether to open it up or not is premature optimization; I'm fine with starting with it for only people with Developer privileges on the issue tracker.

I believe auto-filling it from the commit message (for those who don't put in the effort to treat the two audiences differently :) was something we discussed, but it would be a separate enhancement after the news box is added to the tracker. Which I might get to next month or I might not...

I think Larry's "split news" and "merge news" ideas are still useful as a bridging mechanism to help us get from the status quo to a tracker based solution.

It would just mean that, at some point in the future, the commit hook would change to be a post-commit hook that pushed to the issue tracker, rather than a pre-commit hook writing to a file in the source repo, and "merge news" would start pulling from the tracker, rather than from the source repo.

[...] commit messages are for other developers, which news entries are
ultimately for users. Some developers want (insist on) the freedom to
make the two different, with different details.

That's easy enough to accommodate. I updated "MergeNEWS" with this feature. Well, a similar feature:

The file you edit at checkin time starts with "Issue #:". You fill that out and that becomes your Misc/NEWS item, and the top of your checkin comment.

There's also a commented-out line of dashes below there. Anything you put there is appended to *the checkin comment only*.

So, you get the user summary as per Misc/NEWS at the top, then you can elaborate in all the technical and flowery language you like. Also, each section is optional (though obviously you have to fill out *one* of them).

It'd be easy to make the two comments completely separate. I just figured, the Misc/NEWS entry is presumably a nice high-level summary of the solution, why not leave that there and save the developer from having to summarize it twice.

Because the two summaries should be *different* in most cases. The first line of the checkin should ideally be a single sentence of less than 80 characters, while the NEWS item will almost always be longer. The NEWS item should be a complete summary, while the first checkin line is a 'summary in context' (ie: it can assume the reader knows the files that were changed, since log -v will show you that). This obviously requires that the succeeding paragraphs expand on that minimal summary (in most cases) in a way that doesn't look like a NEWS entry.

Now, this is obviously my preferred style and others may have different styles and/or disagree, but I'd like that style to be supported.

Note also that a big part of putting the NEWS entries in the tracker is so that they can be *edited*. Populating them initially from the checkin message is OK, but I predict that the actual workflow will be to create the NEWS entry in the tracker as part of getting the patch to the 'commit ready' point, which means that having a tool that pre-populates the commit message from the tracker NEWS entry will probably be more useful to most people. (And yeah, I'm sure that means a lot of commits will just have the NEWS entry as the commit message, which will make me sad, but oh well :)

Just noting a potential practical benefit of the "NEWS entry first" approach that David suggests: I believe it will help encourage the creation of a succinct answer to "Why are we making this change?" as part of the patch review process.

That's then useful both during the patch review itself (since there's a shared understanding between reviewers and implementors of the goal to be achieved), as well as when the change is released (since there's hopefully a user centric explanation of *why* something changed, rather than merely *what* changed)

However, the key trap I'd like us to avoid falling into is letting the fact a particular approach falls short of our ideal approach deter the introduction of interim improvements. We're going to need the checked in filesystem database anyway to backfill historical NEWS entries after switching to a generated NEWS file, so it seems harmless to me to switch to it early and then incrementally add to it until a tracker based solution is available.

I don't agree that in all cases the Misc/NEWS entry and the checkin comment must be wholly different. While I concur that that's called for now and again, I believe this to be rare. Most checkins are small changes, and a single-line summary for both Misc/NEWS and the first line of the checkin comment will suffice.

However, clearly the tool should *support* making them wholly separate. But that's easy to support. pyci now supports a "-n" or "--newsonly" flag, which instructs it to only add the Misc/NEWS item and to not check in. This is presumably also nicer for folks used to shell-integrated hg tools (TortoiseHG etc).

I think it's important that this *not* be the default behavior, as it's going to be hard enough to get the core dev community to adopt this tool, and I would prefer it make the experience nicer rather than clunkier.

I never said they always had to be different, and "wholly different" is certainly not the case, nor do I think I implied that.

So if your tool supports my scenario that's all I'm asking for :)

I sometimes put a bit more info in the commit message, but it's true that generally it's a copy/paste job as far as I'm concerned.

I came across OpenStack's tool for this problem today: http://docs.openstack.org/developer/reno/design.html

I think it's significantly more complex than we need for CPython, but also still interesting as a point of reference.

It's already mentioned in PEP-512, but I'll also add a reference here to https://pypi.python.org/pypi/towncrier, Amber Brown's release note manager that allows Twisted style release notes management to be used with other projects.

Has this been superseded by core_workflow issues?

Yep, core-workflow work has superseded this issue.