classification
Title: Find a less conflict prone approach to Misc/NEWS
Type: enhancement Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, dstufft, ezio.melotti, georg.brandl, lemburg, ncoghlan, pitrou, r.david.murray, sbt, terry.reedy, tshepang, zach.ware
Priority: normal Keywords: patch

Created on 2013-09-08 02:04 by ncoghlan, last changed 2014-06-13 13:19 by dstufft.

Files
File name Uploaded Description Edit
newsworthy.py brett.cannon, 2013-09-13 18:45
very_early_output.txt brett.cannon, 2013-09-13 18:53 Output from tip:v3.3.0 on default
news.py.diff zach.ware, 2014-03-20 20:03 For experimenting without cloning review
newsmerge.py ezio.melotti, 2014-04-25 00:39 Merge script to tell HG how to merge Misc/NEWS.
Repositories containing patches
http://hg.python.org/sandbox/new_news#playground
Messages (67)
msg197211 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-08 02:04
We keep muttering about coming up with a less conflict-prone approach to Misc/NEWS updates, without ever settling on a concrete solution.

For the last big discussion on this, see the subthread starting at https://mail.python.org/pipermail/python-committers/2013-May/002554.html
msg197222 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-08 04:51
For what it is worth, I haven't had a non-trivial merge conflict on Misc/NEWS for quite some time now.  It seems to me that they are much rarer than they were with svn.  (By 'trivial' I mean that the conflicts I see is that there's a new entry on 3.4 that wasn't in 3.3, and the conflict is just the addition of the new entry in the merge, which is trivial to fix.)
msg197223 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-08 04:53
Then, again, I pretty much only do Library fixes.  Maybe 'Core and builtins' fixes get messier because of the proximity of the date and version-specific header?  That should be easy enough to fix....
msg197230 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-08 07:37
The one I ran into earlier today tried to merge the entirety of the
3.3.x RC NEWS into the 3.4 NEWS :P
msg197260 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-08 14:23
Were you changing something in core/builtins?  I'm wondering if just making some same-between-versions space between the version header/date and the beginning of that section would be enough to solve 99% of the problems.
msg197329 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-08 22:02
No, way down in Tests.
msg197421 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2013-09-10 05:15
Couple of strategies I thought for tacking this problem.

1) Documenting the resolution of NEWS specific merge conflict. Observing/recording when those arises and when it was there was no-conflict will be helpful.  This could help us in organizing NEWS file in a manner that less conflicts occur.

2) Automatic generation of NEWS file from commit logs. We can tag commit logs like #NEWS-Lib-2.7, #NEWS-Core-3.4 and NEWS file can be auto generated from the log. This could be interesting, it could also be error prone. That's probably the reason why we (or any other OSS project) has not adopted this method already.
msg197478 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-11 08:41
> Automatic generation of NEWS file from commit logs.

Commit messages can't be modified after being published, this would be annoying to fix typos etc.

From what I record of the ML discussions, the proposed solutions actually sounded more annoying than the occasional merge annoyance.
msg197485 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-11 10:51
I agree with Antoine, but then as I said I don't run into huge merge conflicts very often.
msg197487 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-11 11:07
IMHO the status quo is fine.
Most of the time HG gets the merge right on its own.  Some times kdiff3 pops up and asks me to solve the conflict [0].  Rarely something goes wrong so I just revert Misc/NEWS and re-add the entry manually.

> It seems to me that they are much rarer than they were with svn.

I agree -- I think many of us still see merges as something "bad" that should be avoided, while this is generally no longer true with HG (it was with SVN).
A similar argument is "whitespace/PEP8 changes on 3.4 should be avoided because it makes merging changes from 3.3 more difficult".  While it's true that it makes 3.4 diverge from 3.3, IME this doesn't really bother HG and merges usually goes smoothly (there are other arguments against that kind of changes, but I think this is not a very strong one).

[0]: this usually happens when there is an entry in 3.4 only and I'm merging from 3.3.  kdiff3 asks me if it should use the entry I added in 3.3 or the one that is in 3.4, and all I have to do is telling it to keep them both (ctrl+3/ctrl+2).
msg197490 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-11 12:08
I don't think I've *ever* had NEWS successfully merge when working on
a bug fix (even after the move to hg). It's one of the reasons I
prefer working on new features, or bugs where the fix is too intrusive
for a maintenance release :P
msg197625 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-09-13 18:45
Attached is my rough attempt at a script to auto-generate Misc/NEWS from commit logs. Key points:

- Works from the hg log in XML format as dumped to a file
- Considers a commit newsworthy if it isn't a merge and either has an issue number or has an "interesting" path (e.g. not an HTML file) and details (i.e. the stuff after a \n\n)
- Uses paths in the commit to determine what category the commit belongs to

Decisions to make:

- How to exactly flag a commit should not be in NEWS (I say it doesn't contain "#\d+:" to signify an issue and lacks any details after a \n\n)
- How often to run (e.g. at release time, anyone, post-commit hook, etc.)
msg197626 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-13 18:46
I like the script name :-)... Other than that, can you attach a NEWS file such as generated by this script?
msg197628 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-09-13 18:53
Example file at Antoine's request. Just don't forget, very rough and not at all finished. =) We probably can't use something like this until Python 3.5 in order to make sure everyone uses commit messages that make sense, but you can get a sense of whether it has a chance of working.

I should also mention that it would be easy to have multiple outputs, e.g. a reST one that links to the actual commit as well as the issue in order to have a pretty version for the website, etc.
msg197633 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-13 19:28
> Decisions to make:

Another important question: how do we manage manual edits to Misc/NEWS? Is the script smart enough to recognize those edits and not override them?
msg197639 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-13 20:12
I am opposed to generating NEWS from the commit messages.  I thought Antoine was too.
msg197640 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-13 20:17
> I am opposed to generating NEWS from the commit messages.  I thought Antoine was too.

I am *a priori* skeptical that auto-generating NEWS can give good
enough results, but I don't mind being proven wrong :-)
msg197641 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-09-13 20:17
Antoine: Script isn't far enough along to care about manual edits.

David: That's fine, but I'm opposed to having to manually edit Misc/NEWS. =)
msg197643 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-13 20:23
Just to be clear: the reason I am opposed is that the audience for Misc/NEWS is different from the audience for commit messages, and I would prefer that the text reflect that.  Mine do :)
msg197645 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-13 20:25
If the goal is avoiding merging conflicts, a Mercurial hook might solve this problem.  This has been already suggested and discussed on python-dev a while ago.

Anything more complex than that would likely cause more problems that it solves.
msg197650 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-09-13 20:43
I get the argument against, but from my experience the difference between the opening line of a commit and what goes into Misc/NEWS is negligible at best. All of the commit-specific stuff for me goes "below the fold" after \n\n.

And while the goal is partially to avoid merge conflicts, it's also to avoid writing the same thing essentially twice along with controlling the format of Misc/NEWS.
msg197652 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-09-13 20:44
On 13.09.2013 22:23, R. David Murray wrote:
> 
> R. David Murray added the comment:
> 
> Just to be clear: the reason I am opposed is that the audience for Misc/NEWS is different from the audience for commit messages, and I would prefer that the text reflect that.  Mine do :)

I'm with David and -1 on auto-generating documentation that
is meant for Python users. Here's my take on the intentions
behind the various forms of documentation we have:

 * Commit messages only provide a very terse hint at what a
   particular patch set was meant for. The target audience
   is other core developers.

 * News entries explain these patches (there may be more than one
   for a particular issue or project) to Python programmers who
   need to know what to watch out for when upgrading to a new
   version.

 * The "What's New" document explains the reasoning and effects
   of major changes and gives a high level overview of what
   to expect in a new version.

 * Finally, the Python documentation, with it's "Version added"
   markers, provides details to application programmers who need
   to know how the new techniques work and when they were added
   (in order to determine which APIs they can use if they want to
   support multiple Python versions).
msg197653 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-13 20:50
> it's also to avoid writing the same thing essentially twice

Before committing I always make an "hg di" to double-check, and then copy (part of) the Misc/NEWS entry (that usually happens to be conveniently placed at the end of the diff) in the commit message.

Automating the generation of Misc/NEWS would likely make whatever modification we might want to do to the content more difficult.
msg197654 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-09-13 20:51
I would argue, MAL, that what you want Misc/NEWS to be should instead be merged into the end of What's New. If something changed which could potentially break a user it should be listed in the most visible upgrade doc we have. Otherwise pouring through Misc/NEWS is a trudge and you might as well read the commit logs.
msg197667 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-09-13 21:35
On 13.09.2013 22:51, Brett Cannon wrote:
> 
> Brett Cannon added the comment:
> 
> I would argue, MAL, that what you want Misc/NEWS to be should instead be merged into the end of What's New. If something changed which could potentially break a user it should be listed in the most visible upgrade doc we have. Otherwise pouring through Misc/NEWS is a trudge and you might as well read the commit logs.

There are often lots of small changes between releases that don't
end up in the What's New, but which can still make changes necessary
in user applications. Also, "What' New" is often not updated for
patch level releases, so the NEWS file is the only source to
track patch level changes.

And again, no, going through commit messages does not give the
same kind of information as in the NEWS file :-)

As you can see in your auto-generated file, there are multiple
entries for the same ticket, some entries don't reference a ticket,
there's no structure in the file, it includes everything (including
changes that were reverted again before the release), etc.

The auto generation would only result in the same kind of NEWS
file if everyone were really stringent in the way they write commit
messages. With hg not supporting editing commit messages after
the fact (apart from perhaps the last one), this is bound to
cause extra overhead to e.g. fix typos, add more infos, remove/hide
unnecessary entries, etc. We're all humans after all ...
msg197966 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-09-17 07:55
[MAL]

> * Commit messages only provide a very terse hint at what a
>   particular patch set was meant for. The target audience
>   is other core developers.
>
> * News entries explain these patches (there may be more than one
>   for a particular issue or project) to Python programmers who
>   need to know what to watch out for when upgrading to a new
>   version.

I agree.  But if you have to write two different versions, putting them
both in the commit message has two advantages:

- it's arguably easier not to have to touch another file
- the extended explanation can be useful for core developers too :)

In any case, I would very much like to see a unified style for commit messages a la Mercurial: a very terse, but self-sufficient first line, and more explanation below.

Not to speak of the non-arguable advantage of getting rid of merge conflicts in Misc/NEWS.

So if the script had a mode to "set off" the intended NEWS entry from both a header (the terse 1-line commit message) and an optional footer (explanation of technical details only of interest to committers), I would say +1.
msg197974 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-17 09:09
I agree the "different audiences" problem can be addressed by using appropriate commit message formatting to say "this bit goes in NEWS" (perhaps with some metadata to say which section).

However, that doesn't solve the question of how we fix inevitable mistakes. One thought that occurs to me is a "fixNEWS" subdir where we can put text files named after commit hashes. If there is a file in fixNEWS, then the script would ignore the corresponding commit and use the file contents instead.
msg197976 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-09-17 09:19
On 17.09.2013 11:09, Nick Coghlan wrote:
> 
> Nick Coghlan added the comment:
> 
> I agree the "different audiences" problem can be addressed by using appropriate commit message formatting to say "this bit goes in NEWS" (perhaps with some metadata to say which section).
> 
> However, that doesn't solve the question of how we fix inevitable mistakes. One thought that occurs to me is a "fixNEWS" subdir where we can put text files named after commit hashes. If there is a file in fixNEWS, then the script would ignore the corresponding commit and use the file contents instead.

I don't think commit messages are the right place for such text.
They cannot be corrected after the fact, which makes them less
than ideal for the intended purposes: to produce an edited news
compilation.

I'm still not convinced we actually have a problem that needs to
be solved. Please consider the time it takes to resolve a merge
conflict vs. the time it takes to correct mistakes in commit
message formatting, educating committers and tweaking the script
to handle the inevitable mistakes.

Aside: It may be better to add support for news entries to the
tracker. At eGenix we're using a custom field called "News" which
is then listed in a report to build the change log. These can be
edited, there's only one entry per ticket and the report can be
sorted by priority/component/etc, putting the more important
news item at the top.
msg197978 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-17 09:50
An alternative would be to have separate files NEWS-3.2, NEWS-3.3, NEWS-3.4 etc.  If a fix is added to 3.2 and will be merged to 3.3 and 3.4 then you add an entry to NEWS-3.2 and append some sort of tags to indicate merges:

    - Issue #1234: Fix something that was not working properly.
      #Merge-3.3 #Merge-3.4

Then NEWS can be auto-generated from NEWS-3.*.
msg198297 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-22 19:30
+lots on ending News Merge Hell. After too many bad experiences, I now avoid pushing non-Idle patches that require a News entry. Even for those who do not mind merge conflicts, there is still the waste of time.

The premise of non-checkout code management is that code expands with the number developers, so merge conflicts should stay rare. Making everyone edit the same few areas of one file subverts this premise.

The switch to hg introduced push races. A merge conflict lengthens the time required to fix things after losing a race. This increases the chance of loosing the race a second time. This has happened to me.

These problems would be worse if there were more active developers, which we need.


Some possible solutions.

1. Get hg to ignore lower context, which is the source of conflicts. The upper context of
==============
Section Title
-------------

==============
is never a problem.

2. Generate news entry from commit message on an *opt-in* basis using a section tag line **Section, where Section is one of Code, Lib, Text, Doc, Idle. Only pay attention to the first letter. Only text before the tag line would go in News. Example commit message:

Issue #12345: fix the bug. This is also the news entry.
Patch by New Contributor
**Lib
Comment for developers only.

This would work for at well over half of existing commits. I consider the elimination of duplication a bonus. A developer who preferred to write an independent News entry still could.

3. I though of separate News.3.3, News.3.4. While this would prevent forward merge conflicts (very helpful), it would not solve within version pull-merge conflicts, as when loosing a push race.

4. I thought of putting adding enough blank lines between header and first entries so that if more blanks were added with entries, the lower context could be 3 blank lines, and hence no conflict. However, when hg does diffs and must choose, it sees upper blank lines as context and lower blank lines as new, whereas the opposite would be needed for this to work. But this choice enables the following idea.

5. Add a new Log file with each entry consisting of a tag line, the entry, and 3 blank lines. If
==========
**Lib
Issue #12345. Fix bug. This is News entry.
Patch by New Contributor



===========
were augmented to
==========
**Lib
Issue #12345. Fix bug. This is News entry.
Patch by New Contributor



**Code
Issue #12348. Add feature. This is another News entry.



=================
the diff would start with three blank lines of context.
===================



**Code
Issue #12348. Add feature. This is another News entry.
+
+
+
==============

6. Have a directory of personal log files, one for each developer, that get periodically merged into news and emptied (or deleted when a release is tagged). The advantage, and only advantage, over a joint log file is not having to enforce the spacing of a joint log file.
msg198303 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-22 21:26
> Even for those who do not mind merge conflicts,
> there is still the waste of time.

If you know what you are doing the waste is minimal.  I sometimes import, check, commit, merge, and push a patch on 3 branches in less than a minute (it was a trivial patch that didn't require to run the test suite), and fixing conflicts in Misc/NEWS is something that can be done in a few seconds, even when the merge goes completely wrong (ctrl+2, ctrl+1, ctrl+s, ctrl+q when it goes right; ctrl+q, hg revert Misc/NEWS, copy/paste the entry when it goes wrong).

If you are trying to figure out how to use a merge tool or understanding why there is a merge conflict, what branches are involved, what should be merged and in what order, what should be committed/reverted/updated, then it will take more time (even more if you get it wrong and you have to start over), but the pattern is always the same once you learn it.

> The switch to hg introduced push races. A merge conflict lengthens
> the time required to fix things after losing a race. This increases
> the chance of loosing the race a second time. This has happened to me.

There are two kind of conflicts:
1) conflicts while merging files between branches on the same machine;
2) conflicts with changesets pushed by other developers;

The first kind is usually trivial to fix (see above).  The second kind (i.e. push race) might indeed be more complex [0], but it can still be solved in less than a minute.  Of course in that minute someone else might push again, but you must be very unlucky (or very slow).

> These problems would be worse if there were more active developers,
> which we need.

Even if we get twice as many active developers, I think push races would still be quite rare since there are usually hours between each push.  Once 2.7 is closed the chances of conflict will further decrease.


> Some possible solutions.

> 1. Get hg to ignore lower context, which is the source of conflicts. 

This could possibly be done with a merge hook (see one of my previous message).  I'm not sure it can be done by ignoring the lower context, but something should be doable.

> 2. Generate news entry from commit message on an *opt-in* basis using
> a section tag line **Section, where Section is one of Code, Lib,
> Text, Doc, Idle.

I'm -1 on adding tags and use them to generate Misc/NEWS.

> 3. I though of separate News.3.3, News.3.4. While this would prevent
> forward merge conflicts (very helpful), it would not solve within
> version pull-merge conflicts, as when loosing a push race.

Also if I fix a bug in 3.3 and 3.4, will I have to add the news manually in two different files?  Now HG does it for me automatically since there's only one file -- having to edit separate files seems like a step backward.
I don't know if creating News.3.4 by using "hg cp" on News.3.3 will automatically add the entry on both, but even if it does it won't probably solve the original problem.

> 4. I thought of putting adding enough blank lines between header and
> first entries so that if more blanks were added with entries, the 
> lower context could be 3 blank lines, and hence no conflict.

This sounds like an hack, and I'm not sure it will work (hg might try to include more context in the diff if there are only empty lines).

> 5. Add a new Log file with each entry consisting of a tag line, the
> entry, and 3 blank lines.

See previous point.

> 6. Have a directory of personal log files, one for each developer,
> that get periodically merged into news and emptied (or deleted when a
> release is tagged). The advantage, and only advantage, over a joint
> log file is not having to enforce the spacing of a joint log file.

IIRC the Twisted devs do something similar, but with a separate file for each NEWS entry.  Something similar was also discussed when this came up on python-dev a few months ago.

[0]: http://docs.python.org/devguide/faq.html#i-got-abort-push-creates-new-remote-heads-while-pushing-what-do-i-do
msg203686 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-21 22:33
Here's my half-baked idea on the topic, based largely on what I could learn of Twisted's method from looking through their source repo online.  I liked their use of separate files for each NEWS entry, especially since it should make merge conflicts exceedingly rare.

Basically, the plan is to add a couple of scripts to Tools/scripts named news.py and news_release.py.

Spec outline for news.py:
- four possible usages:
    - interactive
        - ask for each necessary datum
            - issue number(s), section(s), message

    - single command
        - command line switches for section and issue number, remaining
          arguments constitute message

    - blend
        - ask about anything not given in the single command, unless told
          not to with -f/--force (and then use defaults)

    - invoke editor
        - mirror hg's handling of `hg commit` without -m or -l, open an editor
          and use the saved file

- output a text file named "issue<number>.<count>.news" or
  "<current hg rev>+<something random>.news" (default in case of no issue
  number) containing the message into the appropriate folder(s)
  (e.g. Misc/NEWS.parts/Build)

- print the contents of the written file to stdout
    - allows `news.py --section build --issue 12345 "Fix building" | hg com -l -`

- print the output file's name to stderr
    - to make it easy to use `hg com -l Misc/NEWS.parts/<filename>.news`
      and not interfere with the above

- `hg add` the output file

This would necessitate a bit of new structure in the repo, namely a Misc/NEWS.parts/ dir, containing directories for each section (Core & Builtins, Library, etc.), which should also have a 'header.news' file to make each dir non-empty and for use with news_release.py.  This second script would be used at release time, and would simply walk the Misc/NEWS.parts dirs, building Misc/NEWS from the individual files and deleting them as it goes.

Beyond this basic functionality, there are other possible extensions, such as news.py gathering any information about the current change that is readily available and saving it in either comments at the tail end of the generated .news file, or in a separate .data file with the same base name, which news_release.py could then use for alternate NEWS formats (such as suitable output for Doc/whatsnew/changelog.rst with links to anything relevant).

To make the usage more convenient, both scripts could be added to Makefile, a la `make patchcheck`.  For Windows committers, there can either be a convenience batch file just for news.py, or I still hope to eventually get configure.bat/make.bat into a committable state.

If anyone thinks this half-baked idea looks like it might be edible when it's done, I'd be happy to cobble together news.py and news_release.py in a sandbox repo.
msg203692 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-21 22:59
Sounds plausible to me - I'd be interested in seeing an experimental
version :)
msg203828 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-22 18:17
There's a bare-bones experimental version in hg.python.org/sandbox/new_news that you can play around with in the 'playground' branch.  It's not pretty, but it (mostly) works :).  So far, there's just news.py, no news_release.py; but news_release.py should be pretty simple and only really matter to RMs anyway.
msg214269 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-03-20 20:01
The earlier in a release cycle we switch NEWS methods, the better.  Does anyone have any thoughts to share about this, either on Brett's newsworthy.py, my news.py/news_release.py, or another approach entirely?

I've updated the playground branch of sandbox/new_news to mirror current default (3.5.0a0), and have added .news files in the NEWS.parts tree to match the current state of Misc/NEWS.  I've also linked the repository to this issue.

You can play with adding NEWS entries by running Tools/scripts/news.py and see how it looks put together by running Tools/scripts/news_release.py (or write Misc/NEWS and remove all parts with "Tools/scripts/news_release.py write").

Other interesting things to try:
- Create a .news file with news.py, then use `hg commit -l Misc/NEWS.parts/<section>/<filename>.news`
- Do the above in a single step: `Tools/scripts/news.py | hg commit -l -`
msg214276 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-03-20 20:35
So the `hg commit -l` bit is going to run afoul of the same group of people who didn't like my commit message reuse idea unless you explicitly try to make it very clear that e.g. the first line is for Misc/NEWS and the latter is for the commit. Otherwise we have typically not include the "thanks" bit of a commit in Misc/NEWS which would also water down the `-l` usage. I know it's not critical for this, but I just wouldn't worry about it as a big selling point (unless you get really fancy and let the entries vary and instead of using the file as the input to the commit you write to a temp file or pipe in stdout and use the script to generate both the file and execute the commit with the more thorough message as a separate thing).

BUT if people like the "one entry, one file" approach and seriously using the output for both Misc/NEWS and the commit message then I see the script for generating the entry asking the following questions (which could even use Tkinter =):

* Issue #s
* Did someone else help out with this; allows making sure they were not accidentally left out of Misc/ACKS and add them if necessary or at least that they are mentioned in the commit message
* One line explanation for NEWS
* Optional extra bits to go into the commit message
* Should this go into What's New (e.g. new feature, backwards-compatible breakage); can add a `WN` or `WhatsNew` to the file name or something so that if someone like David tries to update the What's New doc they can tell by the file name that it may be important to cover (and maybe even only add the marker if What's New is NOT edited to know it's more like a TODO item)

Another side-effect of marking entries as worthy of being in What's New is that it should be easy to tell what should potentially be added as an addendum to What's New and to highlight at the end of the doc so that every micro release people can notice what was added. Heck, the logical conclusion is to split up What's New into individual files with a placeholder of the issue that was marked as worthy of inclusion and then edit it before release and auto-generate What's New (but one thing at a time =).

Anyway, ignoring all of this unstructured brain dump of mine, I'm can support doing a separate file approach.
msg214283 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-03-20 21:04
The `hg commit -l ...` tricks are just for those who tend to use exactly the same text for commit message and NEWS, and I only point them out simply because I don't know how many people are even aware of the -l option :).  If you want to use a NEWS entry as a base for a commit message without the two being exactly the same, it's also fairly easy to do `news.py | hg commit -l - && hg commit --amend` which will open the commit message in your usual commit message editor (at the cost of rolling back and re-doing the commit).

I also just thought of adding a 'who wrote it' question, and I like the idea of using it to manage Misc/ACKS.  I'm also intrigued by the suggestion of using Tkinter (when available), and the WhatsNew management.  But as you said, one thing at a time :)

First thing is, decide what we're actually going to do.  news.py obviously needs a fair bit of work before we can use it regularly, which I don't mind doing, but only if it will actually be used ;)
msg214285 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-20 21:31
I want no script asking me questions.  Post-facto errors for omissions are fine (and if I have to positively say no in the input file, that's fine).  tkinter is right out.
 
If you *also* want to make a script that asks questions (or even a tkinter ap), that's fine, but it should not be the main interface, it should be a wrapper.

One thing I really don't like about the separate file approach is that you lose the obvious chronology.  It's probably not a blocker, but it is definitely a disadvantage.
msg214354 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-03-21 12:50
You lose chronology in the directory, but not necessarily in the output; if you sort based on first commit time then you retain the chronological ordering in the merge.
msg214372 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-03-21 14:52
R. David Murray wrote:
> I want no script asking me questions.  Post-facto errors for omissions are
> fine (and if I have to positively say no in the input file, that's fine).
> tkinter is right out.

I have been planning a command line interface, something along the lines of
`news.py --issue 12345 --section Core --message "Converted ham to spam"`.  Of
course, there would be short options as well.  The interactive interface was
just what I wrote first because it seemed easiest at the time :).  Adding the
command line part shouldn't be too hard with argparse, though; I'll see if I
can do it this afternoon.
 
> If you *also* want to make a script that asks questions (or even a tkinter
> ap), that's fine, but it should not be the main interface, it should be a
> wrapper.

I'm fine with that.

> One thing I really don't like about the separate file approach is that you
> lose the obvious chronology.  It's probably not a blocker, but it is
> definitely a disadvantage.

If we give the .news files names of the form "nnn-issuexxxxx.news" where 'nnn'
is the number of .news files that already exist in the target folder (or the
whole tree) and "-issuexxxxx" is only present if an issue number was given,
chronology would be kept, though it might get a bit hairy merging between
branches.  It would be unlikely to have actual merge conflicts (unless your
news entry wasn't linked to an issue), but might cause things to get a bit out
of order.

Assuming a satisfactory interface to give news.py the proper information and a
good resolution to the chronology issue, how do you feel about the rest of the approach?
msg214589 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-23 15:24
Well, I guess you could sum it up as -0.  I personally don't feel the need for a change, but if the chronology problem is solved and it isn't *harder* to make the needed NEWS changes, then I'm not going to object.
msg214653 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-23 23:32
Like David I don't really care, but it seems Zach's script may show promise. I think it would be nice if "news.py --commit":
1) popped up an editor window (like hg commit does)
2) processed the editor results and looked for hints of an issue number
3) automatically created that file, hg add'ed it and then hg commit'ed the whole thing with the right commit message

I don't like Brett's approach.
msg214654 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-23 23:36
Note that the editor may be pre-filled with a skeleton:

# --- Fill in the NEWS message here
# (lines prefixed with '#' will be removed)
Issue #xxx: ....

# --- Additional text for the commit message below
# (won't be reflected in NEWS)

# --- Commit information
# user: Antoine Pitrou <email>
# branch: 3.4
# modified files:
# <diffstat>
msg214655 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-23 23:49
There also needs to be a way to specify a different NEWS entry from the commit message first line.
msg216906 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-04-20 15:45
Have you considered how this is going to change if/when PEP 462 will be implemented?  AFAIU PEP 462 suggests that commits happen directly from the bug tracker (or something equivalent), so we will likely start to add the NEWS entry there as well.  Assuming this will happen and it's not too far away in the future, it might be wiser to wait or move toward a system that will make the switch to PEP 462 easier.

FWIW I still think that the current situation is fine as is.
msg216917 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-20 20:43
PEP 462 is months away from going anywhere, and I personally find the
current NEWS handling a major barrier to feeling inclined to work on bug
fixes.

The status quo will *definitely* be PEP 462 incompatible, though, since it
would typically prevent applying the same patch to multiple branches.
msg216922 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-20 23:32
Yes, after thinking about this this weekend, it is clear to me that in the future we will *need* a scheme where by the NEWS entry can be safely included in the patch in some form.

I think the commit message is also going to be in the patch, which will be in 'hg export' form, if I'm understanding the stuff the Mecurial folks were telling us at PyCon correctly.

So, in theory the script approach would still work, if anybody can run it (to commit locally and then export) and if what it produces is something that won't get stale.
msg216923 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-20 23:41
A file per news entry seems a bit much, but an optional file per developer would solve most of the problem for those who have a problem with the status quo. Add a directory named, for instance, news.3.4.1. Put in a template with the allowed section headings. Call it something like aaTemplate.txt to sort first. To avoid merge conflicts due to entries for new features in default, quadruple space between headers so the only non-blank context would be the section header above. Developers like me who have a problem with the existing system could copy the file, rename it with their name (terry.reedy.txt, etc), hg add it, and use it for news entries (with extra blank lines to maintain the clean context.

A script could be written to sync the working directory (pull and merge), move entries into NEWS (skipping over blank lines), recopy the template, commit and push.

If I were working on non-Idle code issues, I would seriously consider doing the above with a private, non-repository file in /MISC that I might merge one a week or so.

Someone objected to changes that result in news entries being out patch push order. This is already not a rule because the devguide mentions inserting new items at random positions to avoid conflicts due to another commit. Also, News items are frequently pushed sometime after the corresponding patch. I don't know if this is because people forget or because they want to isolate any hassle with a news conflict. In any case, the News entries are not necessarily time ordered now.

From a user perspective, having library news items sorted by affected module, with code, doc, and test changes collected together, would generally be more useful.
msg216925 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-20 23:59
> This is already not a rule because the devguide mentions inserting new 
> items at random positions to avoid conflicts due to another commit.

Really?

"""New NEWS entries are customarily added at or near the top of their respective sections, so that entries within a section appear in approximate order from newest to oldest. However, this is customary and not a requirement."""

> In any case, the News entries are not necessarily time ordered now.

IME, they mostly are. It's true it's not a requirement.
msg216931 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-21 00:34
On 4/20/2014 7:59 PM, Antoine Pitrou wrote:
>
> Antoine Pitrou added the comment:
>
>> This is already not a rule because the devguide mentions inserting new
>> items at random positions to avoid conflicts due to another commit.
>
> Really?
>
> """New NEWS entries are customarily added at or near the top of their respective sections, so that entries within a section appear in approximate order from newest to oldest. However, this is customary and not a requirement."""

Random is too loose for what was actually committed and I will correct 
it. Further down in the same section: "A nice trick to make Mercurial’s 
automatic file merge work more smoothly is to put a new entry after the 
first or first two entries rather than at the very top. This way if you 
commit, pull new changesets and merge, the merge will succeed 
automatically."

>> In any case, the News entries are not necessarily time ordered now.
>
> IME, they mostly are. It's true it's not a requirement.
msg216932 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-21 00:49
For the recommendation actually put in the devguide, change 'random position' to 'position near but not at the top'.
https://docs.python.org/devguide/committing.html#news-entries ends with
"A nice trick to make Mercurial’s automatic file merge work more smoothly is to put a new entry after the first or first two entries rather than at the very top. This way if you commit, pull new changesets and merge, the merge will succeed automatically."

My main point was that the devguide a) recognizes that there is an issue and b) does not require strict date order. Merging items into the NEWS on a daily basis would keep approximate time order. Hourly merges would do even better.

This recommendation, however, is not very effective. It only only avoids conflict with another patch if the other person does not use the 'nice trick' but puts the news entry at the top. Unless one looks carefully, it does not help the problen of merging maintenance bugfix items into a default list that also contains enhancement news not in the maintenance list*. In fact, blindly putting an item between two news items rather than a header and one news item makes that conflict more likely.

* There have been times, like last Jan-Feb, when default only items dominated the default news list.
msg216943 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-21 13:27
In order for things to work with a patch gating system, I believe it will be the most practical for the news items to be each in a separate file.
msg216945 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-04-21 13:57
> Unless one looks carefully, it does not help the problen of merging 
> maintenance bugfix items into a default list that also contains 
> enhancement news not in the maintenance list*.

What if instead of having sections in Misc/NEWS for core/library/documentation/etc., we simply make sections for bug fixes and enhancements?  Wouldn't this basically solve the conflict problem (assuming all/most bug fixes are backported, and new features are only on default)?
msg216955 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-21 17:53
That would probably work for now, but it wouldn't work for the patch gating system.  On the other hand, it would sure make it easier to build/check whatsnew.
msg216958 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-04-21 18:37
Also having a list of enhancements and bug fixes might be more meaningful for users than a list of core issues vs library issues vs other similar sections.

This could also be done with two separate files, with the "new features/enhancements" file existing only on default and the "bug fixes" file existing across branches.  From there it shouldn't be difficult for the gating system to append/prepend/insert the news.
msg216959 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-21 18:51
That makes the tooling of the gating system harder, though.  If the NEWS can just be a file in the patch, we don't have to have any special tooling for it in the gating system.
msg216961 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-21 19:19
If we put news items in a database keyed by issue number (and I think it reasonable that all news-worthy patches should have a tracker issue), then there would be no conflicts. (We already avoid simultaneous commits to the same issue.) If the database had fields such as the type of issue/patch and the component affected, then different queries could create text files sorted differently.

We already have such a database with such fields and indexed by issue number -- the tracker itself. We could either copy data from the tracker into a separate database or add to the tracker a textbox News field that is only editable by either by the 'Assigned to' person or perhaps any committer.

I personally would prefer a box that I could fill out when closing the issue. (The tracker could even ask for verification when closing a 'fixed' issue with a blank news.) News summaries could be extracted at any time by scanning commit messages since the last release for issue numbers. A news box on the tracker would give people looking at the issue thereafter a quick summary of the result of the issue without scanning through all the messages and checking the patches. 

Since the issue formatting is being reviewed, I think the issue summary should also include the exact releases patched. This would help people to see if an issue is only open for a possible backport, It might also help a program selecting issues for a news report.

I am, of course, aware that I have glossed over many details.
msg216962 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-21 19:24
Moving News items from the repository to the tracker, where I think they initially belong anyway, would also remove them as an issue for a future gating system.
msg216963 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-04-21 19:33
Terry J. Reedy wrote:
> Moving News items from the repository to the tracker, where I think
> they initially belong anyway, would also remove them as an issue for
> a future gating system.

I think News items are best left in the repository just so that for any given snapshot of the repository, you can see what significant changes are present.
msg216964 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-04-21 19:41
Yeah, Guido was strongly in favor of that too.
msg216965 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-04-21 19:45
Agreed.  Once the gating system is in place nothing prevent us to read the NEWS entry either from the patch that is being committed or from a field in the tracker and then including it together with the patch once it is approved.
msg216969 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-21 20:41
A sequential log of commit messages for a particular branch would give one an even better snapshot view of activity since not all commits have news messages and even when they do, commit messages sometimes have additional info. But see below.

The essence of my proposal is that the ORIGINAL entry of a news item by a human be into the tracker where there is no conflict and where it can be seen by people looking at the issue. Eliminating that conflict is the subject of this issue. After that, items can be mechanically copied wherever desired, including, possibly, into a file in the repository. My apology if this part was not clear enough.

Such mechanical copying and  re-arranging should not have the conflicts that are the subject of this issue. In particular, if the gating system is the only entity that edits particular news files (and I propose that there could be more than one), it will not have conflicts with other editors. 

Since these would be secondary, derived files, I don't see why any or all should be in the hg repository, as opposed to being part of the doc package available online along with other derived files. In other words, the hg repository is for original master files used to create derivative files nearly always not kept in the repository. We do not put .html files derived from master .rst files into the repository. So I do not see why files derived from hg repository commit messages and the tracker sql repository should go into the hg repository either. But I don't especially care if they are.

I doubt Guido favors a system that inhibits commits by making them unnecessariy painful and troublesome.
msg216977 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-22 00:51
Folks, this is *really, really, simple*: one file per NEWS entry. How
we arrange them is just a detail. Don't go off trying to invent wild
exotic alternatives that spread state across multiple sources of truth
- significant historical info belongs in the version control system,
and NEWS entries are how we highlight "this one is significant"
(relative to other commits). A hg extension that prepopulates the
commit message from a NEWS entry included in the patch wouldn't be
difficult (especially since some of the Mercurial devs are likely to
join the new core workflow list).
msg216983 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-22 04:24
My concern with a file for each entry is a possible slowdown of some operations, like TortoiseHg resyncing the diff between repository and working directory (it is not instantaneous even now). However, if there are multiple directories and if they are emptied periodically, so that there are never more than a few hundred in any directory, it might not be too bad. I agree that this idea is otherwise an improvement.
msg216997 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-04-22 10:52
FWIW: I don't think we need to manage the news entries in the NEWS
file. Instead, we could simply add a field to the bug tracker
called "news entry" and populate that as necessary.

During release, this information can then be used to create a
NEWS file per release. This file would not be touched by anyone
other then the release manager.

As a result, there would not longer be any merge conflicts, and
the news entries can still be changed before the release
if necessary, and, of course, the generated NEWS file can also
be edited as the release manager sees fit.

Tickets which do not need a news entry simply leave the news entry
field blank.

We really don't need a file system database for these things.

(PS: eGenix has been using such a system internally for several
years and it works great.)
msg217079 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-04-23 17:24
One of the Mercurial devs convinced me to pursue what I had initially proposed in msg197645 and write a merge script (attached).

The script is still a proof of concept, it makes a few assumptions and doesn't handle all the cases, but I did a few tests and it seems to work for at least some cases.  If people like it it can be improved.

In short it parses Misc/NEWS, see what news entries have been added in the changeset(s) that it's being merged, and adds them at the top of the corresponding section.

To enable it download the file, set it as executable (chmod +x newsmerge.py), and add the following to .hg/hgrc:

[merge-tools]
newsmerge.executable = /path/to/newsmerge.py
newsmerge.priority = 100
newsmerge.premerge = True
newsmerge.args = $base $local $other -o $output

[merge-patterns]
Misc/NEWS = newsmerge

This will kick in only if the regular merge results in a conflict (so if you don't see any of the debug output from newsmerge it means that mercurial managed to merge Misc/NEWS on its own without conflicts).
msg220438 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-06-13 13:19
So Twisted is actually in the process of pulling out their tooling they use for the separate files technique and making it a stand alone project. Seems like it'd make sense to reuse/contribute to that?

It's at https://github.com/twisted/newsbuilder
History
Date User Action Args
2014-06-13 13:19:40dstufftsetnosy: + dstufft
messages: + msg220438
2014-04-25 00:39:43ezio.melottisetfiles: - newsmerge.py
2014-04-25 00:39:32ezio.melottisetfiles: + newsmerge.py
2014-04-23 17:24:18ezio.melottisetfiles: + newsmerge.py

messages: + msg217079
2014-04-22 10:52:43lemburgsetmessages: + msg216997
2014-04-22 04:24:31terry.reedysetmessages: + msg216983
2014-04-22 00:51:05ncoghlansetmessages: + msg216977
2014-04-21 20:41:48terry.reedysetmessages: + msg216969
2014-04-21 19:45:14ezio.melottisetmessages: + msg216965
2014-04-21 19:41:20r.david.murraysetmessages: + msg216964
2014-04-21 19:33:03zach.waresetmessages: + msg216963
2014-04-21 19:24:46terry.reedysetmessages: + msg216962
2014-04-21 19:19:43terry.reedysetmessages: + msg216961
2014-04-21 18:51:09r.david.murraysetmessages: + msg216959
2014-04-21 18:37:15ezio.melottisetmessages: + msg216958
2014-04-21 17:53:35r.david.murraysetmessages: + msg216955
2014-04-21 13:57:12ezio.melottisetmessages: + msg216945
2014-04-21 13:27:06r.david.murraysetmessages: + msg216943
2014-04-21 00:49:15terry.reedysetmessages: + msg216932
2014-04-21 00:34:30terry.reedysetmessages: + msg216931
2014-04-20 23:59:41pitrousetmessages: + msg216925
2014-04-20 23:41:13terry.reedysetmessages: + msg216923
2014-04-20 23:32:04r.david.murraysetmessages: + msg216922
2014-04-20 20:43:42ncoghlansetmessages: + msg216917
2014-04-20 15:45:01ezio.melottisetmessages: + msg216906
2014-04-19 15:47:27orsenthilsetnosy: - orsenthil
2014-03-23 23:49:21r.david.murraysetmessages: + msg214655
2014-03-23 23:36:41pitrousetmessages: + msg214654
2014-03-23 23:32:22pitrousetmessages: + msg214653
2014-03-23 15:24:32r.david.murraysetmessages: + msg214589
2014-03-21 14:52:10zach.waresetmessages: + msg214372
2014-03-21 12:50:12brett.cannonsetmessages: + msg214354
2014-03-20 21:31:13r.david.murraysetmessages: + msg214285
2014-03-20 21:04:47zach.waresetmessages: + msg214283
2014-03-20 20:35:40brett.cannonsetmessages: + msg214276
2014-03-20 20:04:08zach.waresetfiles: + news.py.diff
keywords: + patch
2014-03-20 20:01:41zach.waresethgrepos: + hgrepo228
messages: + msg214269
2013-11-22 18:17:19zach.waresetmessages: + msg203828
2013-11-21 22:59:34ncoghlansetmessages: + msg203692
2013-11-21 22:33:01zach.waresetmessages: + msg203686
2013-11-12 15:24:58zach.waresetnosy: + zach.ware
2013-09-22 21:26:05ezio.melottisetmessages: + msg198303
2013-09-22 19:30:08terry.reedysetnosy: + terry.reedy
messages: + msg198297
2013-09-17 09:50:57sbtsetnosy: + sbt
messages: + msg197978
2013-09-17 09:19:45lemburgsetmessages: + msg197976
2013-09-17 09:09:11ncoghlansetmessages: + msg197974
2013-09-17 07:55:55georg.brandlsetnosy: + georg.brandl
messages: + msg197966
2013-09-13 21:35:24lemburgsetmessages: + msg197667
2013-09-13 20:51:09brett.cannonsetmessages: + msg197654
2013-09-13 20:50:44ezio.melottisetmessages: + msg197653
2013-09-13 20:44:51lemburgsetnosy: + lemburg
messages: + msg197652
2013-09-13 20:43:21brett.cannonsetmessages: + msg197650
2013-09-13 20:25:20ezio.melottisetmessages: + msg197645
2013-09-13 20:23:22r.david.murraysetmessages: + msg197643
2013-09-13 20:17:57brett.cannonsetmessages: + msg197641
2013-09-13 20:17:28pitrousetmessages: + msg197640
2013-09-13 20:12:27r.david.murraysetmessages: + msg197639
2013-09-13 19:28:09pitrousetmessages: + msg197633
2013-09-13 19:26:26tshepangsetnosy: + tshepang
2013-09-13 18:54:21brett.cannonsetfiles: + very_early_output.txt

messages: + msg197628
2013-09-13 18:46:23pitrousetmessages: + msg197626
2013-09-13 18:45:22brett.cannonsetfiles: + newsworthy.py
nosy: + brett.cannon
messages: + msg197625

2013-09-11 12:08:21ncoghlansetmessages: + msg197490
2013-09-11 11:07:46ezio.melottisetnosy: + ezio.melotti
messages: + msg197487
2013-09-11 10:51:31r.david.murraysetmessages: + msg197485
2013-09-11 08:41:19pitrousetnosy: + pitrou
messages: + msg197478
2013-09-10 05:15:46orsenthilsetnosy: + orsenthil
messages: + msg197421
2013-09-08 22:02:29ncoghlansetmessages: + msg197329
2013-09-08 14:23:11r.david.murraysetmessages: + msg197260
2013-09-08 07:37:00ncoghlansetmessages: + msg197230
2013-09-08 04:53:50r.david.murraysetmessages: + msg197223
2013-09-08 04:51:41r.david.murraysetnosy: + r.david.murray
messages: + msg197222
2013-09-08 02:04:41ncoghlancreate