We keep muttering about coming up with a less conflict-prone approach to Misc/NEWS updates, without ever settling on a concrete solution.

For the last big discussion on this, see the subthread starting at https://mail.python.org/pipermail/python-committers/2013-May/002554.html

For what it is worth, I haven't had a non-trivial merge conflict on Misc/NEWS for quite some time now.  It seems to me that they are much rarer than they were with svn.  (By 'trivial' I mean that the conflicts I see is that there's a new entry on 3.4 that wasn't in 3.3, and the conflict is just the addition of the new entry in the merge, which is trivial to fix.)

Then, again, I pretty much only do Library fixes.  Maybe 'Core and builtins' fixes get messier because of the proximity of the date and version-specific header?  That should be easy enough to fix....

The one I ran into earlier today tried to merge the entirety of the
3.3.x RC NEWS into the 3.4 NEWS :P

Were you changing something in core/builtins?  I'm wondering if just making some same-between-versions space between the version header/date and the beginning of that section would be enough to solve 99% of the problems.

No, way down in Tests.

Couple of strategies I thought for tacking this problem.

1) Documenting the resolution of NEWS specific merge conflict. Observing/recording when those arises and when it was there was no-conflict will be helpful.  This could help us in organizing NEWS file in a manner that less conflicts occur.

2) Automatic generation of NEWS file from commit logs. We can tag commit logs like #NEWS-Lib-2.7, #NEWS-Core-3.4 and NEWS file can be auto generated from the log. This could be interesting, it could also be error prone. That's probably the reason why we (or any other OSS project) has not adopted this method already.

> Automatic generation of NEWS file from commit logs.

Commit messages can't be modified after being published, this would be annoying to fix typos etc.

From what I record of the ML discussions, the proposed solutions actually sounded more annoying than the occasional merge annoyance.

I agree with Antoine, but then as I said I don't run into huge merge conflicts very often.

IMHO the status quo is fine.
Most of the time HG gets the merge right on its own.  Some times kdiff3 pops up and asks me to solve the conflict [0].  Rarely something goes wrong so I just revert Misc/NEWS and re-add the entry manually.

> It seems to me that they are much rarer than they were with svn.

I agree -- I think many of us still see merges as something "bad" that should be avoided, while this is generally no longer true with HG (it was with SVN).
A similar argument is "whitespace/PEP8 changes on 3.4 should be avoided because it makes merging changes from 3.3 more difficult".  While it's true that it makes 3.4 diverge from 3.3, IME this doesn't really bother HG and merges usually goes smoothly (there are other arguments against that kind of changes, but I think this is not a very strong one).

[0]: this usually happens when there is an entry in 3.4 only and I'm merging from 3.3.  kdiff3 asks me if it should use the entry I added in 3.3 or the one that is in 3.4, and all I have to do is telling it to keep them both (ctrl+3/ctrl+2).

I don't think I've *ever* had NEWS successfully merge when working on
a bug fix (even after the move to hg). It's one of the reasons I
prefer working on new features, or bugs where the fix is too intrusive
for a maintenance release :P

Attached is my rough attempt at a script to auto-generate Misc/NEWS from commit logs. Key points:

- Works from the hg log in XML format as dumped to a file
- Considers a commit newsworthy if it isn't a merge and either has an issue number or has an "interesting" path (e.g. not an HTML file) and details (i.e. the stuff after a \n\n)
- Uses paths in the commit to determine what category the commit belongs to

Decisions to make:

- How to exactly flag a commit should not be in NEWS (I say it doesn't contain "#\d+:" to signify an issue and lacks any details after a \n\n)
- How often to run (e.g. at release time, anyone, post-commit hook, etc.)

I like the script name :-)... Other than that, can you attach a NEWS file such as generated by this script?

Example file at Antoine's request. Just don't forget, very rough and not at all finished. =) We probably can't use something like this until Python 3.5 in order to make sure everyone uses commit messages that make sense, but you can get a sense of whether it has a chance of working.

I should also mention that it would be easy to have multiple outputs, e.g. a reST one that links to the actual commit as well as the issue in order to have a pretty version for the website, etc.

> Decisions to make:

Another important question: how do we manage manual edits to Misc/NEWS? Is the script smart enough to recognize those edits and not override them?

I am opposed to generating NEWS from the commit messages.  I thought Antoine was too.

> I am opposed to generating NEWS from the commit messages.  I thought Antoine was too.

I am *a priori* skeptical that auto-generating NEWS can give good
enough results, but I don't mind being proven wrong :-)

Antoine: Script isn't far enough along to care about manual edits.

David: That's fine, but I'm opposed to having to manually edit Misc/NEWS. =)

Just to be clear: the reason I am opposed is that the audience for Misc/NEWS is different from the audience for commit messages, and I would prefer that the text reflect that.  Mine do :)

If the goal is avoiding merging conflicts, a Mercurial hook might solve this problem.  This has been already suggested and discussed on python-dev a while ago.

Anything more complex than that would likely cause more problems that it solves.

I get the argument against, but from my experience the difference between the opening line of a commit and what goes into Misc/NEWS is negligible at best. All of the commit-specific stuff for me goes "below the fold" after \n\n.

And while the goal is partially to avoid merge conflicts, it's also to avoid writing the same thing essentially twice along with controlling the format of Misc/NEWS.

On 13.09.2013 22:23, R. David Murray wrote:
> 
> R. David Murray added the comment:
> 
> Just to be clear: the reason I am opposed is that the audience for Misc/NEWS is different from the audience for commit messages, and I would prefer that the text reflect that.  Mine do :)

I'm with David and -1 on auto-generating documentation that
is meant for Python users. Here's my take on the intentions
behind the various forms of documentation we have:

 * Commit messages only provide a very terse hint at what a
   particular patch set was meant for. The target audience
   is other core developers.

 * News entries explain these patches (there may be more than one
   for a particular issue or project) to Python programmers who
   need to know what to watch out for when upgrading to a new
   version.

 * The "What's New" document explains the reasoning and effects
   of major changes and gives a high level overview of what
   to expect in a new version.

 * Finally, the Python documentation, with it's "Version added"
   markers, provides details to application programmers who need
   to know how the new techniques work and when they were added
   (in order to determine which APIs they can use if they want to
   support multiple Python versions).

> it's also to avoid writing the same thing essentially twice

Before committing I always make an "hg di" to double-check, and then copy (part of) the Misc/NEWS entry (that usually happens to be conveniently placed at the end of the diff) in the commit message.

Automating the generation of Misc/NEWS would likely make whatever modification we might want to do to the content more difficult.

I would argue, MAL, that what you want Misc/NEWS to be should instead be merged into the end of What's New. If something changed which could potentially break a user it should be listed in the most visible upgrade doc we have. Otherwise pouring through Misc/NEWS is a trudge and you might as well read the commit logs.

On 13.09.2013 22:51, Brett Cannon wrote:
> 
> Brett Cannon added the comment:
> 
> I would argue, MAL, that what you want Misc/NEWS to be should instead be merged into the end of What's New. If something changed which could potentially break a user it should be listed in the most visible upgrade doc we have. Otherwise pouring through Misc/NEWS is a trudge and you might as well read the commit logs.

There are often lots of small changes between releases that don't
end up in the What's New, but which can still make changes necessary
in user applications. Also, "What' New" is often not updated for
patch level releases, so the NEWS file is the only source to
track patch level changes.

And again, no, going through commit messages does not give the
same kind of information as in the NEWS file :-)

As you can see in your auto-generated file, there are multiple
entries for the same ticket, some entries don't reference a ticket,
there's no structure in the file, it includes everything (including
changes that were reverted again before the release), etc.

The auto generation would only result in the same kind of NEWS
file if everyone were really stringent in the way they write commit
messages. With hg not supporting editing commit messages after
the fact (apart from perhaps the last one), this is bound to
cause extra overhead to e.g. fix typos, add more infos, remove/hide
unnecessary entries, etc. We're all humans after all ...

[MAL]

> * Commit messages only provide a very terse hint at what a
>   particular patch set was meant for. The target audience
>   is other core developers.
>
> * News entries explain these patches (there may be more than one
>   for a particular issue or project) to Python programmers who
>   need to know what to watch out for when upgrading to a new
>   version.

I agree.  But if you have to write two different versions, putting them
both in the commit message has two advantages:

- it's arguably easier not to have to touch another file
- the extended explanation can be useful for core developers too :)

In any case, I would very much like to see a unified style for commit messages a la Mercurial: a very terse, but self-sufficient first line, and more explanation below.

Not to speak of the non-arguable advantage of getting rid of merge conflicts in Misc/NEWS.

So if the script had a mode to "set off" the intended NEWS entry from both a header (the terse 1-line commit message) and an optional footer (explanation of technical details only of interest to committers), I would say +1.

I agree the "different audiences" problem can be addressed by using appropriate commit message formatting to say "this bit goes in NEWS" (perhaps with some metadata to say which section).

However, that doesn't solve the question of how we fix inevitable mistakes. One thought that occurs to me is a "fixNEWS" subdir where we can put text files named after commit hashes. If there is a file in fixNEWS, then the script would ignore the corresponding commit and use the file contents instead.

On 17.09.2013 11:09, Nick Coghlan wrote:
> 
> Nick Coghlan added the comment:
> 
> I agree the "different audiences" problem can be addressed by using appropriate commit message formatting to say "this bit goes in NEWS" (perhaps with some metadata to say which section).
> 
> However, that doesn't solve the question of how we fix inevitable mistakes. One thought that occurs to me is a "fixNEWS" subdir where we can put text files named after commit hashes. If there is a file in fixNEWS, then the script would ignore the corresponding commit and use the file contents instead.

I don't think commit messages are the right place for such text.
They cannot be corrected after the fact, which makes them less
than ideal for the intended purposes: to produce an edited news
compilation.

I'm still not convinced we actually have a problem that needs to
be solved. Please consider the time it takes to resolve a merge
conflict vs. the time it takes to correct mistakes in commit
message formatting, educating committers and tweaking the script
to handle the inevitable mistakes.

Aside: It may be better to add support for news entries to the
tracker. At eGenix we're using a custom field called "News" which
is then listed in a report to build the change log. These can be
edited, there's only one entry per ticket and the report can be
sorted by priority/component/etc, putting the more important
news item at the top.

An alternative would be to have separate files NEWS-3.2, NEWS-3.3, NEWS-3.4 etc.  If a fix is added to 3.2 and will be merged to 3.3 and 3.4 then you add an entry to NEWS-3.2 and append some sort of tags to indicate merges:

    - Issue #1234: Fix something that was not working properly.
      #Merge-3.3 #Merge-3.4

Then NEWS can be auto-generated from NEWS-3.*.

+lots on ending News Merge Hell. After too many bad experiences, I now avoid pushing non-Idle patches that require a News entry. Even for those who do not mind merge conflicts, there is still the waste of time.

The premise of non-checkout code management is that code expands with the number developers, so merge conflicts should stay rare. Making everyone edit the same few areas of one file subverts this premise.

The switch to hg introduced push races. A merge conflict lengthens the time required to fix things after losing a race. This increases the chance of loosing the race a second time. This has happened to me.

These problems would be worse if there were more active developers, which we need.


Some possible solutions.

1. Get hg to ignore lower context, which is the source of conflicts. The upper context of
==============
Section Title
-------------

==============
is never a problem.

2. Generate news entry from commit message on an *opt-in* basis using a section tag line **Section, where Section is one of Code, Lib, Text, Doc, Idle. Only pay attention to the first letter. Only text before the tag line would go in News. Example commit message:

Issue #12345: fix the bug. This is also the news entry.
Patch by New Contributor
**Lib
Comment for developers only.

This would work for at well over half of existing commits. I consider the elimination of duplication a bonus. A developer who preferred to write an independent News entry still could.

3. I though of separate News.3.3, News.3.4. While this would prevent forward merge conflicts (very helpful), it would not solve within version pull-merge conflicts, as when loosing a push race.

4. I thought of putting adding enough blank lines between header and first entries so that if more blanks were added with entries, the lower context could be 3 blank lines, and hence no conflict. However, when hg does diffs and must choose, it sees upper blank lines as context and lower blank lines as new, whereas the opposite would be needed for this to work. But this choice enables the following idea.

5. Add a new Log file with each entry consisting of a tag line, the entry, and 3 blank lines. If
==========
**Lib
Issue #12345. Fix bug. This is News entry.
Patch by New Contributor



===========
were augmented to
==========
**Lib
Issue #12345. Fix bug. This is News entry.
Patch by New Contributor



**Code
Issue #12348. Add feature. This is another News entry.



=================
the diff would start with three blank lines of context.
===================



**Code
Issue #12348. Add feature. This is another News entry.
+
+
+
==============

6. Have a directory of personal log files, one for each developer, that get periodically merged into news and emptied (or deleted when a release is tagged). The advantage, and only advantage, over a joint log file is not having to enforce the spacing of a joint log file.

> Even for those who do not mind merge conflicts,
> there is still the waste of time.

If you know what you are doing the waste is minimal.  I sometimes import, check, commit, merge, and push a patch on 3 branches in less than a minute (it was a trivial patch that didn't require to run the test suite), and fixing conflicts in Misc/NEWS is something that can be done in a few seconds, even when the merge goes completely wrong (ctrl+2, ctrl+1, ctrl+s, ctrl+q when it goes right; ctrl+q, hg revert Misc/NEWS, copy/paste the entry when it goes wrong).

If you are trying to figure out how to use a merge tool or understanding why there is a merge conflict, what branches are involved, what should be merged and in what order, what should be committed/reverted/updated, then it will take more time (even more if you get it wrong and you have to start over), but the pattern is always the same once you learn it.

> The switch to hg introduced push races. A merge conflict lengthens
> the time required to fix things after losing a race. This increases
> the chance of loosing the race a second time. This has happened to me.

There are two kind of conflicts:
1) conflicts while merging files between branches on the same machine;
2) conflicts with changesets pushed by other developers;

The first kind is usually trivial to fix (see above).  The second kind (i.e. push race) might indeed be more complex [0], but it can still be solved in less than a minute.  Of course in that minute someone else might push again, but you must be very unlucky (or very slow).

> These problems would be worse if there were more active developers,
> which we need.

Even if we get twice as many active developers, I think push races would still be quite rare since there are usually hours between each push.  Once 2.7 is closed the chances of conflict will further decrease.


> Some possible solutions.

> 1. Get hg to ignore lower context, which is the source of conflicts. 

This could possibly be done with a merge hook (see one of my previous message).  I'm not sure it can be done by ignoring the lower context, but something should be doable.

> 2. Generate news entry from commit message on an *opt-in* basis using
> a section tag line **Section, where Section is one of Code, Lib,
> Text, Doc, Idle.

I'm -1 on adding tags and use them to generate Misc/NEWS.

> 3. I though of separate News.3.3, News.3.4. While this would prevent
> forward merge conflicts (very helpful), it would not solve within
> version pull-merge conflicts, as when loosing a push race.

Also if I fix a bug in 3.3 and 3.4, will I have to add the news manually in two different files?  Now HG does it for me automatically since there's only one file -- having to edit separate files seems like a step backward.
I don't know if creating News.3.4 by using "hg cp" on News.3.3 will automatically add the entry on both, but even if it does it won't probably solve the original problem.

> 4. I thought of putting adding enough blank lines between header and
> first entries so that if more blanks were added with entries, the 
> lower context could be 3 blank lines, and hence no conflict.

This sounds like an hack, and I'm not sure it will work (hg might try to include more context in the diff if there are only empty lines).

> 5. Add a new Log file with each entry consisting of a tag line, the
> entry, and 3 blank lines.

See previous point.

> 6. Have a directory of personal log files, one for each developer,
> that get periodically merged into news and emptied (or deleted when a
> release is tagged). The advantage, and only advantage, over a joint
> log file is not having to enforce the spacing of a joint log file.

IIRC the Twisted devs do something similar, but with a separate file for each NEWS entry.  Something similar was also discussed when this came up on python-dev a few months ago.

[0]: http://docs.python.org/devguide/faq.html#i-got-abort-push-creates-new-remote-heads-while-pushing-what-do-i-do

Here's my half-baked idea on the topic, based largely on what I could learn of Twisted's method from looking through their source repo online.  I liked their use of separate files for each NEWS entry, especially since it should make merge conflicts exceedingly rare.

Basically, the plan is to add a couple of scripts to Tools/scripts named news.py and news_release.py.

Spec outline for news.py:
- four possible usages:
    - interactive
        - ask for each necessary datum
            - issue number(s), section(s), message

    - single command
        - command line switches for section and issue number, remaining
          arguments constitute message

    - blend
        - ask about anything not given in the single command, unless told
          not to with -f/--force (and then use defaults)

    - invoke editor
        - mirror hg's handling of `hg commit` without -m or -l, open an editor
          and use the saved file

- output a text file named "issue<number>.<count>.news" or
  "<current hg rev>+<something random>.news" (default in case of no issue
  number) containing the message into the appropriate folder(s)
  (e.g. Misc/NEWS.parts/Build)

- print the contents of the written file to stdout
    - allows `news.py --section build --issue 12345 "Fix building" | hg com -l -`

- print the output file's name to stderr
    - to make it easy to use `hg com -l Misc/NEWS.parts/<filename>.news`
      and not interfere with the above

- `hg add` the output file

This would necessitate a bit of new structure in the repo, namely a Misc/NEWS.parts/ dir, containing directories for each section (Core & Builtins, Library, etc.), which should also have a 'header.news' file to make each dir non-empty and for use with news_release.py.  This second script would be used at release time, and would simply walk the Misc/NEWS.parts dirs, building Misc/NEWS from the individual files and deleting them as it goes.

Beyond this basic functionality, there are other possible extensions, such as news.py gathering any information about the current change that is readily available and saving it in either comments at the tail end of the generated .news file, or in a separate .data file with the same base name, which news_release.py could then use for alternate NEWS formats (such as suitable output for Doc/whatsnew/changelog.rst with links to anything relevant).

To make the usage more convenient, both scripts could be added to Makefile, a la `make patchcheck`.  For Windows committers, there can either be a convenience batch file just for news.py, or I still hope to eventually get configure.bat/make.bat into a committable state.

If anyone thinks this half-baked idea looks like it might be edible when it's done, I'd be happy to cobble together news.py and news_release.py in a sandbox repo.

Sounds plausible to me - I'd be interested in seeing an experimental
version :)

There's a bare-bones experimental version in hg.python.org/sandbox/new_news that you can play around with in the 'playground' branch.  It's not pretty, but it (mostly) works :).  So far, there's just news.py, no news_release.py; but news_release.py should be pretty simple and only really matter to RMs anyway.

The earlier in a release cycle we switch NEWS methods, the better.  Does anyone have any thoughts to share about this, either on Brett's newsworthy.py, my news.py/news_release.py, or another approach entirely?

I've updated the playground branch of sandbox/new_news to mirror current default (3.5.0a0), and have added .news files in the NEWS.parts tree to match the current state of Misc/NEWS.  I've also linked the repository to this issue.

You can play with adding NEWS entries by running Tools/scripts/news.py and see how it looks put together by running Tools/scripts/news_release.py (or write Misc/NEWS and remove all parts with "Tools/scripts/news_release.py write").

Other interesting things to try:
- Create a .news file with news.py, then use `hg commit -l Misc/NEWS.parts/<section>/<filename>.news`
- Do the above in a single step: `Tools/scripts/news.py | hg commit -l -`

So the `hg commit -l` bit is going to run afoul of the same group of people who didn't like my commit message reuse idea unless you explicitly try to make it very clear that e.g. the first line is for Misc/NEWS and the latter is for the commit. Otherwise we have typically not include the "thanks" bit of a commit in Misc/NEWS which would also water down the `-l` usage. I know it's not critical for this, but I just wouldn't worry about it as a big selling point (unless you get really fancy and let the entries vary and instead of using the file as the input to the commit you write to a temp file or pipe in stdout and use the script to generate both the file and execute the commit with the more thorough message as a separate thing).

BUT if people like the "one entry, one file" approach and seriously using the output for both Misc/NEWS and the commit message then I see the script for generating the entry asking the following questions (which could even use Tkinter =):

* Issue #s
* Did someone else help out with this; allows making sure they were not accidentally left out of Misc/ACKS and add them if necessary or at least that they are mentioned in the commit message
* One line explanation for NEWS
* Optional extra bits to go into the commit message
* Should this go into What's New (e.g. new feature, backwards-compatible breakage); can add a `WN` or `WhatsNew` to the file name or something so that if someone like David tries to update the What's New doc they can tell by the file name that it may be important to cover (and maybe even only add the marker if What's New is NOT edited to know it's more like a TODO item)

Another side-effect of marking entries as worthy of being in What's New is that it should be easy to tell what should potentially be added as an addendum to What's New and to highlight at the end of the doc so that every micro release people can notice what was added. Heck, the logical conclusion is to split up What's New into individual files with a placeholder of the issue that was marked as worthy of inclusion and then edit it before release and auto-generate What's New (but one thing at a time =).

Anyway, ignoring all of this unstructured brain dump of mine, I'm can support doing a separate file approach.

The `hg commit -l ...` tricks are just for those who tend to use exactly the same text for commit message and NEWS, and I only point them out simply because I don't know how many people are even aware of the -l option :).  If you want to use a NEWS entry as a base for a commit message without the two being exactly the same, it's also fairly easy to do `news.py | hg commit -l - && hg commit --amend` which will open the commit message in your usual commit message editor (at the cost of rolling back and re-doing the commit).

I also just thought of adding a 'who wrote it' question, and I like the idea of using it to manage Misc/ACKS.  I'm also intrigued by the suggestion of using Tkinter (when available), and the WhatsNew management.  But as you said, one thing at a time :)

First thing is, decide what we're actually going to do.  news.py obviously needs a fair bit of work before we can use it regularly, which I don't mind doing, but only if it will actually be used ;)

I want no script asking me questions.  Post-facto errors for omissions are fine (and if I have to positively say no in the input file, that's fine).  tkinter is right out.
 
If you *also* want to make a script that asks questions (or even a tkinter ap), that's fine, but it should not be the main interface, it should be a wrapper.

One thing I really don't like about the separate file approach is that you lose the obvious chronology.  It's probably not a blocker, but it is definitely a disadvantage.

You lose chronology in the directory, but not necessarily in the output; if you sort based on first commit time then you retain the chronological ordering in the merge.

R. David Murray wrote:
> I want no script asking me questions.  Post-facto errors for omissions are
> fine (and if I have to positively say no in the input file, that's fine).
> tkinter is right out.

I have been planning a command line interface, something along the lines of
`news.py --issue 12345 --section Core --message "Converted ham to spam"`.  Of
course, there would be short options as well.  The interactive interface was
just what I wrote first because it seemed easiest at the time :).  Adding the
command line part shouldn't be too hard with argparse, though; I'll see if I
can do it this afternoon.
 
> If you *also* want to make a script that asks questions (or even a tkinter
> ap), that's fine, but it should not be the main interface, it should be a
> wrapper.

I'm fine with that.

> One thing I really don't like about the separate file approach is that you
> lose the obvious chronology.  It's probably not a blocker, but it is
> definitely a disadvantage.

If we give the .news files names of the form "nnn-issuexxxxx.news" where 'nnn'
is the number of .news files that already exist in the target folder (or the
whole tree) and "-issuexxxxx" is only present if an issue number was given,
chronology would be kept, though it might get a bit hairy merging between
branches.  It would be unlikely to have actual merge conflicts (unless your
news entry wasn't linked to an issue), but might cause things to get a bit out
of order.

Assuming a satisfactory interface to give news.py the proper information and a
good resolution to the chronology issue, how do you feel about the rest of the approach?

Well, I guess you could sum it up as -0.  I personally don't feel the need for a change, but if the chronology problem is solved and it isn't *harder* to make the needed NEWS changes, then I'm not going to object.

Like David I don't really care, but it seems Zach's script may show promise. I think it would be nice if "news.py --commit":
1) popped up an editor window (like hg commit does)
2) processed the editor results and looked for hints of an issue number
3) automatically created that file, hg add'ed it and then hg commit'ed the whole thing with the right commit message

I don't like Brett's approach.

Note that the editor may be pre-filled with a skeleton:

# --- Fill in the NEWS message here
# (lines prefixed with '#' will be removed)
Issue #xxx: ....

# --- Additional text for the commit message below
# (won't be reflected in NEWS)

# --- Commit information
# user: Antoine Pitrou <email>
# branch: 3.4
# modified files:
# <diffstat>

There also needs to be a way to specify a different NEWS entry from the commit message first line.

Have you considered how this is going to change if/when PEP 462 will be implemented?  AFAIU PEP 462 suggests that commits happen directly from the bug tracker (or something equivalent), so we will likely start to add the NEWS entry there as well.  Assuming this will happen and it's not too far away in the future, it might be wiser to wait or move toward a system that will make the switch to PEP 462 easier.

FWIW I still think that the current situation is fine as is.

PEP 462 is months away from going anywhere, and I personally find the
current NEWS handling a major barrier to feeling inclined to work on bug
fixes.

The status quo will *definitely* be PEP 462 incompatible, though, since it
would typically prevent applying the same patch to multiple branches.

Yes, after thinking about this this weekend, it is clear to me that in the future we will *need* a scheme where by the NEWS entry can be safely included in the patch in some form.

I think the commit message is also going to be in the patch, which will be in 'hg export' form, if I'm understanding the stuff the Mecurial folks were telling us at PyCon correctly.

So, in theory the script approach would still work, if anybody can run it (to commit locally and then export) and if what it produces is something that won't get stale.

A file per news entry seems a bit much, but an optional file per developer would solve most of the problem for those who have a problem with the status quo. Add a directory named, for instance, news.3.4.1. Put in a template with the allowed section headings. Call it something like aaTemplate.txt to sort first. To avoid merge conflicts due to entries for new features in default, quadruple space between headers so the only non-blank context would be the section header above. Developers like me who have a problem with the existing system could copy the file, rename it with their name (terry.reedy.txt, etc), hg add it, and use it for news entries (with extra blank lines to maintain the clean context.

A script could be written to sync the working directory (pull and merge), move entries into NEWS (skipping over blank lines), recopy the template, commit and push.

If I were working on non-Idle code issues, I would seriously consider doing the above with a private, non-repository file in /MISC that I might merge one a week or so.

Someone objected to changes that result in news entries being out patch push order. This is already not a rule because the devguide mentions inserting new items at random positions to avoid conflicts due to another commit. Also, News items are frequently pushed sometime after the corresponding patch. I don't know if this is because people forget or because they want to isolate any hassle with a news conflict. In any case, the News entries are not necessarily time ordered now.

From a user perspective, having library news items sorted by affected module, with code, doc, and test changes collected together, would generally be more useful.

> This is already not a rule because the devguide mentions inserting new 
> items at random positions to avoid conflicts due to another commit.

Really?

"""New NEWS entries are customarily added at or near the top of their respective sections, so that entries within a section appear in approximate order from newest to oldest. However, this is customary and not a requirement."""

> In any case, the News entries are not necessarily time ordered now.

IME, they mostly are. It's true it's not a requirement.

On 4/20/2014 7:59 PM, Antoine Pitrou wrote:
>
> Antoine Pitrou added the comment:
>
>> This is already not a rule because the devguide mentions inserting new
>> items at random positions to avoid conflicts due to another commit.
>
> Really?
>
> """New NEWS entries are customarily added at or near the top of their respective sections, so that entries within a section appear in approximate order from newest to oldest. However, this is customary and not a requirement."""

Random is too loose for what was actually committed and I will correct 
it. Further down in the same section: "A nice trick to make Mercurial’s 
automatic file merge work more smoothly is to put a new entry after the 
first or first two entries rather than at the very top. This way if you 
commit, pull new changesets and merge, the merge will succeed 
automatically."

>> In any case, the News entries are not necessarily time ordered now.
>
> IME, they mostly are. It's true it's not a requirement.

For the recommendation actually put in the devguide, change 'random position' to 'position near but not at the top'.
https://docs.python.org/devguide/committing.html#news-entries ends with
"A nice trick to make Mercurial’s automatic file merge work more smoothly is to put a new entry after the first or first two entries rather than at the very top. This way if you commit, pull new changesets and merge, the merge will succeed automatically."

My main point was that the devguide a) recognizes that there is an issue and b) does not require strict date order. Merging items into the NEWS on a daily basis would keep approximate time order. Hourly merges would do even better.

This recommendation, however, is not very effective. It only only avoids conflict with another patch if the other person does not use the 'nice trick' but puts the news entry at the top. Unless one looks carefully, it does not help the problen of merging maintenance bugfix items into a default list that also contains enhancement news not in the maintenance list*. In fact, blindly putting an item between two news items rather than a header and one news item makes that conflict more likely.

* There have been times, like last Jan-Feb, when default only items dominated the default news list.

In order for things to work with a patch gating system, I believe it will be the most practical for the news items to be each in a separate file.

> Unless one looks carefully, it does not help the problen of merging 
> maintenance bugfix items into a default list that also contains 
> enhancement news not in the maintenance list*.

What if instead of having sections in Misc/NEWS for core/library/documentation/etc., we simply make sections for bug fixes and enhancements?  Wouldn't this basically solve the conflict problem (assuming all/most bug fixes are backported, and new features are only on default)?

That would probably work for now, but it wouldn't work for the patch gating system.  On the other hand, it would sure make it easier to build/check whatsnew.

Also having a list of enhancements and bug fixes might be more meaningful for users than a list of core issues vs library issues vs other similar sections.

This could also be done with two separate files, with the "new features/enhancements" file existing only on default and the "bug fixes" file existing across branches.  From there it shouldn't be difficult for the gating system to append/prepend/insert the news.

That makes the tooling of the gating system harder, though.  If the NEWS can just be a file in the patch, we don't have to have any special tooling for it in the gating system.

If we put news items in a database keyed by issue number (and I think it reasonable that all news-worthy patches should have a tracker issue), then there would be no conflicts. (We already avoid simultaneous commits to the same issue.) If the database had fields such as the type of issue/patch and the component affected, then different queries could create text files sorted differently.

We already have such a database with such fields and indexed by issue number -- the tracker itself. We could either copy data from the tracker into a separate database or add to the tracker a textbox News field that is only editable by either by the 'Assigned to' person or perhaps any committer.

I personally would prefer a box that I could fill out when closing the issue. (The tracker could even ask for verification when closing a 'fixed' issue with a blank news.) News summaries could be extracted at any time by scanning commit messages since the last release for issue numbers. A news box on the tracker would give people looking at the issue thereafter a quick summary of the result of the issue without scanning through all the messages and checking the patches. 

Since the issue formatting is being reviewed, I think the issue summary should also include the exact releases patched. This would help people to see if an issue is only open for a possible backport, It might also help a program selecting issues for a news report.

I am, of course, aware that I have glossed over many details.

Moving News items from the repository to the tracker, where I think they initially belong anyway, would also remove them as an issue for a future gating system.

Terry J. Reedy wrote:
> Moving News items from the repository to the tracker, where I think
> they initially belong anyway, would also remove them as an issue for
> a future gating system.

I think News items are best left in the repository just so that for any given snapshot of the repository, you can see what significant changes are present.

Yeah, Guido was strongly in favor of that too.

Agreed.  Once the gating system is in place nothing prevent us to read the NEWS entry either from the patch that is being committed or from a field in the tracker and then including it together with the patch once it is approved.

A sequential log of commit messages for a particular branch would give one an even better snapshot view of activity since not all commits have news messages and even when they do, commit messages sometimes have additional info. But see below.

The essence of my proposal is that the ORIGINAL entry of a news item by a human be into the tracker where there is no conflict and where it can be seen by people looking at the issue. Eliminating that conflict is the subject of this issue. After that, items can be mechanically copied wherever desired, including, possibly, into a file in the repository. My apology if this part was not clear enough.

Such mechanical copying and  re-arranging should not have the conflicts that are the subject of this issue. In particular, if the gating system is the only entity that edits particular news files (and I propose that there could be more than one), it will not have conflicts with other editors. 

Since these would be secondary, derived files, I don't see why any or all should be in the hg repository, as opposed to being part of the doc package available online along with other derived files. In other words, the hg repository is for original master files used to create derivative files nearly always not kept in the repository. We do not put .html files derived from master .rst files into the repository. So I do not see why files derived from hg repository commit messages and the tracker sql repository should go into the hg repository either. But I don't especially care if they are.

I doubt Guido favors a system that inhibits commits by making them unnecessariy painful and troublesome.

Folks, this is *really, really, simple*: one file per NEWS entry. How
we arrange them is just a detail. Don't go off trying to invent wild
exotic alternatives that spread state across multiple sources of truth
- significant historical info belongs in the version control system,
and NEWS entries are how we highlight "this one is significant"
(relative to other commits). A hg extension that prepopulates the
commit message from a NEWS entry included in the patch wouldn't be
difficult (especially since some of the Mercurial devs are likely to
join the new core workflow list).

My concern with a file for each entry is a possible slowdown of some operations, like TortoiseHg resyncing the diff between repository and working directory (it is not instantaneous even now). However, if there are multiple directories and if they are emptied periodically, so that there are never more than a few hundred in any directory, it might not be too bad. I agree that this idea is otherwise an improvement.

FWIW: I don't think we need to manage the news entries in the NEWS
file. Instead, we could simply add a field to the bug tracker
called "news entry" and populate that as necessary.

During release, this information can then be used to create a
NEWS file per release. This file would not be touched by anyone
other then the release manager.

As a result, there would not longer be any merge conflicts, and
the news entries can still be changed before the release
if necessary, and, of course, the generated NEWS file can also
be edited as the release manager sees fit.

Tickets which do not need a news entry simply leave the news entry
field blank.

We really don't need a file system database for these things.

(PS: eGenix has been using such a system internally for several
years and it works great.)

One of the Mercurial devs convinced me to pursue what I had initially proposed in msg197645 and write a merge script (attached).

The script is still a proof of concept, it makes a few assumptions and doesn't handle all the cases, but I did a few tests and it seems to work for at least some cases.  If people like it it can be improved.

In short it parses Misc/NEWS, see what news entries have been added in the changeset(s) that it's being merged, and adds them at the top of the corresponding section.

To enable it download the file, set it as executable (chmod +x newsmerge.py), and add the following to .hg/hgrc:

[merge-tools]
newsmerge.executable = /path/to/newsmerge.py
newsmerge.priority = 100
newsmerge.premerge = True
newsmerge.args = $base $local $other -o $output

[merge-patterns]
Misc/NEWS = newsmerge

This will kick in only if the regular merge results in a conflict (so if you don't see any of the debug output from newsmerge it means that mercurial managed to merge Misc/NEWS on its own without conflicts).

So Twisted is actually in the process of pulling out their tooling they use for the separate files technique and making it a stand alone project. Seems like it'd make sense to reuse/contribute to that?

It's at https://github.com/twisted/newsbuilder

I would like fixing NEWS to be the top infrastructure improvement project, as it is my biggest time waster, and certainly the most obnoxious. It definite makes working on Idle less pleasant.

Since an empty 3.4.2a1 section was added to 3.4 NEWS, after 3.4.1 was released, Idle news items always fail to merge *even when there is no real conflict*.  I think the problem is the offset of hundred of lines (now a thousand) between the position in the 3.4 file and the position in the 3.5 file.  The hg merge jams the Idle entry somewhere in the middle of the Library section, creating spurious and bizarre-looking merge 'conflicts', instead of looking far enough down in 3.5 NEWS to find the proper context for successful insertion.

I am not the only person this happens to, as the same should be true for all the other sections that come after the Library section.  A couple of months ago, someone pushed a post-push patch to remove artifacts not properly removed before the original push.  Today I just removed what looked like an artifact from a Demo news item.

Le 02/12/2014 00:11, Terry J. Reedy a écrit :
> 
> Terry J. Reedy added the comment:
> 
> I would like fixing NEWS to be the top infrastructure improvement
> project, as it is my biggest time waster, and certainly the most
> obnoxious. It definite makes working on Idle less pleasant.

You could pair with Pierre-Yves David:
https://mail.python.org/pipermail/python-dev/2014-December/137393.html

Good idea. I will send him a note.

As an interim step, should we add Ezio's "newsmerge.py" to Tools/scripts and instructions for enabling it to the devguide?

That seems straightforward enough, and doesn't require any global workflow changes.

Thanks for the reminder/suggestion.  Re-reading Ezio's two patches (newsmerge.py and .hg/hgrc additions in msg217079), they appear to jointly automate what I now do by hand (revert to local and dump merge artifacts, replace with edited file with new entries pasted in, and mark as resolved) .  I also checked that the current Windows installers make .py files executable, so hg should be able to run newsmerge on Windows.  I have downloaded the file and made the additions to my 3.5 hgrc to test next time I have a news entry.

It should be mentioned in this issue that the core-workflow mailing list decided on having NEWS entries being attached to the related issue in the issue tracker. This was then presented to python-committers and no one objected to the idea.

Is that something that might happen 'soon' or only when the whole workflow is redone?

As soon as someone finds the time to make the change we can switch. While this is a necessary requirement to change the workflow it isn't gated by it either.

Okay.  I got tired of the constant Misc/NEWS merge conflicts, so I wrote a tool to fix the problem.  It's checked in to Bitbucket here:

https://bitbucket.org/larry/mergenews/

There's a readme, which you'll see rendered on that page.

I don't know what you mean by "attach Misc/NEWS entries to the issue tracker".  I figure, the Misc/NEWS entry is used as the checkin comment, so Roundup Robot already adds it for you, so maybe that's good enough.

Can we maybe switch to this for 3.6?  Should we consider switching all 3.x branches under active development to use this approach?

In case you're too lazy to go visit the link to my "mergenews" repository and read the readme...

mergenews has three tools:

* splitnews, which splits the existing Misc/NEWS file into hundreds of individual files,
* mergenews, which merges the hundreds of individual files back into a single coherent Misc/NEWS file, and
* pyci, a front-end for "hg ci" which automatically writes out the checkin comment as a new news entry into the right spot so mergenews will see it.

It's all basically working already.  pyci limits you in what command-line options it supports, other than that I think it all does what you want.  Can we use it please?  ;-)

What I mean by "attach Misc/NEWS entries to the issue tracker" is there will literally be a field in the issue tracker to enter the entry for the change log and Misc/NEWS will simply be auto-generated from the issue tracker. You can't use the commit message for what to put in the change log because too many people are against that solution (trust me, I tried to convince them otherwise).

Larry: commit messages are for other developers, which news entries are ultimately for users.  Some developers want (insist on) the freedom to make the two different, with different details.  I am using that for Idle patches since NEWS entries become 'What New for Idle' in each release.

Brett, I presume that you mean a new NEWS box like this Comment box, maybe with 3 lines and a new Section: box like Components.  Question: Is it impossible to put labels above a box rather than to the right?   

We could allow that news section to be (initially) filled out by someone other than the committer.  Latter could (optionally) copy and paste as (initial draft of) commit message.  Having message visible on issue might result in typos being corrected sooner, and would make it much easier to edit news message later (But editing after close should be limited to core devs).

Larry, a commit hook would be better that a separate pyci since TortoiseHG [Commit] button would still work.  With Brett's solution, a commit hook could email commit message to tracker as NEWS message (if empty).  And yes, change should apply to all versions.  Otherwise, NEWS commits to older versions would have to be null-merged.

Yes, a text box or something along those lines. I assume the selection label question is to specify a section of the change log for something to go into. That's actually not necessary as the proper section is implied by the Components label.

And we could allow people to initially fill it in if desired, but I think worrying about whether to open it up or not is premature optimization; I'm fine with starting with it for only people with Developer privileges on the issue tracker.

I believe auto-filling it from the commit message (for those who don't put in the effort to treat the two audiences differently :) was something we discussed, but it would be a separate enhancement after the news box is added to the tracker.  Which I might get to next month or I might not...

I think Larry's "split news" and "merge news" ideas are still useful as a bridging mechanism to help us get from the status quo to a tracker based solution.

It would just mean that, at some point in the future, the commit hook would change to be a post-commit hook that pushed to the issue tracker, rather than a pre-commit hook writing to a file in the source repo, and "merge news" would start pulling from the tracker, rather than from the source repo.

> [...] commit messages are for other developers, which news entries are
> ultimately for users.  Some developers want (insist on) the freedom to
> make the two different, with different details.

That's easy enough to accommodate.  I updated "MergeNEWS" with this feature.  Well, a similar feature:

The file you edit at checkin time starts with "Issue #:".  You fill that out and that becomes your Misc/NEWS item, and the top of your checkin comment.

There's also a commented-out line of dashes below there.  Anything you put there is appended to *the checkin comment only*.

So, you get the user summary as per Misc/NEWS at the top, then you can elaborate in all the technical and flowery language you like.  Also, each section is optional (though obviously you have to fill out *one* of them).

It'd be easy to make the two comments completely separate.  I just figured, the Misc/NEWS entry is presumably a nice high-level summary of the solution, why not leave that there and save the developer from having to summarize it twice.

Because the two summaries should be *different* in most cases.  The first line of the checkin should ideally be a single sentence of less than 80 characters, while the NEWS item will almost always be longer.  The NEWS item should be a complete summary, while the first checkin line is a 'summary in context' (ie: it can assume the reader knows the files that were changed, since log -v will show you that).  This obviously requires that the succeeding paragraphs expand on that minimal summary (in most cases) in a way that doesn't look like a NEWS entry.

Now, this is obviously my preferred style and others may have different styles and/or disagree, but I'd like that style to be supported.

Note also that a big part of putting the NEWS entries in the tracker is so that they can be *edited*.  Populating them initially from the checkin message is OK, but I predict that the actual workflow will be to create the NEWS entry in the tracker as part of getting the patch to the 'commit ready' point, which means that having a tool that pre-populates the commit message from the tracker NEWS entry will probably be more useful to most people.  (And yeah, I'm sure that means a lot of commits will just have the NEWS entry as the commit message, which will make me sad, but oh well :)

Just noting a potential practical benefit of the "NEWS entry first" approach that David suggests: I believe it will help encourage the creation of a succinct answer to "Why are we making this change?" as part of the patch review process.

That's then useful both during the patch review itself (since there's a shared understanding between reviewers and implementors of the goal to be achieved), as well as when the change is released (since there's hopefully a user centric explanation of *why* something changed, rather than merely *what* changed)

However, the key trap I'd like us to avoid falling into is letting the fact a particular approach falls short of our ideal approach deter the introduction of interim improvements. We're going to need the checked in filesystem database anyway to backfill historical NEWS entries after switching to a generated NEWS file, so it seems harmless to me to switch to it early and then incrementally add to it until a tracker based solution is available.

I don't agree that in all cases the Misc/NEWS entry and the checkin comment must be wholly different.  While I concur that that's called for now and again, I believe this to be rare.  Most checkins are small changes, and a single-line summary for both Misc/NEWS and the first line of the checkin comment will suffice.

However, clearly the tool should *support* making them wholly separate.  But that's easy to support.  pyci now supports a "-n" or "--newsonly" flag, which instructs it to only add the Misc/NEWS item and to not check in.  This is presumably also nicer for folks used to shell-integrated hg tools (TortoiseHG etc).

I think it's important that this *not* be the default behavior, as it's going to be hard enough to get the core dev community to adopt this tool, and I would prefer it make the experience nicer rather than clunkier.

I never said they always had to be different, and "wholly different" is certainly not the case, nor do I think I implied that.

So if your tool supports my scenario that's all I'm asking for :)

I sometimes put a bit more info in the commit message, but it's true that generally it's a copy/paste job as far as I'm concerned.

I came across OpenStack's tool for this problem today: http://docs.openstack.org/developer/reno/design.html

I think it's significantly more complex than we need for CPython, but also still interesting as a point of reference.

It's already mentioned in PEP 512, but I'll also add a reference here to https://pypi.python.org/pypi/towncrier, Amber Brown's release note manager that allows Twisted style release notes management to be used with other projects.

Has this been superseded by core_workflow issues?

Yep, core-workflow work has superseded this issue.