This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Make json.tool soak up input before opening output for writing
Type: behavior Stage: resolved
Components: Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: CharString, lukasz.langa, methane, miss-islington, remi.lapeyre
Priority: normal Keywords: patch

Created on 2021-10-28 12:59 by CharString, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 29269 closed CharString, 2021-10-28 12:59
PR 29273 merged CharString, 2021-10-28 15:40
PR 29445 merged miss-islington, 2021-11-06 17:11
PR 29446 merged miss-islington, 2021-11-06 17:11
PR 29478 closed CharString, 2021-11-08 22:09
PR 30659 closed methane, 2022-01-18 09:34
Messages (8)
msg405182 - (view) Author: Chris Wesseling (CharString) * Date: 2021-10-28 12:59
json.tool is very cute and handy for making json readable.

But rewriting a file in place requires tools like sponge (on POSIX) or a tmpfile, because 

$ python -m json.tool foo.json foo.json

results in an empty foo.json.

I propose soaking up the infile before opening the outfile for writing, to prevent that. Much like sort -o does, but without the explicit flag.
The patch I have prepared changes no behaviours, other than preventing an empty file... (still I see this as an enhancement and not a bug fix)
msg405866 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-06 16:57
This is arguably a bugfix since truncating the `infile` cannot be construed as a useful feature in any sense. Thus I'm inclined to backport this to 3.10.1 and 3.9.9 too.
msg405868 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-06 17:11
New changeset 815dad42d53fc40a6dc057e067f4a8a885c3b858 by Chris Wesseling in branch 'main':
bpo-45644:  Make json.tool read infile before writing to outfile (GH-29273)
https://github.com/python/cpython/commit/815dad42d53fc40a6dc057e067f4a8a885c3b858
msg405872 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-06 18:01
New changeset 6603f6b5bdf0ed164bad4eb432619da2e95fb518 by Miss Islington (bot) in branch '3.10':
bpo-45644:  Make json.tool read infile before writing to outfile (GH-29273) (GH-29445)
https://github.com/python/cpython/commit/6603f6b5bdf0ed164bad4eb432619da2e95fb518
msg405873 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-06 18:01
New changeset a932631890d148444e5f9b09f4b57305475d6386 by Miss Islington (bot) in branch '3.9':
bpo-45644:  Make json.tool read infile before writing to outfile (GH-29273) (GH-29446)
https://github.com/python/cpython/commit/a932631890d148444e5f9b09f4b57305475d6386
msg405874 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-06 18:02
Thanks, Chris! ✨ 🍰 ✨
msg405913 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2021-11-07 21:12
The proposed path does not work for `python -m json.tool --json-lines test.json test.json`. There is also https://github.com/python/cpython/pull/7865 that aims to fix this bug but I did not have the time to get back to it.
msg405937 - (view) Author: Chris Wesseling (CharString) * Date: 2021-11-08 09:06
@Remi

I left the current behaviour for --json-lines untouched, on purpose.
My reasoning was that the json-lines format is often seen in JSON streaming, and I didn't want to break the case where the input is an endless stream from stdin or a named pipe. And to keep the patch simple with minimal change to the exposed interface.

Simplest fix for the case in this current code would be: iff infile equals outfile (minding their types), call list(objs) on the generator[1] to materialise it in one go. The only case where that would break would be when infile == outfile and is a named pipe, but I can't imagine why I would want to both read and write to the same FIFO, other than comedic effect.

  [1] https://github.com/python/cpython/blob/122ca4d73faba279a579aa2011fa34661ce537a2/Lib/json/tool.py#L65
History
Date User Action Args
2022-04-11 14:59:51adminsetgithub: 89807
2022-01-18 09:34:45methanesetnosy: + methane

pull_requests: + pull_request28860
2021-11-08 22:09:34CharStringsetpull_requests: + pull_request27729
2021-11-08 09:06:27CharStringsetmessages: + msg405937
2021-11-07 21:12:49remi.lapeyresetnosy: + remi.lapeyre
messages: + msg405913
2021-11-06 18:02:21lukasz.langasetstatus: open -> closed
resolution: fixed
messages: + msg405874

stage: patch review -> resolved
2021-11-06 18:01:53lukasz.langasetmessages: + msg405873
2021-11-06 18:01:26lukasz.langasetmessages: + msg405872
2021-11-06 17:11:49miss-islingtonsetpull_requests: + pull_request27700
2021-11-06 17:11:43miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request27699
2021-11-06 17:11:42lukasz.langasetmessages: + msg405868
2021-11-06 16:57:44lukasz.langasetversions: + Python 3.9, Python 3.10, Python 3.11
nosy: + lukasz.langa

messages: + msg405866

type: enhancement -> behavior
2021-10-28 15:40:01CharStringsetpull_requests: + pull_request27537
2021-10-28 12:59:53CharStringsetkeywords: + patch
stage: patch review
pull_requests: + pull_request27532
2021-10-28 12:59:07CharStringcreate