Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend json.tool to handle jsonlines (with a flag) #75734

Closed
EricMoyer mannequin opened this issue Sep 22, 2017 · 9 comments
Closed

Extend json.tool to handle jsonlines (with a flag) #75734

EricMoyer mannequin opened this issue Sep 22, 2017 · 9 comments
Assignees
Labels
3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@EricMoyer
Copy link
Mannequin

EricMoyer mannequin commented Sep 22, 2017

BPO 31553
Nosy @rhettinger, @vstinner, @ezio-melotti, @serhiy-storchaka, @lisroach
PRs
  • bpo-31553: Add --jsonlines option to json.tool. #3846
  • bpo-31553: add --json-lines option to json.tool #10051
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/lisroach'
    closed_at = <Date 2018-11-07.10:25:30.301>
    created_at = <Date 2017-09-22.16:41:06.565>
    labels = ['3.8', 'type-feature', 'library']
    title = 'Extend json.tool to handle jsonlines (with a flag)'
    updated_at = <Date 2018-11-07.10:25:30.300>
    user = 'https://bugs.python.org/EricMoyer'

    bugs.python.org fields:

    activity = <Date 2018-11-07.10:25:30.300>
    actor = 'serhiy.storchaka'
    assignee = 'lisroach'
    closed = True
    closed_date = <Date 2018-11-07.10:25:30.301>
    closer = 'serhiy.storchaka'
    components = ['Library (Lib)']
    creation = <Date 2017-09-22.16:41:06.565>
    creator = 'Eric Moyer'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 31553
    keywords = ['patch']
    message_count = 9.0
    messages = ['302755', '302781', '302783', '302785', '302904', '302953', '328290', '328396', '329411']
    nosy_count = 6.0
    nosy_names = ['rhettinger', 'vstinner', 'ezio.melotti', 'serhiy.storchaka', 'lisroach', 'Eric Moyer']
    pr_nums = ['3846', '10051']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue31553'
    versions = ['Python 3.8']

    @EricMoyer
    Copy link
    Mannequin Author

    EricMoyer mannequin commented Sep 22, 2017

    json.tool should have the ability to format jsonlines data. It is a very commonly used format in many industries. Right now when given jsonlines (for example):

    echo -e '{"ingredients":["frog", "water", "chocolate", "glucose"]} | python -m json.tool
    

    Works. But:

    echo -e '{"ingredients":["frog", "water", "chocolate", "glucose"]}\n{"ingredients":["chocolate","steel bolts"]}' | python -m json.tool
    

    Reports an error: "Extra data: line 2 column 1 - line 3 column 1 (char 58 - 100)"

    I propose that the above behavior be retained but:

    echo -e '{"ingredients":["frog", "water", "chocolate", "glucose"]}\n{"ingredients":"chocolate","steel bolts"}' | python3.7 -m json.tool --jsonlines
    

    Should print:
    {
    "ingredients": [
    "frog",
    "water",
    "chocolate",
    "glucose"
    ]
    }
    {
    "ingredients": [
    "chocolate",
    "steel bolts"
    ]
    }

    If someone else agrees this is an appropriate enhancement, I can start work on a PR in a couple of weeks.

    @EricMoyer EricMoyer mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Sep 22, 2017
    @rhettinger
    Copy link
    Contributor

    This seems like a reasonable enhancement.

    @serhiy-storchaka
    Copy link
    Member

    Jsonlines is an external to JSON format. You should split the data on lines and pass every line to the JSON parser separately. The same you should do with json.tool.

    $ echo -e '{"ingredients":["frog", "water", "chocolate", "glucose"]}\n{"ingredients":["chocolate","steel bolts"]}' | while read -r line; do echo -n "$line" | python -m json.tool; done
    {
        "ingredients": [
            "frog",
            "water",
            "chocolate",
            "glucose"
        ]
    }
    {
        "ingredients": [
            "chocolate",
            "steel bolts"
        ]
    }

    @ezio-melotti
    Copy link
    Member

    I agree with Raymond that this is a reasonable request.
    Even if jsonlines is not part of the JSON specification, the format is quite common, and practicality beats purity (espcially considering that we are just talking about json.tool).

    @rhettinger
    Copy link
    Contributor

    Eric, did you want to write the PR yourself or would you like for Lisa to bring it to fruition? If you're going to do it yourself, Lisa will serve as the primary reviewer (with either Ezio or me doing the final sign-off).

    @EricMoyer
    Copy link
    Mannequin Author

    EricMoyer mannequin commented Sep 25, 2017

    I planned to write the PR myself, on the principle of "it's my itch, I
    should scratch it." But if you think it is better for someone else to write
    it, you know the codebase better than I, and you know how long a PR takes
    to review/fix. My intention is to help. I'll do whatever is easier for you.

    On Sep 24, 2017 10:01 PM, "Raymond Hettinger" <report@bugs.python.org>
    wrote:

    Raymond Hettinger added the comment:

    Eric, did you want to write the PR yourself or would you like for Lisa to
    bring it to fruition? If you're going to do it yourself, Lisa will serve
    as the primary reviewer (with either Ezio or me doing the final sign-off).

    ----------
    assignee: -> lisroach
    nosy: +lisroach


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue31553\>


    @serhiy-storchaka
    Copy link
    Member

    The output is not a valid JSON format, and is not a valid JSON Lines format. What you are going to do with it?

    @vstinner
    Copy link
    Member

    The output is not a valid JSON format, and is not a valid JSON Lines format. What you are going to do with it?

    It looks useful to me to read a nicely indented JSON in the terminal (without specifying an output file). It's more readable that compact JSON on a single line.

    But should we add an option to write the output on a single line? It would be the opposite of the tool description:
    "A simple command line interface for json module to validate and *pretty-print* JSON objects."

    @serhiy-storchaka
    Copy link
    Member

    New changeset f194479 by Serhiy Storchaka (HongWeipeng) in branch 'master':
    bpo-31553: add --json-lines option to json.tool (bpo-10051)
    f194479

    @serhiy-storchaka serhiy-storchaka added 3.8 only security fixes and removed 3.7 (EOL) end of life labels Nov 7, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants