How can I incrementally generate JSON calling jq from bash repeatedly?-CodePudding

Is there some accepted 'best practice' of generating JSON documents using bash and jq? I have a script to gather various data, and to make it easier to further process using other tools I'd like to output the data in JSON format. So I'm using jq to make sure all the quoting etc. gets done correctly, as recommended in this answer: https://stackoverflow.com/a/48470227/75652. However, I'm struggling with how to generate it piecemeal instead of one giant jq call at the end. E.g. something like


read foo <<<$(</path/to/some/oneliner/file)
jq -n --arg f $foo '{foo: $f}'

bar=$(some_command)
jq -n --arg b $bar '{bar: $b}'

Will generate two separate objects (which can be processed with tools that support various more or less informal "JSON streaming" formats, including jq) whereas I'd want a single object, something like


{ "foo": SOMETHING, "bar": SOMETHING_ELSE }

but I can't do that with multiple jq calls as jq will complain that the incomplete JSON is malformed.

And to further add some complexity, in some cases I need to generate nested JSON structures. In another language like python I'd just put all the data in a set of nested dictionaries and then dump it to JSON in the end, but nested dictionaries in bash seem very tedious..

CodePudding user response：

You can save and process intermediary JSON for the next jq command:

#!/usr/bin/env bash

read -r foo <a.txt

json="$(jq -n --arg f "$foo" '{foo: $f}')"


bar="$(pwd)"
jq -n --arg b "$bar" "$json"' {bar: $b}'

# or alternatively
jq --arg b "$bar" '.bar=$b' <<<"$json"

CodePudding user response：

When reaching some complexity (or when I need to externally process some of the data between transformations) I typically end up using something along the lines of

jq --slurpfile foo <(
  
  # first inner shell script
  read foo <<<$(</path/to/some/oneliner/file)
  jq -n --arg f $foo '{foo: $f}'

) --slurpfile bar <(

  # second inner shell script
  bar=$(some_command)
  jq -n --arg b $bar '{bar: $b}'

) -n '$foo[0]   $bar[0]'

That way, the outermost jq call may still have a 'real' input on its own, and the inner calls are fairly maintainable with all bash variables in scope.