Updating invalid json output to make it valid from a file-CodePudding

I have the following data that I already insert in a file called file.json:


{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169149"
}
{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169159"
}
{
  "ip": "1.1.1.1",
  "tested_count": 10,
  "last_scan_date": "1673169326"
}

To be specific, this is how I added it using jq every time I run the script.sh:

#!/bin/bash

# I run this script 3 times to insert into file.json:

#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 10

OUTPUT=`file.json`
JSON_OUTPUT='data:{}'

IP="$1"
TEST_COUNT="$2"

JSON_OUTPUT=$(jq \
  --arg ip "${IP}" \
  --argjson tested_count "${TEST_COUNT}" \
  --arg last_scan_date "$(date  %s)" \
  '{ip: $ip, tested_count: $tested_count, last_scan_date: $last_scan_date}' <<<"${JSON_OUTPUT}")

# Write the output to the file
echo "${JSON_OUTPUT}" >>"${OUTPUT}"

Now, based on the output from file.json it looks like it is not a valid json. So, my aim is to transform that into a valid json that looks like below:

{
  "data": [
    {
      "ip": "1.1.1.1",
      "tested_count": 17,
      "last_scan_date": "1673169149"
    },
    {
      "ip": "1.1.1.1",
      "tested_count": 17,
      "last_scan_date": "1673169159"
    },
    {
      "ip": "1.1.1.1",
      "tested_count": 10,
      "last_scan_date": "1673169326"
    }
  ]
}

I have seen an example of this here but the example shown there is using a looping method which is not necessary in my case as my json data is dynamically added each time the script.sh above run.

What I have tried so far is using an update inputs filter like this but obviously, this does not fix the json as a valid syntax.

echo "$json_query" | jq -n '.data |= [inputs]' >> file.json

CodePudding user response：

You can you sed:

echo '{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169149"
}
{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169159"
}
{
  "ip": "1.1.1.1",
  "tested_count": 10,
  "last_scan_date": "1673169326"
}' | sed 's/}/},/;s/^/    /; 1 s/^/{\n  "data": [\n/; $ s/,/\n  ]\n}/'

CodePudding user response：

Doing it properly with jq.

I added lots of comments for both the shell and the jq script parts. Don't hesitate to question me in comments if something is unclear, anyway. That would be preferable to blindly copying code.

#!/usr/bin/env bash

#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 10

# No reason to use ALL_CAPS variable names unless these are environment
output_file='file.json'

# ${var:?} expansion means var is mandatory and it will error if not provided
arg_ip="${1:?}"
arg_test_count="${2:?}"

# If output file does not exist
if ! [ -f "${output_file}" ]; then
  # Initializes it with object of empty JSON data array
  jq --null-input '{"data":[]}' > "${output_file}"
fi

# Buffer the output of processing updated JSON data
json_buffer=$(
  jq \
    --arg arg_ip "${arg_ip}" \
    --argjson tested_count "${arg_test_count}" \
    '
# Add new object entry to the data array
.data  = [{
  "arg_ip": $arg_ip,
  "tested_count": $tested_count,
  "last_scan_date": now|rint # builtin jq timestamp integer part
}]
' "${output_file}"
)

# Write the buffer content back to the file
# printf '%s\n' will preserve the buffer content
# and restores the trailing newline at end of file
printf '%s\n' "${json_buffer}" > "${output_file}"