python: Error while converting bytes array to json format-CodePudding

I am trying to convert bytes array to json by following the link

Below is the code:

command = "my command"
    my_bytes_value = self.subprocess_cmd(command)
    print("OUTPUT----")
    print(my_bytes_value)
    my_json = my_bytes_value.decode('utf8').replace("'", '"')
    print('- ' * 20)

    # Load the JSON to a Python list & dump it back out as formatted JSON
    #milestone 1
    data = json.loads(my_json)
    s = json.dumps(data, indent=4, sort_keys=True)
    print(s)

I am getting below error, also some print statement is executing

    File "1.py", line 50, in getlogs
    data = json.loads(my_json)
  File "/usr/local/Cellar/[email protected]/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/[email protected]/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/[email protected]/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
OUTPUT----
b'signing string: (request-target): post /v2/search\\ndate: Sat, 04 Jun 2022 05:41:48 GMT\\nhost: example.com\\nx-content-sha256: askhKJADKJ/Bdm sdlfkhlkh1376/m6ti5fo=\\ncontent-type: application/json\\ncontent-length: 342\n[ {\n  "timestamp" : "2022-06-04T05:16:50.522Z",\n  "segmentPartitionId" : null,\n  "component" : null,\n  "region" : null,\n  "availabilityDomain" : null,\n  "filename" : null,\n  "casperUrl" : null,\n  "host" : null,\n  "source" : "EventStats: \\n Some message String\\nSome message String\\n",\n  "fullSource" : "{\\"cursor\\":-2002349868963,\\"bucket\\":\\"xyz\\",\\"kafkaAd\\":\\"gr-10\\",\\"kafkaOffset\\":239846986,\\"kafkaPartition\\":15,\\"kafkaRecord\\":273,\\"kafkaTopic\\":\\"logs\\",\\"lineOffset\\":2343,\\"logGroup\\":\\"dev_log\\",\\"logStreamId\\":\\"xdfklhklsdhfkl\\",\\"logType\\":\\"STANDARD\\",\\"namespace\\":\\"dev-api\\",\\"rawSource\\":\\"{\\\\\\"logpersisting\\\\\\":\\\\\\"some_logs\\\\\\",\\\\\\"level\\\\\\":\\\\\\"INFO\\\\\\",\\\\\\"logger\\\\\\":\\\\\\"com.example.CollectorTaskHelper\\\\\\",\\\\\\"msg\\\\\\":\\\\\\"LogStats: \\\\\\\\n Some message String\\\\\\\\nSome message String\\\\\\\\n\\\\\\",\\\\\\"ts\\\\\\":1243454354,\\\\\\"workflowid\\\\\\":\\\\\\"12345\\\\\\",\\\\\\"workflowstepname\\\\\\":\\\\\\"LOG_ALERTS\\\\\\"}\\",\\"resource\\":\\"askjgjf\\",\\"tenant\\":\\"xyz\\",\\"ts\\":123214234,\\"tsIndexed\\":239486986,\\"tsIngested\\":324868623,\\"#eventpersisting\\":\\"exadata_events\\",\\"#level\\":\\"INFO\\",\\"#logger\\":\\"com.example\\",\\"#thread\\":\\"flow-Processor-Thread-137\\",\\"#workflowid\\":\\"8kjhsdf76784378\\",\\"#flowname\\":\\"CollectorWorkflowV2\\",\\"#flownodename\\":\\"example.com\\",\\"#workflowstepname\\":\\"LOG_ALERTS\\"}"\n} ]'


- - - - - - - - - - - - - - - - - - - - 

Process finished with exit code 1

So, upto milestone it is working fine but not working after that. Looks like some character is spoiling here. but not able to identify it.

CodePudding user response：

The reason why it isn't working in your case is because your string isn't properly formatted json even after making the adjustments you found in that other post.

There is a some json in the string though, so depending on if you really need the full part of the string or not your could do this.

command = "my command"
    my_bytes_value = self.subprocess_cmd(command)
    print("OUTPUT----")
    print(my_bytes_value)
    print('- ' * 20)

    # Load the JSON to a Python list & dump it back out as formatted JSON
    #milestone 1
    text = my_bytes_value.decode('utf-8')
    my_json = json.loads(text[text.index("[") :])
    my_json[0]["fullSource"] = json.loads(my_json[0]["fullSource"])
    my_json[0]["fullSource"]["rawSource"] = json.loads(my_json[0]["fullSource"]["rawSource"])
    print(my_json)
    json.dumps(my_json, indent=4, sort_keys=True)

CodePudding user response：

So, I found after converting from byte to string, first line is not part of json string, so i removed it. After that I am able to convert it into json

command = "my command"
my_bytes_value = self.subprocess_cmd(command)
print("OUTPUT----")
print(my_bytes_value)
list1 = my_bytes_value.decode('utf8').replace("'", '"').split('\n')[1:]
my_json = ''.join(list1)
print('- ' * 20)

# Load the JSON to a Python list & dump it back out as formatted JSON
#milestone 1
data = json.loads(my_json)
s = json.dumps(data, indent=4, sort_keys=True)
print(s)