I have a pretty complicated JSON file that I encourage you to copy-paste here for better visualization.
{
"data":[
{
"title":"Title1",
"paragraphs":[
{
"context":"Context1",
"qas":[
{
"answers":[
{
"answer_start":515,
"text":"String text"
}
],
"question": "Why something something?",
"id":"5733be284776f41900661182"
},
{
"answers":[
{
"answer_start":505,
"text":"String something text"
}
],
"question": "Why?",
"id":"5733be284776f4190066345"
}
]
},
{
"context":"Context2",
"qas":[
{
"answers":[
{
"answer_start":515,
"text":"String text"
}
],
"question": "Why something something?",
"id":"5733be284776f41900661182"
},
{
"answers":[
{
"answer_start":505,
"text":"String something text"
}
],
"question": "Why?",
"id":"5733be284776f4190066345"
}
]
}
]
},
{
"title":"Title2",
"paragraphs":[
{
"context":"Context10",
"qas":[
{
"answers":[
{
"answer_start":585,
"text":"String text"
}
],
"question": "Why something something?",
"id":"5733be284776f41900661682"
},
{
"answers":[
{
"answer_start":545,
"text":"String something text"
}
],
"question": "Why?",
"id":"5733be284776f41900663"
}
]
},
{
"context":"Context7",
"qas":[
{
"answers":[
{
"answer_start":525,
"text":"String text"
}
],
"question": "Why something something?",
"id":"5733be284776f41982"
},
{
"answers":[
{
"answer_start":595,
"text":"String something text"
}
],
"question": "Why?",
"id":"5733be284776f419005"
}
]
}
]
}
],
"version":"1.1"
}
I don't need all this information and I want my data format to look simpler, like this:
[
{
"id": "5733be284776f41900661182",
"context": "Context1",
"question": "Why something something?",
"answers": {"text": ["String text"]}
},
{
"id": "5733be284776f4190066345",
"context": "Context1",
"question": "Why?",
"answers": {"text": ["String something text"]}
},
{
"id": "5733be284776f41900661182",
"context": "Context2",
"question": "Why something something?",
"answers": {"text": ["String text"]}
},
{
"id": "5733be284776f4190066345",
"context": "Context2",
"question": "Why?",
"answers": {"text": ["String something text"]}
},
{
"id": "5733be284776f41900661682",
"context": "Context10",
"question": "Why something something?",
"answers": {"text": ["String text"]}
},
{
"id": "5733be284776f41900663",
"context": "Context10",
"question": "Why?",
"answers": {"text": ["String something text"]}
},
{
"id": "5733be284776f41982",
"context": "Context7",
"question": "Why something something?",
"answers": {"text": ["String text"]}
},
{
"id": "5733be284776f419005",
"context": "Context7",
"question": "Why?",
"answers": {"text": ["String something text"]}
}
]
I don't care about the paragraphs, titles, answer start etc., and I would prefer it if the contexts were separated. Is there any way one can turn such a JSON into something much simpler? What I tried was to remove the unnecessary words in Python (replace them with " "
), but it made a mess out of it. I also tried to open it as a dataframe and remove the columns, but there are many nested things that I want removed too.
CodePudding user response:
You can write a simple Python script to translate this format:
import json
with open('file.json', 'r') as fh:
data = json.load(fh)
result = []
for article in data["data"]:
for paragraph in article["paragraphs"]:
for qa in paragraph["qas"]:
answers = {"text": [answer["text"] for answer in qa["answers"]]}
result.append({
"id": qa["id"],
"context": paragraph["context"],
"question": qa["question"],
"answers": answers
})
with open('output.json', 'w') as fh:
json.dump(result, fh)