How can I remove duplicate elements in a JSON file?-CodePudding

I trying to write a program that will output information to me in the form of a JSON file, but I ran into the fact that I duplicate information with the "exchange_id" parameter, but with other info. I want information with this value to be displayed to me once. How can I do this?

Example of JSON file:

{
            "exchange_name": "Huobi Global",
            "market_url": "https://www.huobi.com/en-us/exchange/eth_btc",
            "price": 48637.97781883441,
            "last_update": "2021-12-09T17:11:53.000Z",
            "exchange_id": 102
        },
{
            "exchange_name": "Huobi Global",
            "market_url": "https://www.huobi.com/en-us/exchange/xrp_btc",
            "price": 48656.10607332287,
            "last_update": "2021-12-09T17:10:53.000Z",
            "exchange_id": 102
        },

My code:

json_data = response['data']
            pairs = []
            out_object = {}

            exchangeIds = {102,311,200,302,521,433,482,406,42,400}

            for pair in json_data["marketPairs"]:
                if pair['exchangeId'] in set(exchangeIds):
                    pairs.append({

                        "exchange_name": pair["exchangeName"],
                        "market_url": pair["marketUrl"],
                        "price": pair["price"],
                        "last_update" : pair["lastUpdated"],
                        "exchange_id": pair["exchangeId"]

                        })

                    out_object["name_of_coin"] = json_data["name"]
                    out_object["marketPairs"] = pairs
                    out_object["pairs"] = json_data["numMarketPairs"]
 
                    name = json_data["name"]

                    save_path = '.\swapMarketCap\coins'
                    file_name = f'{name}'
                    path_to_coin = os.path.join(save_path, file_name  ".json")

CodePudding user response：

In the absence of more detail, here's a pattern that you could use. This does not purport to be rewrite of your code and also makes certain assumptions (in the absence of detail in the original question). However, it should help you:

response = {'data':
            {'marketPairs': [{
                "exchange_name": "Huobi Global",
                "market_url": "https://www.huobi.com/en-us/exchange/eth_btc",
                "price": 48637.97781883441,
                "last_update": "2021-12-09T17:11:53.000Z",
                "exchange_id": 102
            },
                {
                "exchange_name": "Huobi Global",
                "market_url": "https://www.huobi.com/en-us/exchange/xrp_btc",
                "price": 48656.10607332287,
                "last_update": "2021-12-09T17:10:53.000Z",
                "exchange_id": 102
            }
            ]}}

exchangeIds = {102, 311, 200, 302, 521, 433, 482, 406, 42, 400}

output = {'data': {'marketPairs': []}}

for d in response['data']['marketPairs']:
    if (id := d.get('exchange_id', None)):
        if id in exchangeIds:
            output['data']['marketPairs'].append(d)
            exchangeIds.remove(id)

print(output)

So, the underlying idea is that you have a set of IDs that you're interested in. Once you find one that's in the set, take the reference to the current dictionary and add it to the new dictionary (output). Then, crucially, remove the previously processed ID from the set.