Remove duplicate 2D dict values-CodePudding

I have the following list of dicts:

defaultdict(<class 'list'>,
            {'192.168.20.10/32': [{1: 'aaaaa11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'}],
             '192.168.20.20/32': [{1: 'aaaaa11111\n'},
                                  {2: 'aaaaa11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'}]})

I would like to iterate over this list object and remove the duplicate list items.

The end result would look something like this:

defaultdict(<class 'list'>,
                {'192.168.20.10/32': [{1: 'aaaaa11111\n'},
                                      {2: 'bbbbb11111\n'}],
                 '192.168.20.20/32': [{1: 'aaaaa11111\n'},
                                      {2: 'bbbbb11111\n'}]})

Please note that I am using defaultdict from collections above:

from collections import defaultdict

CodePudding user response：

Maybe something like this (if resulting order does not matter):

adict = {
    '192.168.20.10/32': [
        {1: 'aaaaa11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'}
    ],
    '192.168.20.20/32': [
        {1: 'aaaaa11111\n'},
        {2: 'aaaaa11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'}
    ]
}

for k, v in adict.items():
    adict[k] = list(map(dict, (set([tuple(obj.items()) for obj in v]))))
print(adict)

CodePudding user response：

If we call your dict first_dict, I've used the approach from https://stackoverflow.com/a/11092607/11574713. This is a not inplace though, I create a new dictionary new_dict.

Basically run the dicts through a set by converting them into a string (JSON), and then bring them back to dicts.

import json
from pprint import pprint

new_dict = dict()

for k in first_dict:
    new_dict[k] = [json.loads(j) for j in set([json.dumps(i) for i in first_dict[k]])]

pprint(new_dict)

{'192.168.20.10/32': [{'2': 'bbbbb11111\n'}, {'1': 'aaaaa11111\n'}],
 '192.168.20.20/32': [{'2': 'aaaaa11111\n'},
                      {'2': 'bbbbb11111\n'},
                      {'1': 'aaaaa11111\n'}]}