Python Remove Duplicates From list of tuples?-CodePudding

Please Note, question got fixed.

In python I have the following list

[('admin', 'admin', None), ('admin', 'admin', None), ('admin', 'admin', 1)]

How can I remove duplicates where I consider an item to be duplicate if it has the same 3 first tuples, so I'm expecting to get the following output:

[('admin', 'admin', None), ('admin', 'admin', 1)]

as we can find ('admin', 'admin', None) twice in the original input.

The provided and only answer fails on the following input:

[('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'})]

While I'm expecting it to return exactly same list

Same error for:

list(set([i for i in lst]))

CodePudding user response：

Try this:

result = []
first_timers = set()
for i in lst:
    if hash(str(i)) not in first_timers:
        first_timers.add(hash(str(i)))
        result.append(i)

CodePudding user response：

Your problem - which was not adequately captured by your simplified example of

[('admin', 'admin', None), ('admin', 'admin', None), ('admin', 'admin', 1)]

is that your tuple contains non-hash-able datatypes (aka: dict's / set's etc.).

You can transform your inner tuples to stringified versions of it, put them into a dict under that key with the original tuple as value - then use the dict.values() as outputs:

data = [('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'}),
    ('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'}),
    ('nonadmin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'})]

def strKey(whatever):
    return tuple(map(str, whatever))

d = { strKey(inner):inner for inner in data}

uniqified = list(d.values())
print(uniqified)

Output:

[('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'}),
 ('nonadmin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'})]