delete duplicates items from a list-CodePudding

i have the following list

list1=['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product',
 'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']

I am trying to delete the duplicates:

mylist = sorted(set(list1))
mylist

['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product',
 'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']

As you can see this does not happen and the initial is still preserved.

Any idea why this happens?

CodePudding user response：

Your problem is that the list1 contains just one element. Creating a set will not have any obvious effect because there are no duplicates. I think this is what you want to achieve:

list(set(list1[0].split(', ')))

If the list has more than one element, you may want to do this:

s = set()

for e in list1:
    for word in e.split(', '):
        s.add(word)

print(list(s))

CodePudding user response：

Both of the strings in your array are not equal. If you want to remove duplicates from inside the strings, you must split first. Try doing:

list1 = [
    'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, '
    'update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, '
    'and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, '
    'know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, '
    'the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, '
    'appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, '
    'our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, '
    'accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, '
    'to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, '
    'validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, '
    'operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, '
    'or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, '
    'product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, '
    'customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, '
    'philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, '
    'question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, '
    'product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, '
    'product',
    'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, '
    'cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, '
    'software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, '
    'an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, '
    'cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, '
    'product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, '
    'to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, '
    'advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, '
    'regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, '
    'specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, '
    'configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, '
    'product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, '
    'product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, '
    'by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, '
    'produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, '
    'the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, '
    'encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, '
    'customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, '
    'team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, '
    'their, philips, product']
list2 = []
for i in list1:
    list2  = i.split(', ')
print("Length of list before removing duplicated =", len(list2))
print("Length of list before after duplicated =", len(set(list2)))

Output is:

CodePudding user response：

filtered_list = list(set(word for phrase in list1 for word in phrase.split(', ')))