Dedeplicating JSON objects based on 1st key:value pair-CodePudding

This is the output format, and based on "CVE_data_meta" I need to deduplicate matching IDs.

#pull references
for ref in item["cve"]["references"]["reference_data"]:
    references = ref["url"]
    cleanData.append({"CVE_data_meta": cve_data_meta_id,
                     "description": description,
                     "baseScore": baseScore,
                     "vectorSring": vectorString,
                     "cweID": cweValue,
                     "cweID URL": ("https://cwe.mitre.org/data/definitions/"
                                      str(cweValue)   ".html"),
                     "references": references,
                     "publishedDate": pub_date,
                     "lastModifiedDate": last_mod_date
                     })

This is the iteration where I pull data from a cleaned up response from an API and output to JSON file:

# # ==========================================================================================
# # narrow response with additional 'keywords'
# # ==========================================================================================
myResults = open("2-cleanData.json", "r")
scope = json.load(myResults)
output_json=[]
results = []
for k in keywords:
    counter = 0
    items = [x for x in scope if k in x['description']]
    for item in items:
        output_json.append(item)
        counter  = 1
    results.append(counter)
with open("3-Final CVEs.json", "w ") as outFile2:
    outFile2.write(json.dumps(output_json, indent=2,))

The keywords variable is changeable by user; but want any to be able to add keywords and not get duplicate entries in output file.

Full code here.

Example Output: (3 CVE entries)

{
  "CVE_data_meta": "CVE-2021-0924",
  "description": "In xhci_vendor_get_ops of xhci.c, there is a possible out of bounds read due to a missing bounds check. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android kernelAndroid ID: A-194461020References: Upstream kernel",
  "baseScore": 7.8,
  "vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
  "cweID": "CWE-125",
  "cweID URL": "https://cwe.mitre.org/data/definitions/CWE-125.html",
  "references": "https://source.android.com/security/bulletin/2021-11-01",
  "publishedDate": "2021-12-15T19:15Z",
  "lastModifiedDate": "2021-12-17T18:12Z"
},
{
  "CVE_data_meta": "CVE-2021-0981",
  "description": "In enqueueNotificationInternal of NotificationManagerService.java, there is a possible way to run a foreground service without showing a notification due to improper input validation. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android-12Android ID: A-191981182",
  "baseScore": 7.8,
  "vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
  "cweID": "CWE-269",
  "cweID URL": "https://cwe.mitre.org/data/definitions/CWE-269.html",
  "references": "https://source.android.com/security/bulletin/pixel/2021-12-01",
  "publishedDate": "2021-12-15T19:15Z",
  "lastModifiedDate": "2021-12-17T18:09Z"

...several entries later...

  "CVE_data_meta": "CVE-2021-0924",
  "description": "In xhci_vendor_get_ops of xhci.c, there is a possible out of bounds read due to a missing bounds check. This could lead to local escalation of privilege with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android kernelAndroid ID: A-194461020References: Upstream kernel",
  "baseScore": 7.8,
  "vectorSring": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
  "cweID": "CWE-125",
  "cweID URL": "https://cwe.mitre.org/data/definitions/CWE-125.html",
  "references": "https://source.android.com/security/bulletin/2021-11-01",
  "publishedDate": "2021-12-15T19:15Z",
  "lastModifiedDate": "2021-12-17T18:12Z"
},

Now, just need to git rid of duplicates...

CodePudding user response：

After reviewing your code, I believe you can do something like this to avoid repeated dictionaries:

results = []
cve_ids = []
for k in keywords:
    counter = 0
    items = [x for x in scope if k in x['description']]
    for item in items if item['cweID'] not in cwe_ids:
        output_json.append(item)
        cwe_ids.append(item['cweID'])
        counter  = 1

CodePudding user response：

You can easily deduplicate the results by using a set to keep track of the 'CVE_data_meta' entries already seen and skipping entries that have already been seen as shown below below. set membership testing is extremely fast, so this will be fast.

Tested with this limited test data:

myResults = [
 {'CVE_data_meta': 'CVE-2021-0924',
  'description': 'In xhci_vendor_get_ops of xhci.c, there is a possible out of '
                 'bounds read due to a missing bounds check. This could lead '
                 'to local escalation of privilege with no additional '
                 'execution privileges needed. User interaction is not needed '
                 'for exploitation.Product: AndroidVersions: Android '
                 'kernelAndroid ID: A-194461020References: Upstream kernel',
  'baseScore': 7.8,
  'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
  'cweID': 'CWE-125',
  'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-125.html',
  'references': 'https://source.android.com/security/bulletin/2021-11-01',
  'publishedDate': '2021-12-15T19:15Z',
  'lastModifiedDate': '2021-12-17T18:12Z'},
 {'CVE_data_meta': 'CVE-2021-0981',
  'description': 'In enqueueNotificationInternal of '
                 'NotificationManagerService.java, there is a possible way to '
                 'run a foreground service without showing a notification due '
                 'to improper input validation. This could lead to local '
                 'escalation of privilege with no additional execution '
                 'privileges needed. User interaction is not needed for '
                 'exploitation.Product: AndroidVersions: Android-12Android ID: '
                 'A-191981182',
  'baseScore': 7.8,
  'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
  'cweID': 'CWE-269',
  'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-269.html',
  'references': 'https://source.android.com/security/bulletin/pixel/2021-12-01',
  'publishedDate': '2021-12-15T19:15Z',
  'lastModifiedDate': '2021-12-17T18:09Z'},
 {'CVE_data_meta': 'CVE-2021-0924',
  'description': 'In xhci_vendor_get_ops of xhci.c, there is a possible out of '
                 'bounds read due to a missing bounds check. This could lead '
                 'to local escalation of privilege with no additional '
                 'execution privileges needed. User interaction is not needed '
                 'for exploitation.Product: AndroidVersions: Android '
                 'kernelAndroid ID: A-194461020References: Upstream kernel',
  'baseScore': 7.8,
  'vectorSring': 'CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H',
  'cweID': 'CWE-125',
  'cweID URL': 'https://cwe.mitre.org/data/definitions/CWE-125.html',
  'references': 'https://source.android.com/security/bulletin/2021-11-01',
  'publishedDate': '2021-12-15T19:15Z',
  'lastModifiedDate': '2021-12-17T18:12Z'}
]

Code:

from pprint import pprint

# Deduplicate results
cleaned = []
seen = set()
for obj in myResults:
    key = obj['CVE_data_meta']
    if key not in seen:
        cleaned.append(obj)
        seen.add(key)

pprint(cleaned)