Find keyword from a list in a page using BeautifulSoup-CodePudding

Using Beautiful Soup, I'd like to detect porn keywords (that i get by concatening two lists of porn-keywords (one in french, the other in english) in a web page.

Here's my code (from BeautifulSoup find two different strings):

proxy_support = urllib.request.ProxyHandler(my_proxies)
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)
lst_porn_keyword_eng = str(urllib.request.urlopen("http://www.cs.cmu.edu/~biglou/resources/bad-words.txt").read()).split('\\n')
# the textfile starts with a LF, deleting it.
if lst_porn_keyword_eng[0] == "b\"":
   del lst_porn_keyword_eng[0]
lst_porn_keyword_fr = str(urllib.request.urlopen("https://raw.githubusercontent.com/darwiin/french-badwords-list/master/list.txt").read()).split('\\n')

lst_porn_keyword = lst_porn_keyword_eng   lst_porn_keyword_fr
lst_porn_keyword_found = []

with urllib.request.urlopen("http://www.example.com") as page_to_check:
     soup = BeautifulSoup(page_to_check, "html5lib")
     for node in soup.find_all(text=lambda text: any(x in text for x in lst_porn_keyword)):
          lst_porn_keyword_found.append(str(node.text))

return lst_porn_keyword_found

This code runs correctly but porn keyword are found even if they shouldn't be. For instance, the text of the second node found in "http://www.example.com" is This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission. And none of these words are in lst_porn_keyword

CodePudding user response：

Your soup.find_all() doesn't return the html but the css instead:

    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }

The words "color", "gin", and the character " appear in lst_porn_keyword and on the css, which triggered your detection.

Partial words like "gin" in "margin" are also problematic using soup.findall(), consider using regular expressions with word delimiters like the example below:

import regex as re

for word in lst_porn_keyword:
    result = re.findall(fr"\W{word}\W", node)
    if len(result) > 0:
        print(f"detected in text: {word}")

CodePudding user response：

I replaced your lambda function with

def testfn(text):
    elms = list([x for x in lst_porn_keyword if x in text])
    if len(elms) > 0:
        print(f"found words {elms} in {text}")
    return len(elms)>0

calling soup.find_all(text=testfn) will result in the following output:

found words ['color', 'gin', '"'] in `
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }
    `
found words ['cum', 'ho'] in `This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.`

I think your problem is that the in keyword also works for partial words. E.g.:

"cum" in "document"
> True