I am using the following code to extract text from a web page:

from bs4.element import Comment
import urllib.request

def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]',]:
        return False
    if isinstance(element, Comment):
        return False
    return True

def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(text=True)
    visible_texts = filter(tag_visible, texts)  
    return u" ".join(t.strip() for t in visible_texts)
    #return ' '.join(texts)
html = urllib.request.urlopen('https://ordoabchao.ca/volume-one/babylon').read()
text = text_from_html(html)

The problem is, when I open text, I get all the links from the bottoms that exist at the top of the page, which I don't want. How can i modify the above code to do so?

I also gets the footnotes, which i may want, but a separate text. Is there a way to separate the footsnotes from the main text?

Thanks

CodePudding user response：

If you want to extract all the text then you can use get_text() method

from bs4 import BeautifulSoup 
import requests
url = 'https://ordoabchao.ca/volume-one/babylon'
res = requests.get(url) 
soup = BeautifulSoup(res.text, 'lxml')

for p in soup.select('.sqs-block-content p'):
    print(p.get_text(strip=True))