i'm scraping urban dictionary

here's my code :

query=input("Enter A Word : ")
data=requests.post(f"https://urbandictionary.com/define.php?term={query}").content
soup=BeautifulSoup(data,"html.parser")
for i in soup.find_all("div",class_="meaning"):
   print(i)

and i'm getting :

<div class="meaning">A person who writes <a class="autolink" href="/define.php?term=funny" onclick="ga('send', 'event', 'Autolink', 'Click', &quot;funny&quot;);">funny</a> and motivational <a class="autolink" href="/define.php?term=posts" onclick="ga('send', 'event', 'Autolink', 'Click', &quot;posts&quot;);">posts</a> on <a class="autolink" href="/define.php?term=Facebook" onclick="ga('send', 'event', 'Autolink', 'Click', &quot;Facebook&quot;);">Facebook</a></div>

how to get inner html of div and anchor tags?

CodePudding user response：

Try this, working for me!

import requests
from bs4 import BeautifulSoup

query=input("Enter A Word : ")
data=requests.post(f"https://urbandictionary.com/define.php?term={query}").content

soup=BeautifulSoup(data,"html.parser")
for div in soup.find_all("div",class_="meaning"):
   inner_html_of_div = "".join([str(x) for x in div.contents])
   print(inner_html_of_div)
   
   anchors = div.find_all("a")
   for anchor in anchors:
       inner_html_of_anchor = "".join([str(x) for x in anchor.contents])
       print(inner_html_of_anchor)

CodePudding user response：

You can use .text for each <div> in your for-loop.

import requests
from bs4 import BeautifulSoup

headers = {"User-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"}
query = input("Enter A Word : ")
data = requests.post(f"https://urbandictionary.com/define.php?term={query}").content
soup = BeautifulSoup(data,"html.parser")
for i in soup.find_all("div",class_="meaning"):
   print(i.text.strip()   '\n')

Enter A Word : mettle
Ability to persevere despite obstacles...usually performed with class or grace.

the ability or driving force of an individual to persevere when they think they can't go any farther.

When you mix skittle and m&ms together and brain gets completely mindfucked because you don't know what you're eating.

Mettle is a person's ability to cope and persevere.Metal fatigue is when metal is stressed, cracks, and breaks-- sometimes with tragic consequences.
So when someone's mettle is exhausted, leading to personal breakdown, we can call it mettle fatigue.

Mettle means your ability to cope and persevere, so when your coping skills are exhausted, this is known as mettle fatigue.

A well balanced combination of the Valve FPS game: Counter Strike: Global Offensive's weapon case system and skins system, and the weapons of the Valve FPS game: Team Foretess 2. This update to the game has been thought to try to help Team Fortress's trading appeal to some Counter Strike: Global Offensive traders. Adding 3 new maps and finishing a promised one (the snowplow was a lie!) Snowplow, Borneo, Powerhouse, and Suijun this gave Team Fortress 2 players a little something to chew on.