I am using Selenium to crawl some Facebook group information:
with open("groups.txt") as file:
lines = file.readlines()
total = len(lines)
count = 1
for line in lines:
group_id = line.strip().split(".com/")[1]
if "groups" not in line:
new_line = "https://www.facebook.com/groups/" str(group_id) "/about"
else:
new_line = line.strip() '/about'
sleep(2)
driver.get(new_line)
page_source = driver.page_source
page_id = page_source.split('"groupID":"')[1].split('","')[0]
page_followers = page_source.split('<!-- --> total members')[0][-15:]
page_followers = str(page_followers.split('>')[1]).replace(',', '')
page_name = page_source.split("</title>")[0].split("<title>")[1]
df1.loc[len(df1)] = [line.strip(), 'https://www.facebook.com/' str(page_id), page_followers, page_name]
print(f"{count}/{total}", line.strip(), 'https://www.facebook.com/' str(page_id), page_followers)
count = 1
df1.to_csv("groups.csv", encoding='utf-8', index=False, header=False)
Facebook has updated something recently so this code fails to return number of group members.
These are the relevant lines:
page_followers = page_source.split('<!-- --> total members')[0][-15:]
page_followers = str(page_followers.split('>')[1]).replace(',', '')
Taking view-source:https://www.facebook.com/groups/764385144252353/about as an example, I find two instances of "total members". Is it possible to get some advice on what I should change to be able to catch this number?
CodePudding user response:
NEW
This code extract the exact number of members and convert it from string to integer
driver.get('https://www.facebook.com/groups/410943193806268/about')
members = driver.find_element(By.XPATH, "//span[contains(text(), 'total members')]").text
members = int(''.join(i for i in members if i.isdigit()))
print(members)
output
15589
OLD
I suggest don't use page_source
to extract this kind of data, instead use find_element
in this way
driver.find_element(By.CSS_SELECTOR, "a[href*='members']").text.split()[0]
output
'186'
Explanation: a[href*='members']
search for a
elements (for example <a class='test'>...</a>
) having a href
attribute containing the string members
(for example <a href="something-members-test">...</a>
)