A follow up from my previous question posted here Selenium Python can only find the first element. The answer in that post was very helpful and fetches me some of the text content from https://www.thestudentroom.co.uk/showthread.php?t=7263973. The problem is that it scrolls to the bottom instantly and as such only returns the first post in the forum and the last 3 replies. Is there any way of going down the page and fetching all of the replies as I go? I believe the issue is that they are lazyload elements.
The current code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://www.thestudentroom.co.uk/showthread.php?t=7263973"
driver.get(url)
wait.until(EC.presence_of_element_located((By.XPATH, "//div[contains(@class,'styles__PostContent')]/span")))
last_reply = wait.until(EC.presence_of_element_located((By.XPATH, "(//button[contains(.,'reply')])[last()]")))
last_reply.location_once_scrolled_into_view
posts = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'styles__PostContent')]/span")))
for post in posts:
print(post.text)
This works well but currently skips most of the replies in that post, fetching the initial post and replies, 9, 10 & 11 only.
Any help on this would be greatly appreciated.
CodePudding user response：
Try this code:
url = "https://www.thestudentroom.co.uk/showthread.php?t=7263973"
driver.get(url)
while True:
driver.execute_script("window.scrollBy(0, 400);")
time.sleep(3)
try:
if driver.find_element(By.XPATH, ".//p[text()='Contact us']").is_displayed():
break
except NoSuchElementException:
driver.execute_script("window.scrollBy(0, 400);")
posts = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'styles__PostContent')]/span")))
for post in posts:
print(post.text)
print("------")
Output:
Welcome to GYG 2022-23
Click here to make a blog
What is GYG?
Grow your Grades is a blogging competition that has run on TSR for 8 years now. Each year we ask students from all levels to blog about their journey from wherever they start at the start of the academic year to the end, sharing the highs, the lows and everything in-between. At the end of the year, we read them through and pick an overall winner among other prizes that we give out throughout the year, it pays to Grow Your Grades!
So, why should you enter?
Keeping a blog helps keep yourself motivated and accountable for your studying, plus the community spirit in GYG is always so encouraging and motivating for people! We know it can be difficult to get back into studying, but you can use GYG to challenge yourself and get back into the rhythm of studying
What else do I need to know?
We'll be offering more regular rep prizes, for everything from a really great blog post to being supportive in the forum on top of the usual spot prizes. Finally, we know how valuable your advice is for other students, so we will be offering prizes for your advice throughout the year. Not only will you see your advice help other students in our articles, there will also be prizes to be won. If you have any other ideas for GYG then please post them, we love to hear them!
I want to start a blog, but I don't know how to make it look pretty.
Firstly, to be in a chance of winning, then your blog doesn't have to be pretty! But if you want it to be so, then look at our guide here on how to work the new TSR editor, this will help you edit your posts to perfection
Do my blog posts have to be a certain length?
No This year in particular, we're aware students may have less time than normal, so we're encouraging you to keep your blogs short and sweet if that means you have more time for the slightly more important stuff like homework and school Or even chilling. You may like to consider writing a few sentences each day on the way home from school rather than a bigger post at the weekend, or adding a picture to show your to do list rather than typing it all out again. It's about how you use the words, not the number you use!
What should I include in my blog?
Whatever you like! Your subjects is always a good start, and what subject level, but after that you can include anything from your timetable, homework struggles, pictures of your revision (or your snacks). Anything relevant to growing your grades. Remember you don't have to have super long blog posts, and a picture can tell a thousand words (and save you time too!)
What kinds of prizes are there?
The main prizes are yet to be confirmed for 2022 , but we can confirm we will be having some extra rep prizes for categories such as "Best title", "Best blog from a newbie" and "Best use of photos in a blog". We'll also be running our usual prize for "Best supporter in GYG", and have a special bonus to this of a £10 amazon voucher for the person most supportive before October half term As usual, we may run spot prizes, and also may be awarding random prizes throughout the year if we spot someone being super helpful, making great blog posts or for any other reason. We'll announce these on this thread so everyone can see
When does the competition close?
We'll be closing the competition near the end of September this year. If you have any other questions, then do say below, we're always happy to answer
------
Hi, would it be possible to move my new GYG into this year's forum.
https://www.thestudentroom.co.uk/showthread.php?t=7252577
Thanks
------
Im glad to see the 2022-2023 GYG is now open
I will be making mine at the weekend at some point, hopefully to track my progress and keep on top of everything
------
Squiggles1238
Im glad to see the 2022-2023 GYG is now open
I will be making mine at the weekend at some point, hopefully to track my progress and keep on top of everything
SEE MORE
That's great to hear, we are glad to have you!
------
Where do users make GYG forums?
------
sciencegcsesss
Where do users make GYG forums?
You make a thread here: https://www.thestudentroom.co.uk/forumdisplay.php?f=1631
------
Sinnoh
You make a thread here: https://www.thestudentroom.co.uk/forumdisplay.php?f=1631
Cheers
------
how can i start my own blog?
------
k.patel18
how can i start my own blog?
Start a new thread with "start a new discussion" in this forum: https://www.thestudentroom.co.uk/forumdisplay.php?f=1631
------
So excited for this!
A few weeks late to the party, but looking forward to actually completing a blog this year - especially since it's my last one of undergrad!
------
do we post another new blog every time we want to share or do we add it on our current blog?
------
Do we start our own blogs for this or just reply in this thread?
------