Home > Software engineering >  Issue extracting specific data with Selenium
Issue extracting specific data with Selenium

Time:08-31

I am back with essentially a somewhat similar problem. I have now learnt that you cannot find elements that are in an iframe if you haven't switched to it, which helped a lot, but I seem to still have issues locating elements even though they are not in an iframe.

I also ask for any advice regarding my script in general, or how one would go about improving it. Yes, I will change the implicitwait to WebDriverWait, but besides that. Is it okay if a script is structured in this way, with task -> task -> task and so forth, or is it simply bad practice?

I don't really see how I would go about throwing in some objective-oriented programming, or what I would gain from it besides if I wanted to customise the script in a major way, besides of course the learning aspect.

In any case, here is the code:

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import accandpass as login
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import datetime

x = datetime.datetime.now()
x = x.strftime("%d")

driver = browser=webdriver.Firefox()
driver.get("https://connect.garmin.com/modern/activities")

driver.implicitly_wait(2)

iframe = driver.find_element(By.ID, "gauth-widget-frame-gauth-widget")
driver.switch_to.frame(iframe)

element = driver.find_element("name", "username")
element.send_keys(login.username)

element = driver.find_element("name", "password")
element.send_keys(login.password)

element.send_keys(Keys.RETURN)
driver.switch_to.default_content()

driver.implicitly_wait(10)

element = driver.find_element("name", "search")
element.send_keys("Reading")
element.send_keys(Keys.RETURN)

element = driver.find_element(By.CLASS_NAME, "unit")
print(element)

The data window I need to extract

So everything actually works fine so far, to my great surprise. The element gives off this: <selenium.webdriver.remote.webelement.WebElement (session="0ef84b2e-e0af-4b0c-b04c-94d5371356c5", element="a70f4ee1-e840-457c-a255-4b2df603efec")> which wasn't really what I was looking for.

I am more looking for some check, to see that the element with name unit has the same date as x, which is today. So basically:

Minutes read

minutes = 0 for i in element: if element == x: minutes = (element with time)

For loop to run through all the elements and check them all for the same date, and if the date matches then add the minutes read that day to the integer minutes for a sum of total minutes read today, for example.

Then do the same for the activities I will do, running, hiking and meditating.

Questions:

How do I get the integer from the element, so I can check it with x? Is the for loop -> if statement -> add time from element a good solution to the case at hand? Is it bad practice to structure a script this way, and how would you improve it?

Thanks in advance

CodePudding user response:

Part of your question sounds like you want a code review. If you do, you'll want to post your code over on https://codereview.stackexchange.com but review their question requirements carefully before posting.

I think the main issue you are asking about is wanting to compare the date on the page to the current system date and you're getting some session and element GUIDs instead. You are printing the element object and not the contained text. You want

print(element.text)

or add an assert to compare it to the current system date in a specific format, something like...

assert element.text == datetime.today().strftime('%m %d')

Some quick additional feedback since you asked for some...

  1. It sounds like you've already been informed that .implicitly_wait() is a bad practice and should be replaced with WebDriverWait for each instance where you need to wait.

  2. If you aren't going to reuse a variable, don't declare one. In most cases you don't need to use one.

    element = driver.find_element("name", "username")
    element.send_keys(login.username)
    

    can be written

    driver.find_element("name", "username").send_keys(login.username)
    
  3. If you are going to use variables, don't reuse the same name over and over, e.g. element. Give each variable a meaningful name so that the next person (or maybe yourself in a few weeks/months) will be able to more easily read and understand your code.

    element = driver.find_element("name", "search")
    element.send_keys("Reading")
    element.send_keys(Keys.RETURN)
    

    should instead be

    search = driver.find_element("name", "search")
    search.send_keys("Reading")
    search.send_keys(Keys.RETURN)
    
  4. If you are going to continue writing scripts, do some reading on the page object model. Done right, it will clean up your code significantly, make maintenance much faster and easier, and make writing new scripts much faster. Basically you create a class for each page of the site and then add methods to the class for actions you need to take on the page.

    I don't have a garmin account so I can't log in but from the screenshot you posted you might have some methods like .search_activities(string), .get_search_results(), .filter_activities(string), etc. Once you've created those methods, they can be called repeatedly from the same script or many scripts.

  • Related