How do I scrape data that is nested using selenium and Python-CodePudding

I am sitting with a project for my masters, where I would like to scrape LinkedIn. As far as I am now, I ran into a problem when I want to scrape the education pages of users (eg. https://www.linkedin.com/in/williamhgates/details/education/)

I would like to scrape all the educations of the users. In this example I would like to scrape "Harvard University" under mr1 hoverable-link-text t-bold, but I can't see to get to it.

Here's the HTML at code from Linkedin:

<li  id="profilePagedListComponent-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-EDUCATION-VIEW-DETAILS-profile-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-NONE-da-DK-0">
                        <!----><div >
  <div>
        <a  target="_self" href="https://www.linkedin.com/company/1646/">
        <div >
    <div >
<!---->      <img width="48" src="https://media-exp1.licdn.com/dms/image/C4E0BAQF5t62bcL0e9g/company-logo_100_100/0/1519855919126?e=1668643200&amp;v=beta&amp;t=BL0HxGNOasVbI3u39HBSL3n7H-yYADkJsqS3vafg-Ak" loading="lazy" height="48" alt="Harvard University logo" id="ember59" >
</div>
  </div>
    </a>

  </div>

  <div >
    <div >
          <a  target="_self" href="https://www.linkedin.com/company/1646/">
        <div >
            <span >
              <span aria-hidden="true"><!---->Harvard University<!----></span><span ><!---->Harvard University<!----></span>
            </span>
<!----><!----><!---->        </div>
<!---->          <span >
            <span aria-hidden="true"><!---->1973 - 1975<!----></span><span ><!---->1973 - 1975<!----></span>
          </span>
<!---->      </a>


<!---->
      <div >
<!---->      </div>
    </div>

      <div >
<!---->    <ul >
        <li >
                <div >
<!----><!----><!----></div>

        </li>
    </ul>
<!----></div>
  </div>
</div>

                </li>

I have tried the following code:

education = driver.find_element("xpath", '//*[@id="profilePagedListComponent-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-EDUCATION-VIEW-DETAILS-profile-ACoAAA8BYqEBCGLg-vT-ca6mMEqkpp9nVffJ3hc-NONE-da-DK-0"]/div/div[2]/div[1]/a/div/span/span[1]/').text

print(education)

I keep getting the error:

Message: no such element: Unable to locate element:

Can anybody help? I would love to have a script that loops through the educations, and save place of education and the year of educations.

CodePudding user response：

I would first get the list for the education section.

education_list = driver.find_element(By.CSS_SELECTOR, 'ul.pvs-list')
# loop through education_list for place and years
# would recommend relative locators for this task.
# find the image and get the first and second span with text inside of them.

I am adding further details to the code now. Please hold.

CodePudding user response：

You can use below properties to identify the school name list:

ancestorClass="optional-action-target-wrapper display-flex flex-column full-width"  tag="DIV"

Use these properties to identify the year list:

ancestorClass="optional-action-target-wrapper display-flex flex-column full-width"  tag="SPAN"

You may use above info to compose an XPath to locate the list, or if you don't mind using other python libraries, there is a sample code in GitHub to scrape the school and year.

CodePudding user response：

@Nadia S. you can try the following code. I have provided comments inline inside the code.

    @Test
    public void linkedInTest() {
        driver.get("https://www.linkedin.com");

        // You need to enter the credentials for your linkedin below for login
        driver.findElement(By.id("session_key")).sendKeys("");
        driver.findElement(By.id("session_password")).sendKeys("");
        driver.findElement(By.className("sign-in-form__submit-button")).click();
        driver.get("https://www.linkedin.com/in/williamhgates/details/education/");

        //Wait for the Education details to get populated. 
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(7));
        wait.until(ExpectedConditions.visibilityOfElementLocated(
                By.xpath("//div[@class = 'pvs-list__container']//div[@class = 'scaffold-finite-scroll__content']/ul")));
        
        //Take all elements showing education details in a list 
        List<WebElement> allEducation = driver.findElements(By
                .xpath("//div[@class = 'pvs-list__container']//div[@class = 'scaffold-finite-scroll__content']/ul/li"));
        //Extract details of each education item in the list. 
        //Below the details are directed to console. You can use a collection to store them.
        for (WebElement oneEducation : allEducation) {
            WebElement education = oneEducation.findElement(
                    By.xpath(".//*[contains(@class,\"mr1 hoverable-link-text\")]/span[@aria-hidden='true']"));
            System.out.print("Education - "   education.getText());
            try {
                WebElement educationType = oneEducation
                        .findElement(By.cssSelector(".t-14.t-normal span[aria-hidden='true']"));
                System.out.print("      Education Type - "   educationType.getText());
            } catch (NoSuchElementException e) {
                System.out.print("      Education Type - "   "is Not Specified");
            }
            try {
                WebElement educationYear = oneEducation
                        .findElement(By.cssSelector(".t-14.t-normal.t-black--light span[aria-hidden='true']"));
                System.out.println("        Education Year - "   educationYear.getText());
            } catch (NoSuchElementException e) {
                System.out.println("        Education Year - "   "is Not Specified");
            }
        }

    }