Pagination in selenium-CodePudding

I'm new to selenium and following a small

If this is not the correct page, please share the URL and the element, I will try to provide a solution.

CodePudding user response：

The website indeed.com have changed since the article Web Scraping using selenium and Java was written. The pagination elements have changed. Currently the pagination is implemented through:

<nav role="navigation" aria-label="pagination" >
    <div >
        <button data-testid="pagination-page-current" >1</button>
    </div>
    <div >
        <a data-testid="pagination-page-2" rel="nofollow" aria-label="2" href="/jobs?q=Api Testing&amp;l=Pune&amp;start=10" >2</a>
    </div>
    <div >
        <a data-testid="pagination-page-3" rel="nofollow" aria-label="3" href="/jobs?q=Api Testing&amp;l=Pune&amp;start=20" >3</a>
    </div>
    <div >
        <a data-testid="pagination-page-4" rel="nofollow" aria-label="4" href="/jobs?q=Api Testing&amp;l=Pune&amp;start=30" >4</a>
    </div>
    <div >
        <a data-testid="pagination-page-5" rel="nofollow" aria-label="5" href="/jobs?q=Api Testing&amp;l=Pune&amp;start=40" >5</a>
    </div>
    <div >
        <a data-testid="pagination-page-next" rel="nofollow" aria-label="Next Page" href="/jobs?q=Api Testing&amp;l=Pune&amp;start=10" >
            <svg xmlns="http://www.w3.org/2000/svg" focusable="false" role="img" fill="currentColor" viewBox="0 0 24 24" aria-hidden="true" >
                <path d="M9.888 5.998a.506.506 0 00-.716-.008l-.707.707a.506.506 0 00.01.716L13.06 12l-4.587 4.587c-.2.2-.204.521-.009.716l.707.707a.507.507 0 00.717-.009l5.647-5.648c.1-.1.148-.233.144-.366a.492.492 0 00-.144-.34v-.001a.611.611 0 00-.009-.009L9.888 5.998z">
                </path>
            </svg>
        </a>
    </div>
</nav>

Solution

Instead of creating a list of the pagination elements, as the pagination keeps on increasing as you move forward you can use the following solution:

System.setProperty("webdriver.chrome.driver", "C:\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
ChromeDriver driver = new ChromeDriver(options);
driver.get("https://in.indeed.com/");
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(5));
// Create a Scanner object to prompt for user input
Scanner myObj = new Scanner(System.in);
System.out.println("What jobs are you looking for ? ");
String job = myObj.nextLine();
System.out.println("Which city are you looking for ? ");
String city = myObj.nextLine();
wait.until(ExpectedConditions.elementToBeClickable(By.cssSelector("input#text-input-what"))).sendKeys(job);
driver.findElement(By.cssSelector("input#text-input-where")).sendKeys(city   Keys.RETURN);
for (int j=1; j<6; j  )
{
    driver.findElement(By.xpath("//button[@data-testid='pagination-page-current']//following::div[1]/a[@data-testid]")).click();
    Thread.sleep(15000);
}

Note:

Here 6 is the number of (pages-1) you want to scrape.
Thread.sleep(15000) is to provide the window to pulate Email Address after the first click on pagination item.