I'm new to this, I want to take html from the url. I decided to try to use Selenium and sketched a service that looks like this:
package dataox.keithgellmanrestaurantmenuscraping;
import lombok.RequiredArgsConstructor;
import lombok.extern.log4j.Log4j2;
import org.apache.commons.lang3.RandomUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Service;
import java.util.List;
@Log4j2
@Service
@RequiredArgsConstructor
public class ParserService {
private final ChromeOptions options = new ChromeOptions();
@EventListener(ApplicationReadyEvent.class)
public void a() {
String url = "some url";
WebDriver webDriver = getWebDriver(url);
log.info(webDriver.getPageSource());
quitWebdriver(webDriver);
}
private WebDriver getWebDriver(String url) {
WebDriver webDriver = new ChromeDriver(options);
webDriver.get(url);
return webDriver;
}
private void quitWebdriver(WebDriver webDriver) {
webDriver.close();
webDriver.quit();
}
}
When I launch it, I get the following error at the moment "WebDriver webDriver = new ChromeDriver(options)":
java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method
HandlerMethod details:
Bean [dataox.keithgellmanrestaurantmenuscraping.ParserService]
Method [public void dataox.keithgellmanrestaurantmenuscraping.ParserService.a()]
Resolved arguments:
Caused by: java.lang.NoClassDefFoundError: org/openqa/selenium/virtualauthenticator/HasVirtualAuthenticator
I work with Gradle, my gradle.build looks like this:
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.12.0'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-api', version: '3.141.59'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-chrome-driver', version: '3.141.59'
implementation group: 'org.jsoup', name: 'jsoup', version: '1.15.3'
compileOnly 'org.projectlombok:lombok'
developmentOnly 'org.springframework.boot:spring-boot-devtools'
annotationProcessor 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
}
What is the best thing to do in this situation?
CodePudding user response:
I found the answer on my own. It turned out that it was enough for Gradle to remove the versions altogether. Spring pulls those versions that are relevant for work.
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.12.0'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-api'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-chrome-driver'
implementation group: 'org.jsoup', name: 'jsoup', version: '1.15.3'
compileOnly 'org.projectlombok:lombok'
developmentOnly 'org.springframework.boot:spring-boot-devtools'
annotationProcessor 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
}