I am trying to web scrap this website here

using JSOUP, But in this website when we put link in search bar and click search button the website dynamically load and show Some downloads links that what i want to scrap, my problem is how to load link in JSOUP with the link search without clicking search button and show result (scrap result ).Is there any way to search link and load without clicking any button and get result ?? Below code what I Tired "Not Getting required result".

  val result:Document = Jsoup.connect(Constants.BASE_URL)
            .data("url", Constants.YOUTUBE_LINK)
            .data("sid", "9823478982349872384789273489238904790234")
            .userAgent("Mozilla").post()

CodePudding user response：

JSOUP is a Static HTML parser. You cannot parse the content that is loaded by javascript dynamically. For that, you have to use a web drive.

The best web drives that you can use are

• HTML Unit
• JBrowserDriver

You can also use selenium but it may not be ideal for android :-

CodePudding user response：

As mentioned by ʀᴀʜɪʟ, JSOUP is a static HTML parser only. If you want to scrape a website that uses JS generated content you should probably take a look at skrape.it library

fun getDocumentByUrl(urlToScrape: String) = skrape(BrowserFetcher) { // <--- pass BrowserFetcher to include rendered JS
    request { url = urlToScrape }
    response { htmlDocument { this } }
}


fun main() {
    // do stuff with the document
}