scraping using cheerio returns empty array-CodePudding

I am trying to scrape a website https://buff.163.com/market/csgo#tab=buying&page_num=1 using request-promise and cheerio. my entire code is below :

const request = require('request-promise');
const cheerio = require('cheerio');

const url = "https://buff.163.com/market/csgo#tab=buying&page_num=1";

const scrapeArr = [];

async function scrape() {
    try {
        const htmlResult = await request.get(url);
        const $ = await cheerio.load(htmlResult);
        
        $(".card_csgo li")
            .each((i, e) => { 
                const title = $(e).children("h3").text() 
                const link = $(e).children("a").attr("href")
                const scrapeObj = { title , link }
                scrapeArr.push(scrapeObj)
            })
            console.log(scrapeArr);
    } catch(e) {
        console.log(e)
    }
}

scrape()

this results in a empty scrapeArr, The jQuery commands work perfectly in the chrome dev tools console but when I copy paste the same command in cheerio it results in empty array. can someone tell me what's the problem here?

CodePudding user response：

If you open the source of the page, you will see that there are no such DOM elements in HTML. They are generated by browser in runtime, so you need full-fledged browser (and not raw network request cheerio) to render them.

The good news is that this website has hidden API which you should better use to extract the data you need, just use Chrome Dev Tools to discover it: https://www.youtube.com/watch?v=kPe3wtA9aPM