recursively fetch all pages from api-CodePudding

I am trying to fetch all records from a remote api that has a limitation on how many can be fetched at once. I am trying to use recursion, but the general idea of what works in Python I can't get to work in JS as well.

This is what I am currently doing, and it will fetch 100 records, but then keeps going in the background. I have tried many different ways to do it and nothing seems to work at expected.

Any help would be great, thanks.

  async getBooks(offset = 0) {
    return await bookGenie.find(
      'fiction',
      100,
      offset,
    )
  }

CodePudding user response：

Your await getBooks() call:

getBooks((offset = offset  = 100))

should probably be written like:

getBooks((offset  = 100))

And the if statement before that looks wrong to me. Maybe try:

if (result.length < 1000) { // will continue until result.length is 1000
    result.push(await getBooks(offset  = 1)); /*   1 so the .push will add one element to the array each time.   100 may work too */
}

I wasn't able to actually test this, but this (or something similar to this) is more conventional for JS. Here is a link that might help you out https://www.javascripttutorial.net/javascript-recursive-function/

CodePudding user response：

I suggest this one:

async function getBooks(offset = 0, result = []) {
  try {
    // get new page
    const page = await bookGenie.find(
      'fiction',
      100,
      offset,
    );
    // add page to result array
    result = result.concat(page);
    // check is this the end
    if (page.length !== 100) { // or what condition do you need
      // return result array
      return result;  
    } else {
      // or increase offset and continue processing
      return getBooks(offset   100, result);
    }
  } catch (err) {
    throw err; // or handling somehow
  }
}

Now recursive getBook returns promise with all records found merged to one array on resolve and throwed error on reject.
All requests will be executed one-by-one increasing offset by 100 on each step until request return less then 100 records (not full page = end of list).

Usage:

const list = await getBooks();
console.log(list); 
// [ ... ]
// List of all records

Extra: same method but compact

async function getBooks(offset = 0, result = []) {
  try {
    const page = await bookGenie.find('fiction', 100, offset);
    return page.length !== 100 ? result : getBooks(offset   100, result.concat(page));
  } catch (err) {
    throw err; // or handling somehow
  }
}