Fetch paginated data without knowing the page number-CodePudding

I have an API that only allows fetching 1000 rows/fetch.

So for instance, if I want to retrieve all the data from the API, the idea is to loop through the response data each time fetching and check the length of it (if responseData.length !== 0, then continue fetching, stop when responseData.length === 0, also increase the firstRow every time it starts the new loop until it reaches to the end (responseData.length === 0)

const fetchDataByRowCount = async (url, token, rowCount = 2, firstRow = 0) => {
// firstRow is the value where the next fetch starts (E.g: 0-999, 1000-1999, etc.).
// rowCount is the value for total rows fetched (E.g: 1000 rows for each fetching time).
  const data = await axios({
    method: "get",
    url: `${url}?rowCount=${rowCount}&firstRow=${firstRow}`,
    headers: {
      client_id: "",
      Authorization: `Bearer ${token}`,
    },
  });
  return data.data;
};

export const paginatedFetch = async (url, type, rowCount = 2, firstRow = 0) => {
  let newResponse;
  let total = [];
  let token = await getToken(type); // stored to reuse token within an hour
  do {
    if (!token) {
      const newToken = await getToken(type);
      newResponse = await fetchDataByRowCount(url, newToken);
    } else {
      newResponse = await fetchDataByRowCount(
        url,
        token,
        (rowCount = 2),
        (firstRow = 0)
      );
    }
    // console.log(total, "total");
    total = [...total, ...newResponse];
    // newResponse = [];
    let newFirstRow = firstRow   1000;

    newResponse = await fetchDataByRowCount(
      url,
      token,
      (rowCount = 2),
      newFirstRow
    );
    total = [...total, ...newResponse];
  } while (newResponse.length !== 0);
  return total;
};

But the problem is that my function didn't exit the do while loop, the newResponse always returns value !==0. Also, the function only runs once.

Could you guys help me check this, please?

CodePudding user response：

From the code you posted, there's still a thing I can't figure out, and it's rowCount, so I let it as it is in the following "remastered" code:

export const paginatedFetch = async (url, type, rowCount = 2, firstRow = 0) => {
  let newResponse;
  let total = [];
  let token;
  let numberOfRows = firstRow;
  do {
    if (!token) token = await getToken(type);

    newResponse = await fetchDataByRowCount(
      url,
      token,
      (rowCount = 2),
      numberOfRows
    );
    total = [...total, ...newResponse];
    numberOfRows  = 1000;
  } while (newResponse.length !== 0);
  return total;
};

I got rid of a few things that were redundant, and made the code a tiny bit more efficient with variable assignments etc.

You also mention this:

the newResponse always returns value !==0.

Be careful doing this, as newResponse is initially undefined. Now I never used do...while loops so I don't know exactly what could happen, but it could for example not run at all. Hence the Also, the function only runs once if you're speaking about the paginatedFetch function.

Now if I were to re-write it, I would do it like so:

export const paginatedFetch = async (url, type, rowCount = 2, firstRow = 0) => {
  let total = [];
  let token;
  let numberOfRows = firstRow;
  while (true) {
    if (!token) token = await getToken(type);
    let res = await fetchDataByRowCount(
      url,
      token,
      (rowCount = 2),
      numberOfRows
    );
    total = [...total, ...res];
    numberOfRows  = 1000;
    if (res.length <= 0) break;
  }
  return total;
};

Again, be careful with while (true), you have to be absolutely sure of what the API returns and res is indeed an array.

The better solution would be the API (if you are the developer) giving an endpoint to count the total number of rows. That way, you would have a way to be absolutely sure how many rows there are and write your code around that.