Dealing with rate limiting APIs in NodeJS-CodePudding

So I need a way to run a function x amount of times in a given second, then wait until the next second to run the next set. I'm using the Yelp Fusion API to call a 'search' https://api.yelp.com/v3/search and then running a details query on each of the search results: https://api.yelp.com/v3/businesses/business_id.

So the first search query returns an array of businesses like this:

const businesses = response.body.businesses
for (business of businesses) {
    fetchPlaceDetails(business.id)
}

Theoretically I can run fetchPlaceDetails() in an async manner, but going through 50 results takes too long. Instead, I'd like to run the query maybe 5 times, wait a second (to get past the rate limit), run the next 5, wait a second, run the next 5, etc.

Not sure how to implement this but I figure it's got to be a pretty standard issue to have when using APIs with rate limits. Also, I'm not sure but nowhere in the documentation do I see an actual call/second limit specified, but I'd assume it's maybe 10 requests/second or so?

CodePudding user response：

I'd like to run the query maybe 5 times, wait a second (to get past the rate limit), run the next 5, wait a second, run the next 5, etc.

Then just do that:

const {businesses} = response.body;
for (let i=0; i<businesses.length; i =5) {
    await Promise.all([
        ...businesses.slice(i, i 5).map(({id}) => fetchPlaceDetails(id)),
        new Promise(resolve => { setTimeout(resolve, 1000); })
    ]);
}

This will also ensure that you wait more than one second between the batches if the 5 requests take longer than that.

CodePudding user response：

Here is a "generic" method that checks if storage array contains any data, if it does, it processes nn number of items from the array then waits ii number of seconds and repeats the process.

const queue = (() =>
{
  let itemsPerStep = 5, //number of items process per step
      intervalSpeed = 5000, //how long to wait between steps in miliseconds
      stopWhenDone = true, //when queue list depleted stop the loop
      prevDate = 0,
      timer;

  const list = []; //queue list

  const commands = {
    add: item => list.unshift(item),
    get list() { return list; },
    clear: () => list.length = 0,
    speed: (val = intervalSpeed) => intervalSpeed = val,
    itemsPerStep: (val = itemNumberPerStep) => itemNumberPerStep = val,
    stop: () => (timer && console.log("stopped"), timer = clearInterval(timer)),
    start: (t = true) => (stopWhenDone = t, commands.stop(), timer = setInterval(loop), loop())
  };

  const loop = () =>
  {
    const now = new Date();
    if (now - prevDate < intervalSpeed)
      return;

    if (!list.length)
    {
      if (stopWhenDone)
        commands.stop();
      return;
    }

    prevDate = now;

    let res = []; //example result storage

    for(let i = 0; i < itemsPerStep && list.length; i  )
    {
      const value = list.pop(); //get item from the queue list

      res.push(value); //process value
    }
    
    console.log("processed ids: "   res);
  } //loop()

  return commands;
})();


//generate ids
for(let i = 100; i < 230; i  = 11)
  queue.add(i);

queue.start();