I have a service that must receive pagnation queries in offset format, receiving offset and limit parameters. For example, if I receive offset=5&limit=10, I would expect to receive items 5-14 back. I am able to enforce some validation against these parameters, for example to set a maximum value for limit.

My data source must receive pagination requests in page number format, receiving page_number and page_size parameters. For example, if I send page_number=0&page_size=20, I would receive items 0-19. The data source has a maximum page_size of 100.

I need to be able to take the offset pagination parameters I receive and use them to determine appropriate values for the page_number and page_size parameters, in order to return a range from the data source that includes all of the items I need. Additional items may be returned padding out the start and/or end of the range, which can then be filtered out to produce the requested range.

If possible, I should only make a single request to the data source. Optionally, performance can be improved by minimising the size of the range to be requested from the datasource (i.e. fetching 10 items to satisfy a request for 8 items is more efficient than requesting 100 items for the same).

This feels like it should be relatively simple to achieve, but my attempts at simple mathematical solutions don't address all of the edge cases, and my attempts at more robust solutions have started to head into the more complex space of calculating and iterating over factors etc.

Is there a simple way to calculate the appropriate values?

I've put together a test harness REPL with a set of example test cases that makes it easy to trial different implementations here.

CodePudding user response：

An appropriate implementation is to start at page_size=limit, and then increment page_size until there's a single page that contains the whole range from offset to offset limit.

If you're thinking that you don't want to waste time iterating, then consider that the time taken by this method is at most proportional to the size of the result set, and it's completely insignificant comparted to the time you'll take reading, marshaling, unmarshaling, and processing the results themselves.

I've tested all combinations with offset limit <= 100000. In all cases, page_size <= 2*limit 20. For large limits, the worst case overhead always occurs when limit=offset 1. At some point it will become more efficient to make 2 requests. You should check for that.

CodePudding user response：

How about this ?

if (limit > 100) { 
// return error for exceeding limit...
}

mod_offset = (offset % limit)
page_number = (offset / limit) ;
page_size = limit;

// Sample test cases ..

// offset=25, limit=20 .. mod_offset = 5

(a) page_number = 1, page_size = 20  // skip first 'n' values equal to 'mod_offset'
(b) page_number = 1 1 = 2, page_size = 20 // include only first 'n' values equal to 'mod_offset'


// offset=50, limit=25 .. mod_offset = 0
(a) page_number = 2, page_size = 25  // if offset is multiple of limit, no need to fetch twice...

// offset=125, limit=20  .. mod_offset = 5

(a) page_number = 6, page_size = 20  // skip first 'n' values equal to 'mod_offset'
(b) page_number = 6 1 = 7, page_size = 20 // include only first 'n' values equal to 'mod_offset'