Any better solution than custom Iterators for handling pagination?-CodePudding

I'm working with an API that paginates it's results, that API returns a response which contains a boolean flag "isLastPage" indicating whether there are any records left, my intention is to "yield" a single record at a time while calling the database.

I'm new to Scala and was expecting to be able to create something similar to a Python generator where an item is yielded every call and the context persists, however I couldn't find such a solution in Scala (not a nice one anyway) so I've solved this by extending Iterator, I'm not very happy with this solution because it relies on mutability.

Is this a good approach? I'd love to get some input and better alternatives

this is a simulation of my code just for illustration: I'm representing the paged data here as Page and the database client library with the accessData function, the entry point is iterateItems:

case class Page(items: List[Int], nextPage: Option[Int], isLastPage: Boolean = false)

def accessData(nextPage: Option[Int]): Option[Page] = {
  val data = List(Page(List(1, 2, 3), Some(1)),
    Page(List(4, 5, 6), Some(2)),
    Page(List(7, 8, 9), Some(3)),
    Page(List(10, 11, 12), None, isLastPage = true))
    val item = Option(nextPage.fold(data(0))(idx => data(idx)))
    println(f"accessing data $item")
    Thread.sleep(50)
    item

}

class PageIterator extends Iterator[Page] {
  var hasNextPage = true
  var nextPage: Option[Int] = None
  override def hasNext = hasNextPage
  override def next(): Page = {
    val page = accessData(nextPage).get
    nextPage = page.nextPage
    hasNextPage = !page.isLastPage
    page
  }
}

def iterateItems = (new PageIterator).flatMap(_.items)


iterateItems.foreach(item => println("now working on "   item))

Notice how I must rely on isLastPage in the response to know whether there needs to be a another call to the API.
I'm representing nextPage as int here but in reality it doesn't have any consecutive quality I could possibly rely on

Thanks!

CodePudding user response：

As I always say, the Scaladoc is your friend; you can use the unfold method on Iterator

def accessData(nextPage: Int = 0): Page = ???

Iterator.unfold((0, true)) {
  case (idx, true) =>
    val page = accessData(idx)
    Some(((idx   1), !page.isLastPage) -> page)

  case (_, false) =>
    None
}.flatMap(_.items)

PS: Since you are probably doing an asynchronous call, you will want to mix something like a Future there, but then mixing Futures and an Iterator is not simple and will be error-prone.
My suggestion would be to use fs2 Stream and cats-effect IO instead of Iterator & Future, the solution would be similar but using unfoldChunkEval instead.

Other alternatives may be Akka Streams & *ZIO.