Scala Requests module: Is it possible to make concurrent requests?-CodePudding

I am using Scala concurrent module but I am not sure if it works asynchronously or synchronously ? I have a list of urls which are image urls and I want to make concurrent requests and get Array[Bytes]:


object Runner extends App {
  def getBytesFromUrl(url:String) = {
    requests.get(url).bytes
  }
  val urls = Seq(
    "https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg",
    "https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg",
    "https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg"
  )
  
  // Would this make concurrent or sequential requests ?
  val result = urls.map(url => getBytesFromUrl(url))
  
}

Would the above code make concurrent or sequential requests? And if it makes sequential requests then what's the right way to make concurrent requests?

CodePudding user response：

I am using Scala concurrent module

You are not, at least not in the code shown.
Maybe you meant "request-scala library" in there?

but I am not sure if it works asynchronously or synchronously

For concurrency to exist you need asynchronously.
Now, again, if you were wondering about requests-scala then it is synchronously, that should be clear from the README and from the type signature.

Would the above code make concurrent or sequential requests?

Sequential, as the library explicitly states.

And if it makes sequential requests then what's the right way to make concurrent requests?

For one, you may consider using a different ecosystem since the author has been very clear that he doesn't think you need to be asynchronous at all to be efficient most of the time.
Another thing to consider is if you really need to make three HTTP calls concurrently.

Anyways, you can just use Future.traverse to do what you want.

object Runner {
  def getBytesFromUrl(URL: String): Array[Byte] = {
    requests.get(url).bytes
  }

  def getBytesFromUrls(urls: List[String]): Future[List[Array[Byte]]] =
    Future.traverse(urls)(url => Future(getBytesFromUrl(url)))
}

You can then either compose that Future or Await it if you want to return to synchronous-land.

Personal disclaimer, if you actually have a lot of URLs and you need to do other asynchronous things with those bytes, like writing them to disk. Then, I personally would recommend you to look to other ecosystems like typelevel, since those provide tools to write that kind of program in a more principled way.