The LWP::UserAgent module allows using callback to read web content in chunks.

$r = $ua->get($url, ':content_cb' => sub
{
  my ($chunk, $resp, $proto) = @_;
  # ...
});              

It seems that the chunk size is preset to 16384 bytes, which is very small for me.

How can I set the chunks to be of a different size, …let's say 4MB, i.e. 4*1024*1024 ?

I added :read_size_hint as follows, but seen no effect.

$r = $ua->get($url, ':read_size_hint' => 4*1024*1024, ':content_cb' => sub
{
  my ($chunk, $resp, $proto) = @_;
  # ...
});              

CodePudding user response：

:read_size_hint is only a hint given to sysread of the underlying socket module. This does not mean that there will be that much data read at once, only that they might be if they are already available.

How much is actually available and thus can be returned by sysread depends on how fast the sender will send data (sysread will not wait for more but just return what's available so far) and how large the sockets read buffer is.

In case of TLS it will also depend on the TLS record size used by the sender, i.e. a single sysread will not return results spanning multiple TLS records. Since the maximum record size in TLS is 16384 any larger settings for :read_size_hint have therefore no effect.

CodePudding user response：

I'm afraid this is entirely at the sending server's discretion.

What I've done is accumulate their response in the response_data handler, until some criterion at which further action is taken.

Another option is to let the object accumulate it and use other handlers.