I have a list which contain lat and lng values I just traverse this list fetching the lat and lng and find the weather report for particular Lat and lng but it takes around 75 sec for 100 times. how can I decrease this time

for(LatandlngList value:latandlngList)
{
   Double lat= value.getLat();
   Double lng= value.getLng();
   String URL="http://api.openweathermap.org/data/2.5/weather?lat=" lat "&lng=" lng
              "&appid=" appId;

HttpClient weatherClient=new DefaultHttpClient();

HttpGet weatherReq=new HttpGet(url);
HttpResponse weatherresult=weatherClient.execute(weatherReq);
}

The consider loop runs 100 time for finding weatherReport it takes around 75 second how can I decrease this time

CodePudding user response：

You could try reusing connections. Your example should not need a new HTTP client each time. See: https://hc.apache.org/httpclient-legacy/performance.html

CodePudding user response：

Few points to consider based on code-

1. If possible, create singleton HttpClient. In your code, you are creating HttpClient in every iterations-- which is not good idea.
   Once singleton is created, use it in your loop.

2. If sequential execution is not constraint, you can use parallel execution-- by using Threads or even by using parallelStreams()

CodePudding user response：

Run them in parallel. You can't speed up a single request, its performance is determined by network and http server response speed. E.g. here's a post that goes through the exercise for POSTs, but it applies for GETs as well.

Btw. you don't need a new HttpClient for each request, one for all will do. This may save you a little time, but nowhere near levels that make a difference.

Another classical issue when scraping data may be throttling by the server, which can be circumvented e.g. using IP rotation, but that's a whole topic by itself.

Finally you can check for overlapping areas in your lat/long list to prevent multiple requests returning the same data. There's some headache about map projections used, but its definitely doable. I used the JTS topology suite once for such stuff.