Using Elasticsearch to collect information from Yahoo-Finance query server-CodePudding

I have been trying to query yahoo finance using the following url:

https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US&region=US&query=A*&type=equity&count=10001&start=0&scroll=true

I receive this message:

{"finance":{"result":null,"error":{"code":"internal-error","description":"Elasticsearch exception [type=illegal_argument_exception, reason=Result window is too large, from size must be less than or equal to: [10000] but was [10001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]"}}}

I am not familiar with Elasticsearch so for the last few days I have been trying to learn about it and spent almost all my time trying to figure this out but I can't find much information about this. I'd rather completely avoid using Elasticsearch and use python to write a request to that link and somehow set the "index.max_result_window" so the website would return more than 10,000 rows of data. I am trying to save the data I receive from the above url as a json file.

I have tried to register for the free Elasticsearch service online but I could not figure out how to provide the above url as an input source.

any help is highly appreciated.

CodePudding user response：

This setting is set at index level, that is the elastic used by yahoo finance. You cant make this change with query, only index managers can do.

Elastic search has scroll API which can be used for this elastic.co/guide/en/elasticsearch/reference/2.2/search-request-scroll.html

CodePudding user response：

You're not using Elasticsearch. Yahoo is using Elasticsearch. You're using Yahoo's API.

You'll have to do multiple API requests to Yahoo to retrieve more than 10,000 data points.

You can use the start query parameter to specify the index of the next data point that you would want to retrieve. So you would make API calls to, for example:

https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US&region=US&query=A*&type=equity&count=10000&start=0&scroll=true
https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US&region=US&query=A*&type=equity&count=10000&start=1&scroll=true
https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US&region=US&query=A*&type=equity&count=10000&start=2&scroll=true
...
etc.