I am pretty new to elasticsearch and I want to create statistics and kibana dashboards on queries sent to elasticsearch index , what is the best approach to do so ? Any advice or recommendations will be highly appreciated? The idea is to analyze all queries sent to the index and do some performance optimisation in the future when the userbase increase ...

I am planning for the moment to store the logs in different index , but parsing seems to be kind of complex activity ...

Ideally I need to have:

-Counting of user queries

-Counting of queries that returned no results

-Logging of all search terms

-Sorting of queries, and queries that returned no results, by most frequently contained search term

-A view of top queries, including the search term not found results for and the exact query

-A view of top queries returning no results, including the search term not found results for and the exact query

Thanks

CodePudding user response：

There is no OOTB functionality available in Elasticsearch for search analysis. But there are some workaround you can do for same and get information what you are asking.

First option, you can enable slow log in Elasticsearch by executing below command and it will log each and every request to coming to Elasticsearch.

PUT /my-index-000001/_settings
{
  "index.search.slowlog.threshold.query.info": "0s",
  "index.search.slowlog.threshold.fetch.info": "0s"
}

Second option, You can log all the query the application layer or intermediate level using which application and elasticsearch talking to each other.

Once you have logs, You can configured Logstash / Filebeat / Fleet to read log and transform and index to Elasticsearch. Logstash provide differnt kind of filter which you can use and easily transofrm your plain text logs to strcture logs (grok filter).