Elasticsearch not showing running log in python-CodePudding

I am new in Elasticsearch. I have a running Elasticsearch instance in cloud and accessing it via python, i want to see running logs which has a field - "type"- "filebeat". I have following lines of code:

import elasticsearch
from elasticsearch import Elasticsearch
import elasticsearch.helpers

# Creating the client instance
es = Elasticsearch(
    cloud_id=CLOUD_ID,
    basic_auth=("elastic", ELASTIC_PASSWORD)
)

# Successful response!
print(es.info())
ES_INDEX = <my index>
ES_TYPE="filebeat"
results_gen = elasticsearch.helpers.scan(
    es,
    query={"query": {"match_all": {}}},
    index=ES_INDEX)

results = list(results_gen)
print(results)

The output shows the instance details and 4407 logs in result (obviously all logs). My question is how to obtain running logs and how to modify the query to show only logs with "type"-"filebeat"?

CodePudding user response：

You need to do some filtering in your query.

results_gen = elasticsearch.helpers.scan(
    es,
    query={"query": {"match_all": {}}},
    index=ES_INDEX)

In here, you are using match_all. This will return all data on your index.

Here is a query sample above. term query will filter the data according to type: filebeat.

results_gen = elasticsearch.helpers.scan(
    es,
    query={"query": {"term": {"type": "filebeat"}}},
    index=ES_INDEX)

Also, you can check the documentation for more.

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html