I have a web crawler that crawls thousands of websites and store that in a persistent volume on Kubernetes.

After that pod is terminated, I want to ingest the .json file inside the PV into ECK. I have achieved running elastic search and Kibana successfully. Also, I'm following the Quickstart guide.

CodePudding user response：

Filebeat will do this for you - https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html and https://www.elastic.co/guide/en/beats/filebeat/7.14/decode-json-fields.html

ps - it's Elasticsearch, not elasticsearch ;)

CodePudding user response：

You can use the daemonset of filebeat for kubernetes setup

https://www.elastic.co/guide/en/beats/filebeat/7.14/running-on-kubernetes.html