I'm using ElasticSearch (OpenSearch) to store multiple objects in an index. Each object has a unique ID, creation date and an "amount" field that stores integer value. This "amount" value changes periodically, so I'm updating objects in-place using the IDs to match the existing objects.

What I want is to display a histogram of total amount across all objects over time. The problem is that I need to preserve historical data somehow in order for it to be displayed on the graph.

What are my option to implement this? Is there an automatic way to aggregate and preserve these totals? Or should I do this manually from my application? E.g. send these totals instead of individual objects?

CodePudding user response：

(from an Elasticsearch point of view) you would normally store each each change as an individual record, rather than updating the existing one. that way you get the history and can do your aggs, and still show the latest values

just treat it as any other time based data, logs, click streams, transaction. etc

CodePudding user response：

Elasticsearch isn't very fit for processing historical or time-series data. It has some good tooling for cases like logging but fails when the use case assume some continuity in the data, like metric gauges or real-world prices. It is technically possible to create a cron job that would retrieve current values every minute and store them somewhere but it will be inefficient and poorly scalable.

Instead, it would make sense to use a specialized time-series database, like InfluxDB that allow aggregations like that together with the ability to zoom in and out of the timeline.