I am trying to understand if I have an issue with my AWS Elasticsearch garbage collection time but all the memory-related issues that I find are relating to memory pressure which seems OKish.

So while I run a load test on the environment, I observe a constant rise in all GC collection time metrics, for example:

But when looking at memory pressure, I see that I am not passing the 75% mark (but getting near..) which,

So I fear that once I add more load or run a longer test, I might start seeing real issues which will have an impact on my environment. So, do I have an issue here? how should I approach rising GC time when I cant take memory dumps and see what's going on?

CodePudding user response：

The top graph reports aggregate GC collection time, which is what's available from GarbageCollectorMXBean. It continues to increase because every young generation collection adds to it. And in the bottom graph, you can see lots of young generation collections happening.

Young generation collections are expected in any web-app (which is what an OpenSearch cluster is): you're constantly making requests (queries or updates), and the those requests create garbage.

I recommend looking at the major collection statistics. In my experience with OpenSearch, these happen when you're performing large numbers of updates, perhaps as a result of coalescing indexes. However, they should be infrequent unless you're constantly updating your cluster.

If you do experience memory pressure, the only real solution is to move to a larger node size. Adding nodes probably won't help, due to the way that indexes are sharded across nodes.

CodePudding user response：

I sent a query to AWS technical support and, counter to any intuitive behavior, the values of the Young and Old Collection time and count in Elasticsearch is cumulative. This means that this value keeps increasing and does not drop down to a value of 0 until there is a node drop or node restart