AWS java lambda - "java.lang.OutOfMemoryError: Java heap space"-CodePudding

Is there any way to get the java heap dump of a java lambda? I get an out of memory on the second run of my lambda, but unfortunately I can't find any tools similar to jprofiler that can be used with aws lambda (I've tried codeGuru and x-ray but it didn't help much).

Any advices on how to "debug" memory allocation of a lambda running on aws?

** Edit - I know how to run locally and check the memory using JProfiler and other such tools, but I need a specific scenario that is very hard to reproduce locally so I want specific solutions for aws lambda environment. **

CodePudding user response：

Googling for "trigger heap dump from inside JVM" brought me to this page by Baeldung, which has this code:

public static void dumpHeap(String filePath, boolean live) throws IOException {
    MBeanServer server = ManagementFactory.getPlatformMBeanServer();
    HotSpotDiagnosticMXBean mxBean = ManagementFactory.newPlatformMXBeanProxy(
      server, "com.sun.management:type=HotSpotDiagnostic", HotSpotDiagnosticMXBean.class);
    mxBean.dumpHeap(filePath, live);
}

I haven't tried it, but have verified that the HotSpotDiagnosticMXBean exists in the Corretto 17 distribution, so it's likely to be available to a Lambda. You should, of course, write a test Lambda to verify.

I would execute this code as the first thing in the Lambda invocation handler, looking only for live objects, and upload the file to S3. This will tell you if there's anything that you are unexpectedly holding from the previous call. If there isn't, you simply need more memory.

The problem that you might run into is that Lambda only provides 512 MB of file storage space, and heap dumps can easily be larger than that. If this is the case for you, then you'll need to use an EFS volume to hold the file (and there's no need to upload to S3).

I recommend using the Lambda's request ID (available from the context) to uniquely name the files. This will make it easy to trace back from the execution logs.

And finally, this is something for development only. It's going to add a lot of time to the Lambda invocation.

CodePudding user response：

Setup AWS CLI

aws iam list-mfa-devices --user-name <ur_user>

use the SerialNumber from list-mfa-devices to get-session-token

aws sts get-session-token --duration-seconds 3600 --serial-number
        <SerialNumber> --token-code <<Say from Google authenticator>>

AWS_ACCESS_KEY=AccessKeyId
AWS_SECRET_KEY=SecretAccessKey
AWS_SESSION_TOKEN=SessionToken

Pass above as environment variable when you launch your handler method locally (Say from Junit) with same memory settings as suggested by Parsifal. Same as ur Lambda memory setting value Additionally pass below Vm argument -XX: HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:\ur_path
This will generate HeapDump file and you can use Eclipse MAT to analyze memory leak This is what I normally follow. This will work in any complex setup. Thanks