I am working on a SpringBoot\Spring Batch application. I tested it on my development machine and I found no problem . It is working fine. The problem happened when I tried to deploy it on a production machine.
First of all I say that at the momement I installed Oracle Java 17 on this prod machine in this simple way: I only downloaded the tar.gx version of the Oracle Java 17 SE from here: https://www.oracle.com/java/technologies/javase/jdk17-archive-downloads.html
I extract it on this Linux Centos 7 machine, then into the bin folder I can execute java command:
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]# pwd
/home/webadmin/java/jdk-17.0.1/bin
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]# ./java -version
java version "17.0.1" 2021-10-19 LTS
Java(TM) SE Runtime Environment (build 17.0.1 12-LTS-39)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.1 12-LTS-39, mixed mode, sharing)
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]#
I am not sure that it is totally corect but untill now it seems to works because I am obtaining the Java version as output.
Then I tried to execute my Spring Boot\Spring Batch application, in this way:
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]# ./java -jar /home/webadmin/notartel-import-data-batch/UpdateInfoBatch-0.0.1-SNAPSHOT.jar
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.3)
2021-11-14 11:27:04.878 INFO 23288 --- [ main] c.n.u.UpdateInfoBatchApplication : Starting UpdateInfoBatchApplication v0.0.1-SNAPSHOT using Java 17.0.1 on c00lc5-notariato.sphostserver.com with PID 23288 (/home/webadmin/notartel-import-data-batch/UpdateInfoBatch-0.0.1-SNAPSHOT.jar started by webadmin in /home/webadmin/java/jdk-17.0.1/bin)
2021-11-14 11:27:04.880 INFO 23288 --- [ main] c.n.u.UpdateInfoBatchApplication : No active profile set, falling back to default profiles: default
2021-11-14 11:27:06.759 INFO 23288 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-11-14 11:27:06.783 INFO 23288 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-11-14 11:27:06.783 INFO 23288 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.50]
2021-11-14 11:27:06.831 INFO 23288 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-11-14 11:27:06.831 INFO 23288 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1898 ms
[2.997s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[2.998s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[3.001s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
2021-11-14 11:27:07.013 WARN 23288 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
2021-11-14 11:27:07.028 INFO 23288 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-11-14 11:27:07.046 ERROR 23288 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:163) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) ~[spring-context-5.3.9.jar!/:5.3.9]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.3.jar!/:2.5.3]
at com.notariato.updateInfo.UpdateInfoBatchApplication.main(UpdateInfoBatchApplication.java:42) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[UpdateInfoBatch-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[UpdateInfoBatch-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[UpdateInfoBatch-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[UpdateInfoBatch-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method) ~[na:na]
at java.base/java.lang.Thread.start(Thread.java:802) ~[na:na]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.startDaemonAwaitThread(TomcatWebServer.java:203) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:137) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:450) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:199) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182) ~[spring-boot-2.5.3.jar!/:2.5.3]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160) ~[spring-boot-2.5.3.jar!/:2.5.3]
... 16 common frames omitted
As you can see I am trying to execute my jar file but I am obtaining this exception:
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
So, since on my development machine it works fine (and I am using the same Java version of my dev machine) I suppose that it should be some JVM configuration or something like this. But what?
I tried to explore what show here: http://www.mastertheboss.com/jbossas/monitoring/how-to-solve-javalangoutofmemoryerror-unable-to-create-new-native-thread/
But I have not root access on this machine so I can't perform a lot of checks. The only checks that give me an output using this webadmin user was the one related to the current number of running threads, this one:
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]# cat /proc/loadavg
0.15 0.27 0.30 1/365 23666
and the number of process per user:
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]# ulimit -a
core file size (blocks, -c) 200000
data seg size (kbytes, -d) 200000
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128613
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 200000
open files (-n) 100
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 35
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
This is the command that should show the count the number of processes running:
webadmin@webadmin.xxx.it [~/java/jdk-17.0.1/bin]# ps -elfT | wc -l
365
I don't know if these last system information could be relevant for this specific problem.
Why am I obtanining this error? What can I try to do to fix this issue? Or what have I to ask to the system engineer handling this server to change?
CodePudding user response：
max user processes (-u) 35
this seems really low. You should be checking the amount of processes currently run by your user - the exception, to me, seems to indicate that this limit is hit. Thus an increase on user process limits would be needed by someone with admin rights.
Also, this:
max memory size (kbytes, -m) 200000
isn't much either for a java app. You only get 200M total, which for most java web apps isn't enough.
Your production account seems to be configured to really low amount of resources, which would either need to be increased, or then you need to change your approach for the app and start developing it for a low-resource environment, instead of regular environment.