I was excited to see Tomcat 9 finally get support for Unix domain sockets, but it appears my setup is bad or incomplete.
Unfortunately I can't seem to find a working example of a server.xml anywhere on the internet. What seems to be happening is that a unix domain socket doesn't have a remote IP, and Tomcat uses that in several places of it's stack.
When I kick off a curl request to the socket, the second request crashes and freezes the server up.
sudo curl -kv -H "Connection: close" -H "x-forwarded-for: 127.0.0.1" http://localhost/hawtio/auth/login --unix-socket /opt/app-name/http.sock
I get a crash in the Tomcat log:
#first request
SEVERE [http-apr-/opt/app-name/http.sock-exec-1] org.apache.coyote.http11.Http11Processor.service Error processing request
#011java.lang.NullPointerException
#011#011at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1770)
#011#011at java.base/java.util.regex.Matcher.reset(Matcher.java:416)
#011#011at java.base/java.util.regex.Matcher.<init>(Matcher.java:253)
#011#011at java.base/java.util.regex.Pattern.matcher(Pattern.java:1133)
#011#011at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:624)
#011#011at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
#011#011at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
#011#011at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
#011#011at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
#011#011at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
#011#011at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2113)
#011#011at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
#011#011at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
#011#011at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
#011#011at java.base/java.lang.Thread.run(Thread.java:829)
# second request
SEVERE [http-apr-/opt/app-name/http.sock-Acceptor] jdk.internal.reflect.NativeMethodAccessorImpl.invoke Error allocating socket processor
#011java.lang.NullPointerException
#011#011at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:820)
#011#011at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:87)
#011#011at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:149)
#011#011at java.base/java.lang.Thread.run(Thread.java:829)
Setup:
- Ubuntu 18.04
- Tomcat 9.0.64
- Java 11 Latest
- Apache Portable Runtime / Tomcat Native 1.2.35
Config:
<Service name="CatalinaLocal">
<Connector
protocol="org.apache.coyote.http11.Http11AprProtocol"
unixDomainSocketPath="/opt/app-name/http.sock" />
<Engine
defaultHost="localhost"
name="Catalina">
<Host
name="localhost"
appBase="webapps"
unpackWARs="true"
autoDeploy="false"
deployIgnore="(?!.*hawtio).*">
<Valve
className="org.apache.catalina.valves.RemoteIpValve" />
</Host>
</Engine>
</Service>
If you take out the RemoteIpValve, same behavior.
EDIT
The relevent code is here: https://github.com/apache/tomcat/blob/9.0.x/java/org/apache/tomcat/util/net/AprEndpoint.java#L807
// Do the duplicate accept check here rather than in serverSocketaccept()
// so we can cache the results in the SocketWrapper
AprSocketWrapper wrapper = new AprSocketWrapper(socket, this);
// Bug does not affect Windows. Skip the check on that platform.
if (!JrePlatform.IS_WINDOWS) {
long currentNanoTime = System.nanoTime();
if (wrapper.getRemotePort() == previousAcceptedPort) {
if (wrapper.getRemoteAddr().equals(previousAcceptedAddress)) {
if (currentNanoTime - previousAcceptedSocketNanoTime < 1000) {
throw new IOException(sm.getString("endpoint.err.duplicateAccept"));
}
}
}
previousAcceptedPort = wrapper.getRemotePort();
previousAcceptedAddress = wrapper.getRemoteAddr();
previousAcceptedSocketNanoTime = currentNanoTime;
}
This is a bit interesting because this code is only present specifically in the AprEndpoint connector (That is Apache Portable Runtime, or invoking the Apache Server via JNI (Java Native Interface)), but I was unable to find the bug it is referencing that it is trying to fix. It may not be relevant to using the APR directly and this could be an oversight.
CodePudding user response：
For anyone also experiencing the issue.... I started a conversation with the Tomcat devs, they agreed this was a bug, and A PR opened for Tomcat with a fix. Not released yet, but hopefully soon.