I am trying to establish a TLSV1.2 connection to a Java-based MQTT broker (Active MQ).
My client has the private key in an HSM module and hence inaccessible to me.
x509 cert is available.
CA is self-signed.
Cert details:
Signature Algorithm: ecdsa-with-SHA256
Public Key Algorithm: id-ecPublicKey (256bit)
curve: prime256v1
Behavior observed:
The server is throwing the following error:
43 2022-04-01 23:22:26.772084 serverip clientip TLSv1.2 73 Alert (Level: Fatal, Description: Handshake Failure)
This is happening right after Client key exchange
41 2022-04-01 23:22:26.739193 clientip serverip TLSv1.2 303 Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
What I have tried so far:
I tried establishing a connection through curl to the same broker using an x509 cert signed by the same CA. This is being accepted. Hence I have concluded this is not a bad certificate issue.
Note: I am using --insecure in the curl command, I am okay with this as client side auth is something I am not concerned about, given that server is throwing an error.
Observations/Assumptions:
The handshake failure is coming from the server after:
Client Hello
Server Hello
Server Hello, Certificate, Server Key Exchange, Certificate Request, ServerHelloDone
Certificate
Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
Alert (Level: Fatal, Description: Handshake Failure) ( from server)
By this I am concluding there is no issue with the cipher suite /protocol compatibility.
The only difference I see between the X509 cert generated by HSM and the cert I generated for verification is in the common name format.
Wireshark says:
- HSM generated cert CN is DirectoryString: teletexString (0)
- Manually generated cert CN is utf-8
The CN is the same, the format as shown by wireshark differs.
I have never looked at a handshake so in depth and know just basics of a TLS handshake.
The server is a managed Kubernetes deployment and hence really difficult currently to associate logs with clients.
But, based on observation , I see this message (activemq-netty-threads)","message":"AMQ222208: SSL handshake failed for client. java.io.IOException: Sequence tag error."
I think this is the corresponding broker logs.
Any way forward/ opinions appreciated.
Results of parsing x509 certs
- HSM cert
CN=A_00000066,OU=AA,O=Organisation Limited,L=Place,ST=State,C=C - FS certs
1.2.840.113549.1.9.1=#160a726f6f74406174686572,CN=A_00000066,OU=AA,O=O,L=Place,ST=State,C=C
Adding relevant Server logs
SSL logs on the server:
"throwable" : {
java.security.SignatureException: Invalid encoding for signature
at java.base/sun.security.util.ECUtil.decodeSignature(ECUtil.java:279)
at jdk.crypto.ec/sun.security.ec.ECDSASignature.engineVerify(ECDSASignature.java:477)
at java.base/java.security.Signature$Delegate.engineVerify(Signature.java:1247)
at java.base/java.security.Signature.verify(Signature.java:675)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:651)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyConsumer.consume(CertificateVerify.java:771)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:689)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.io.IOException: Sequence tag error
at java.base/sun.security.util.DerInputStream.getSequence(DerInputStream.java:336)
at java.base/sun.security.util.ECUtil.decodeSignature(ECUtil.java:255)
... 32 more}
)
javax.net.ssl|WARNING|32|Thread-4 (activemq-netty-threads)|2022-04-06 10:01:32.559 UTC|SSLEngineOutputRecord.java:168|outbound has closed, ignore outbound application data
Update 2:
The signing is handed over to the rustls library. There is a custom function that uses the HSM private key to sign the digest as required.
example signature: 0349003046022100838bde8a902f9ebb18cdd9bc5af263dc978a670d95770c11e2e8d29e3c7b2c28022100d345fa7245fb34c8cf710958da80a638c598c44e2cbd724571dfd9e9ade95008
Actual Client encrypted handshake message after which failure happens:
Consuming ECDHE ClientKeyExchange handshake message (
"ECDH ClientKeyExchange": {
"ecdh public": {
0000: 04 EB B8 76 96 C5 E0 C6 20 73 F0 4C AB 93 F1 A6 ...v.... s.L....
0010: E9 6C 64 B0 BB 72 64 A4 74 75 26 4B E2 79 C0 26 .ld..rd.tu&K.y.&
0020: 42 C8 C8 8F D4 C5 CA EC 22 DA B5 3B 03 E8 E8 19 B......."..;....
0030: 28 28 EF C6 9D EE 80 3A CD A1 60 2B 62 83 52 8F ((.....:..` b.R.
0040: 23 B4 5B 46 1F 76 86 00 0D DF F3 1E 6B 86 01 A4 #.[F.v......k...
0050: 64 09 C9 80 0A 03 C6 EE A4 AA 36 05 F4 45 7A 91 d.........6..Ez.
0060: A5 .
},
}
CodePudding user response：
The signing is handed over to the rustls library. There is a custom function that uses the HSM private key to sign the digest as required.
example signature: 0349003046022100838bde8a902f9ebb18cdd9bc5af263dc978a670d95770c11e2e8d29e3c7b2c28022100d345fa7245fb34c8cf710958da80a638c598c44e2cbd724571dfd9e9ade95008
Assuming you are displaying hex for data that is actually binary, that is almost correct. Dehexed it is actually a valid (or at least validly formatted) Ecdsa-Sig for a 256-bit curve such as P-256, embedded in a (ASN.1) BITSTRING. Specifically 03 49 00 is the tag and length for a BITSTRING containing 72 bytes with no unused/extra bits. Those 72 bytes consist of 30 46 02 21 (33bytes) 02 21 (33bytes) which is the correct encoding of: a constructed SEQUENCE (tag 0x10 plus 0x20) with value length 70 bytes, containing two primitive INTEGER items (tag 0x02) each with value length 33 bytes.
Whatever code is building the CertificateVerify message should remove the first 3 bytes and use the rest. It needs to handle varying lengths; at most the DER signature will be 72 bytes as in this example, but often it will be 71 or 70 bytes instead, and occasionally but rarely even less. Compare openssl rust crate: ECDSA signature size is not 64 bytes? and (as linked there) how to specify signature length for java.security.Signature sign method also (my) Java's BouncyCastle doesn't always verify OpenSSL ECDSA signature .
(PS: the ClientKeyExchange message is irrelevant here. Client signature is always in the CertVerify message. For server below TLS1.3, the signature does go in ServerKeyExchange, but that's not relevant to your case.)
CodePudding user response：
The signing is handed over to the rustls library. There is a custom function that uses the HSM private key to sign the digest as required.