I am trying to create self-signed certificate for usage with a Netty (4.1.86) Websocket Server. It's running fine without SSL as well as with a certificate created with the SelfSignedCertificate class. But I am struggling when creating a self signed certificate with openssl.

This:

SslContext sslCtx = SslContextBuilder.forServer(new File(certFile), new File(keyFile), password).build();

Throws the following error:

ERROR Thread-5 com..application.NettyWSServer - Exception caught:
java.lang.IllegalArgumentException: File does not contain valid private key: /home/johnny/testbench/application/app.pkcs8.key
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:386)
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120)
        at com..application.NettyWSServer.start(NettyWSServer.java:78)
        at com..application.ApplicationLauncher$2.run(ApplicationLauncherncher.java:315)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
        at java.base/javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122)
        at java.base/javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168)
        at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:1084)
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1170)
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1133)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384)
        ... 4 more

Here's how I create certFile (app.pem) and keyFile (app.pkcs8.key):

openssl genrsa -out app.key 2048

openssl pkcs8 -topk8 -in app.key -out app.pkcs8.key

openssl req -x509 -new -nodes -key app.key -sha256 -days 1024 -out app.pem

As per my understanding Netty needs a pkcs8 format key:

https://netty.io/4.1/api/io/netty/handler/ssl/SslContextBuilder.html#forServer-java.io.File-java.io.File-

Working solution thanks to dave_thompson_085:

openssl genrsa -out app.key 2048

openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES -nocrypt -in app.key -out app.pkcs8.key

openssl req -x509 -new -nodes -key app.key -sha256 -days 1024 -out app.pem

CodePudding user response：

More exactly the default SSLContext path needs either PKCS8-unencrypted or PKCS8-encrypted using a password-based encryption (PBE) scheme that is defined uniquely by the OID at the beginning of the AlgorithmIdentifier, like the schemes in PKCS5v1 (now retronymed PBES1) or PKCS12, but not the newer/better family of schemes in PKCS5v2 named PBES2, because the OID for PBES2 -- 1.2.840.113549.1.5.13 which you see in your exception message -- does not identify a single scheme and thus is not sufficient for Java to instantiate a SecretKeyFactory (causing the exception). And openssl pkcs8 -topk8 by default uses a PBES2 scheme since 1.1.0 in 2016.

But the netty 4.1 SSLContext.toPrivateKey method(s) (which you can see and your stacktrace confirms SSLContextBuilder calls) will instead use BouncyCastlePemReader if available, and Bouncy can handle both older PBE schemes and PBES2, and also OpenSSL 'traditional' format files that are not PKCS8 at all (and if/when encrypted use a scheme based on OpenSSL's own PBKDF EVP_BytesToKey).

Thus I think (but can't test) you have 3 options:

1. supply BouncyCastle. I think you need bcprov (or bcprov-ext) and bcpkix; you may now need bcutil (due to a reorg in recent versions I haven't fully grokked).

2. encrypt your file with a non-PBES2 scheme. The PBES1 schemes are all now broken or easily breakable; the 'best' (least bad) PKCS12 scheme is officially named pbeWithSHA1And3-KeyTripleDES-CBC (!) but OpenSSL has a more convenient 'shortname':

   openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES [-in oldfile -out newfile]

   Technically you could also accomplish this by leaving the default on a very old version of OpenSSL, but that's usually more difficult(*) and in my view not really different. (* not for me, because I have my own archive of old versions, but I'm weird)

3. don't encrypt your keyfile, (only!) if you can adequately secure it otherwise, for example by access controls on your system(s) (and any backups). Use:

   openssl pkcs8 -topk8 -nocrypt [-in oldfile -out newfile]

   or a bit more simply:

   openssl pkey [-in oldfile -out newfile]