I have 2 certificate files, truststore.jks and keystore.jks. keystore.jks contains a full certificate chain for the kafka endpoint I'm using as well as a private key for my application. According to the group who gave me truststore.jks it contains "a new root CA chain" and is essential for getting my application to connect with the kafka endpoint.
I have tried using this script to generate 3 .pem files that I use in my Python code.
#!/bin/bash
srcFolder=$1
keyStore=$1/$2
password=$3
alias=$4
outputFolder=$5
echo $keyStore
echo "Generating certificate.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/certificate.pem -storepass $password
echo "Generating key.pem"
keytool -v -importkeystore -srckeystore $keyStore -srcalias $alias -destkeystore $outputFolder/cert_and_key.p12 -deststoretype PKCS12 -storepass $password -srcstorepass $password
openssl pkcs12 -in $outputFolder/cert_and_key.p12 -nodes -nocerts -out $outputFolder/key.pem -passin pass:$password
echo "Generating CARoot.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/CARoot.pem -storepass $password
I use the generated CARoot.pem, certificate.pem, and key.pem files in my code like so.
>>> from kafka import KafkaProducer
>>> producer = KafkaProducer(bootstrap_servers='kafka-dev.kafka.com:port', security_protocol="SSL",
... ssl_check_hostname=True,
... ssl_cafile="CARoot.pem",
... ssl_certfile="certificate.pem",
... ssl_keyfile="key.pem",
... ssl_password="")
It gives this error:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)
I have tried combining truststore.jks and keystore.jks like so:
keytool -importkeystore -srckeystore truststore.jks -destkeystore keystore_copy.jks
I then put keystore_copy.jks through the same .pem conversion process as before, but it produces the same error.
For some context, this is how the Kafka endpoint owners suggest to configure the ssl certs in Java, unfortunately I am the first to connect with Python so they haven't been able to help me with it.
props.setProperty(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
props.setProperty(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, "<kafka-truststore.jks>");
props.setProperty(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "<password>");
props.setProperty(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, "<client-keystore.jks>");
props.setProperty(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "<password>");
props.setProperty(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "<password>");
Is there something I'm missing? I believe I need to utilize both .jks files in my code somehow but the only approach I can think of taking failed.
Additional things I have tried:
pip install --upgrade certifi based on reports of the same error due to that library being out of date.
CodePudding user response：
The following properties which KAFKA endpoint owners are asked you is correct, but looks like the issue with your keystore/truststore having a self signed certificate. It appears ssl.verify or hostName verification is enabled at the server level.
props.setProperty(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
props.setProperty(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, "<kafka-truststore.jks>");
props.setProperty(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "<password>");
props.setProperty(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, "<client-keystore.jks>");
props.setProperty(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "<password>");
props.setProperty(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "<password>");
This is one give you better clarity on this error.
CodePudding user response：
I managed to figure out the solution on my own. The key here was realizing that I needed to have the entire cert chain from my truststore.jks file present in CARoot.pem for the code to work. Here are the steps that I took:
Generate CARoot.pem from the trust store:
keytool -importkeystore -srckeystore <<truststore-name.jks>> -destkeystore keystore.p12 -srcstoretype jks -deststoretype pkcs12
openssl pkcs12 -in keystore.p12 -out CARoot.pem
Generate certificate.pem from keystore
keytool -exportcert -alias <<alias>> -keystore ./<<keystore—name.jks>> -rfc -file ./certificate.pem -storepass <<password>>
Extract private key and generate RSAkey.pem from keystore:
keytool -importkeystore -srckeystore <<keystore-name.jks>> -destkeystore keystore.p12 -deststoretype PKCS12
openssl pkcs12 -in keystore.p12 -nodes -nocerts -out RSAkey.pem
Example with Kafka-Python Producer:
producer = KafkaProducer(bootstrap_servers='kafka-dev.kafka.com:port', security_protocol="SSL",
ssl_check_hostname=False,
ssl_cafile="CARoot.pem",
ssl_certfile="certificate.pem",
ssl_keyfile="RSAkey.pem",
ssl_password=<<password>>)