Bouncy Castle is throwing "mac check in OCB failed" when trying to decrypt an AES message-CodePudding

I need to decrypt an AES message. I was able to make it work on python(using the pycryptodome library) but I wasn't successful on kotlin/java. Since the decryption mode is OCB and Java does not support it by default, I had to use the Bouncy Castle library.

Here's the working python code:

def decrypt_aes_message(shared_key, encrypted_message):
    encrypted_msg = b64decode(encrypted_message["encryptedMessage"].encode())
    tag = b64decode(encrypted_message["tag"].encode())
    nonce = b64decode(encrypted_message["nonce"].encode())
    cipher = AES.new(shared_key.encode(), AES.MODE_OCB, nonce=nonce)
    return cipher.decrypt_and_verify(encrypted_msg, tag).decode()

Here's the java code:

fun decryptAesMessage2(sharedKey: String, encryptedMessageData: Map<String, String>): ByteArray {
    var encryptedMessage = encryptedMessageData["encryptedMessage"]!!.utf8Base64Decode()
    var tag = encryptedMessageData["tag"]!!.utf8Base64Decode()
    var nonce = encryptedMessageData["nonce"]!!.utf8Base64Decode()

    var key = KeyParameter(sharedKey.toByteArray(Charsets.UTF_8))
    var params = AEADParameters(key, tag.size*8, nonce)
    var cipher = OCBBlockCipher(AESEngine(), AESEngine())

    cipher.init(false, params)

    val out = ByteArray(cipher.getOutputSize(encryptedMessage.size))
    var offset = cipher.processBytes(encryptedMessage, 0, encryptedMessage.size, out, 0)
    offset  = cipher.doFinal(out, offset) // Throwing exception here

    return out
}

The java code is throwing the exception org.bouncycastle.crypto.InvalidCipherTextException: mac check in OCB failed on cipher.doFinal

The file debug.zip has the complete problem reproducer. Inside the zip file you'll find:

py_working_code.py - the working python script(needs pycryptodome in order to work. You can install pycryptodome using pip install pycryptodome)
bc-debug - gradle project reproducing the problem

CodePudding user response：

There are two issues, one is a bug in your Kotlin code, the other is a library bug:

Bug in Kotlin code

While PyCryptodome processes ciphertext and tag separately, BC/Kotlin expects the concatenation of both in the order: ciphertext|tag.
Therefore the line encryptedMessage = tag must be added in the Kotlin code:

fun decryptAesMessage2(sharedKey: String, encryptedMessageData: Map<String, String>): ByteArray {
    var encryptedMessage = encryptedMessageData["encryptedMessage"]!!.utf8Base64Decode()
    var tag = encryptedMessageData["tag"]!!.utf8Base64Decode()
    encryptedMessage  = tag // Fix
    var nonce = encryptedMessageData["nonce"]!!.utf8Base64Decode()

    var key = KeyParameter(sharedKey.toByteArray(Charsets.UTF_8))
    var params = AEADParameters(key, tag.size*8, nonce)
    var cipher = OCBBlockCipher(AESEngine(), AESEngine())

    cipher.init(false, params)

    val out = ByteArray(cipher.getOutputSize(encryptedMessage.size))
    var offset = cipher.processBytes(encryptedMessage, 0, encryptedMessage.size, out, 0)
    offset  = cipher.doFinal(out, offset) // Throwing exception here

    return out
}

Test: Below, identical test data is successfully decrypted using the Python code and the fixed Kotlin code:

Python:

encrypted_message = {
  'encryptedMessage': 'LzoelJ9Nv4cruj0JUlxFrNR mqyO2rvwqDHYwnj0OkvJ BBvug ORYVkxA==', 
  'tag': 'hl56drXePWiLkVavVwF3/w==', 
  'nonce': b64encode(b'012345678901').decode()
}
dt = decrypt_aes_message('01234567890123456789012345678901', encrypted_message)
print(dt) # The quick brown fox jumps over the lazy dog

Kotlin:

val encrypted_message = mutableMapOf<String, String>()
encrypted_message["encryptedMessage"] = "LzoelJ9Nv4cruj0JUlxFrNR mqyO2rvwqDHYwnj0OkvJ BBvug ORYVkxA=="
encrypted_message["tag"] = "hl56drXePWiLkVavVwF3/w=="
encrypted_message["nonce"] = Base64.getEncoder().encodeToString("012345678901".toByteArray(Charsets.UTF_8))
val dt = decryptAesMessage2("01234567890123456789012345678901", encrypted_message)
println(String(dt, Charsets.UTF_8)) // The quick brown fox jumps over the lazy dog

Library bug

Another problem is that both implementations produce different results for a nonce length of 120 bits (15 bytes), the maximum allowed nonce length for OCB (see RFC 7253, 4.2. Encryption: OCB-ENCRYPT):

The following test uses a fixed key and plaintext and compares the results between the Python and BC/Kotlin code for nonce lengths 13, 14, and 15 bytes:

Key: b'01234567890123456789012345678901'
Plaintext: b'testmessage'

Nonce Länge     13 bytes                                                    14 bytes                                                    15 bytes
Nonce           b'0123456789012'                                            b'01234567890123'                                           b'012345678901234'
Python (ct|tag) 0xb35a69a245ab18fe3b6bae38b179c2a43b341f67c0451256b76bd7    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e
BC/Kotlin       0xb35a69a245ab18fe3b6bae38b179c2a43b341f67c0451256b76bd7    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e    0xa4355068324065f2ad194b058bdb86caa67c225b99021dbd588034

The Python implementation returns the same ciphertext/tag for nonce lengths 14 and 15, while the results for BC/Kotlin differ. This indicates a bug in the Python implementation.

Unfortunately, RFC 7253, Appendix A. Sample Results only provides test vectors whose nonces are all 12 bytes in size, so the bug cannot be assigned more clearly.

I.e. if you use a 15 bytes nonce, both implementations are not compatible, where the problem is most likely caused by the Python implementation.

Edit

Analysis of linked example:

You are affected by both bugs. According to your comment, you have already fixed the bug in the Kotlin code (concatenation of ciphertext and tag). Since the nonce you used in the example is 15 bytes (knQgYf1MsOs8smx9GtWM corresponds Base64 decoded to 0x92742061fd4cb0eb3cb26c7d1ad58c) the bug described in the second part of my answer is the cause of the problem. This bug is not in your code, but in one of the libraries, most likely in the Python library. Therefore you cannot fix it (at least not without greater effort).

Workarounds

As shown in the above test, the Python code for a nonce length of 15 bytes seems to simply ignore the 15th byte, i.e. if you use the 14 bytes nonce knQgYf1MsOs8smx9GtU= (0x92742061fd4cb0eb3cb26c7d1ad5) in your Python code, the Python code returns the same ciphertext and the same tag as with the 15 bytes nonce knQgYf1MsOs8smx9GtWM, which is why decryption with the 14 bytes nonce is also possible:

key = "f009Cip5hM4Obbb6E2MT5npJBHlc82vD"
message_data = {"encryptedMessage":"XMQx/xbVVTbMdpMiTXVp5XPICm11Vw2pgALpVI0NgbdqLLmikhPuu9M qQzyOVZlZZBRlscijpyAZDsLGcTSPP54O35oKNp//PuOrWsN/ZZMkCByKCSBysJLRiZV1OjZDg01gi5/nYNbUgGGd8uRGKfBaKjjXngZ1J89GOvDeWPQcjbfbdzd9w jbZGZ5jnAIChOL1Uqohf 6KHtjR/H06fFTHwB1abzAQrGbCNBNXBmN9 zEu7Auy3NPWKrZ SL5Nk=","tag":"ZcqXSBqYU5TjgdMC bMeUQ==","nonce":"knQgYf1MsOs8smx9GtU="}

decrypted_message = decrypt_aes_message (key, message_data)
print (decrypted_message) # https://app.passiv.com/snapTrade/redeemToken?token=v9uJsXYsi+6s9kyohisc6DFntJ/yD6m/2zhmO5xp6Vmezcyi8nwx63YtkqnnaogZvFmqs7L99EtZ0mxN9mAQTNoThHj3GaypXXUdiQIzig==&clientId=SCANZ&broker=ALPACA

If you use this 14 bytes nonce knQgYf1MsOs8smx9GtU= in the Kotlin code, the decryption is also successful. This is the workaround for this specific example!

The general workaround, as long as the library bug is not fixed, is not to use a 15 bytes nonce, but only a maximum 14 bytes nonce!

On this website there are also testvectors with 15 bytes nonces listed. As expected, these are not met by the PyCryptodome implementation. I have filed an issue on PyCryptodome based on one of the 15 bytes nonce test vectors: Issue #664.