EVP_DecryptUpdate is giving segmentation fault-CodePudding

In c while using the openssl/evp.h library.

If I am doing

EVP_CIPHER_CTX_new();
EVP_DecryptInit(ctx, EVP_aes_256_wrap_pad(), NULL, key, iv);
EVP_DecryptUpdate(ctx, buf, &cipher_len, 32);

I am getting a seg fault in the last line but when I change the EVP_aes_256_wrap_pad() to EVP_aes_128_ecb() and keeping all other parameters same seg fault is gone.

Here is an example code Working

#include <openssl/evp.h>
#include <stdio.h>
#include <string.h>

void dump_head(unsigned char *buf, size_t len)
{
    unsigned end, i;
    for (end = len; end > 0; end--)
        if (buf[end-1] != 0)
            break;
    printf("buf = {");
    for (i = 0; i < end; i  )
        printf(" hhx,", buf[i]);
    printf(" }\n");
}

int main(void)
{
    unsigned char key[] = "0123456789abcdef";
    unsigned char iv[] = "1234567887654321";
    unsigned char indata[32] = "0123456789abcdeffedcba9876543210";
    unsigned char buf[4096];
    unsigned pos;
    int cipher_len;
    EVP_CIPHER_CTX *ctx;

    ctx = EVP_CIPHER_CTX_new();
    EVP_DecryptInit_ex(ctx, EVP_aes_128_ecb(), NULL, key, iv);
    
    EVP_DecryptUpdate(ctx, buf, &cipher_len, indata, 32);
    
    printf("Got %d\n", cipher_len);
    dump_head(buf, sizeof(buf));
        
    printf("Final!\n");
    memset(buf, 0, sizeof(buf));
    EVP_DecryptFinal_ex(ctx, buf, &cipher_len);
    printf("Got %d\n", cipher_len);
    dump_head(buf, sizeof(buf));

    return 0;
}

Not working

#include <openssl/evp.h>
#include <stdio.h>
#include <string.h>

void dump_head(unsigned char *buf, size_t len)
{
    unsigned end, i;
    for (end = len; end > 0; end--)
        if (buf[end-1] != 0)
            break;
    printf("buf = {");
    for (i = 0; i < end; i  )
        printf(" hhx,", buf[i]);
    printf(" }\n");
}

int main(void)
{
    unsigned char key[] = "0123456789abcdef";
    unsigned char iv[] = "1234567887654321";
    unsigned char indata[32] = "0123456789abcdeffedcba9876543210";
    unsigned char buf[4096];
    unsigned pos;
    int cipher_len;
    EVP_CIPHER_CTX *ctx;

    ctx = EVP_CIPHER_CTX_new();
    EVP_DecryptInit_ex(ctx, EVP_aes_128_wrap_pad(), NULL, key, iv);
    
    EVP_DecryptUpdate(ctx, buf, &cipher_len, indata, 32);
    
    printf("Got %d\n", cipher_len);
    dump_head(buf, sizeof(buf));
        
    printf("Final!\n");
    memset(buf, 0, sizeof(buf));
    EVP_DecryptFinal_ex(ctx, buf, &cipher_len);
    printf("Got %d\n", cipher_len);
    dump_head(buf, sizeof(buf));

    return 0;
}

Can you please suggest is there anything more that I have to add for EVP_aes_256_wrap_pad().

CodePudding user response：

It's not documented that I can find, and may not actually be supported, but to use a wrap-mode cipher in EVP_{Cipher,Encrypt,Decrypt}* you must first set a flag on the context:

EVP_CIPHER_CTX_set_flags(ctx, EVP_CIPHER_CTX_FLAG_WRAP_ALLOW);

and then use Init_ex as you did.

Since you didn't set this, your call to DecryptInit_ex failed, but you ignored the return code, and then called DecryptUpdate on an uninitialized context, which crashes. You should always check the return code from any OpenSSL routine that returns one.

However even when I set this flag, DecryptUpdate returns 0, I suspect because your data is not valid ciphertext for this algorithm (the keywrap algorithms include an integrity check), although I would expect it to put something in the errorstack and it doesn't. I will look more deeply at that if I have time.