I'm trying to encrypt with private key and decrypt with public key, with RSA_PKCS1_PADDING as padding. The encryption works fine, but when I do the decryption I got an invalid padding error:

processed 9 of 256 bytes, RSA_public_decrypt() error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Anyone know what's wrong here? Attached full source code:

void encrypt_stdout(const char *from_file, int to_base64, int padding)
{
    unsigned char *input = NULL, *output = NULL, *output2 = NULL;
    int output_fd = -1, input_len = 0, output_len = 0, output2_len = 0;

    if (readfile(from_file, &input, &input_len) != 0)
    {
        goto end;
    }

    output = malloc(RSA_size(rsa));
    if (output == NULL)
    {
        fprintf(stderr, "malloc() on output: %s\n", strerror(errno));
        goto end;
    }

    output_len = RSA_private_encrypt(input_len, input, output, rsa, padding);
    if (output_len == -1)
    {
        fprintf(stderr, "RSA_private_encrypt() %s\n",ERR_error_string(ERR_get_error(), errbuf));
    }
    else
    {
        if (to_base64)
        {

        }

        write(1, output, output_len);
    }

end:
    if (input != NULL)
    {
        free(input);
    }

    if (output != NULL)
    {
        free(output);
    }   
}

void decrypt_stdout(const char *from_file, int is_base64, int skip_bytes, int padding)
{
    unsigned char *input = NULL, *output = NULL, *output2 = NULL;
    int output_fd = -1, input_len = 0, output_len = 0, output2_len = 0, total_read = skip_bytes;

    if (readfile(from_file, &input, &input_len) != 0)
    {
        goto end;
    }

    if (is_base64)
    {
        if (base64_decode(input, input_len, &output2, &output2_len) != 0)
        {
            if (output2 != NULL) 
                free(output2);
            goto end;
        }

        free(input);
        input     = output2;
        input_len = output2_len;
    }

    output = malloc(RSA_size(rsa));
    if (output == NULL)
    {
        fprintf(stderr, "malloc() on output: %s\n", strerror(errno));
        goto end;
    }

    while (total_read < input_len)
    {
        memset(output, 0, RSA_size(rsa));

        output_len = RSA_public_decrypt(RSA_size(rsa), input   total_read, output, rsa, padding);
        if (output_len == -1)
        {
            fprintf(stderr, "\nprocessed %d of %d bytes, RSA_public_decrypt() %s\n", total_read, input_len, ERR_error_string(ERR_get_error(), errbuf));
            break;
        }
        else
        {
            write(STDOUT_FILENO, output, output_len);
        }

        total_read  = output_len;
    }

end:
    if (input != NULL)
    {
        free(input);
    }

    if (output != NULL)
    {
        free(output);
    }
}

CodePudding user response：

The problem is caused by the while loop in decrypt_stdout().

To me the sense of this loop is not clear. Instead of the loop it should be quite analogous to encrypt_stdout():

output_len = RSA_public_decrypt(input_len, input, output, rsa, padding);
if (output_len == -1)
{
    fprintf(stderr, "RSA_public_decrypt() %s\n", ERR_error_string(ERR_get_error(), errbuf));
}
else
{
    _write(1, output, output_len);
}

With this, decryption works on my machine.

However, if the loop is used, the first decryption succeeds and returns the plaintext, while the subsequent decryption fails and returns a -1, resulting in the output of the error message.

A few notes on the logic used in the code:

Since you talk about encryption and decryption: RSA_private_encrypt() and RSA_public_decrypt() are not really meant for encryption and decryption, but for low level signing and verification.

Typically, when signing with RSA_private_encrypt(), it is not the data itself that is passed, but a hash of the data prefixed with a digest ID (more precisely, the DER encoding of the DigestInfo value). In the posted code, there is neither hashing nor prepending a digest ID. Of course, the loaded file might already contain the concatenation of digest ID and hash, but that cannot be said without sample data. If this is not the case, the generated signature is not compliant with PKCS#1 v1.5 padding.

If no hashing takes place, care must be taken that the length criterion is met, i.e. the data to be signed must be smaller than the key size minus the minimum space required by the padding (11 bytes for PKCS#1 v1.5 padding or flag_padding = 1).

For completeness: PKCS#1 v1.5 padding is determinstic (in the context of signing). The padded data looks like this: 0x00 || 0x01 || PS || 0x00 || T, where PS consists of so many 0xff values that the size is equal to the key size. T is the data or, if compliant with the specification, the concatenation of digest ID and hashed data, s. RFC8017.