WinAPI BCrypto RSA algorithm limits?-CodePudding

So here is a very simple code. Avoiding all the check and frees to keep it clean.

// the key is an RSA key 1024 bits
static const std::string PublicKey =
    "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfQ5ApaNvZN xAZhbqaSV ZAd"
    "N161lWDbQUQlKQdJgJtFQd22jyu0U7Nu88qQKV JTKNJgnegQ9U7vsTchE8gqcjp"
    "jLgTqId6DZWxZ5w41o0Dp/14Fkf3ixJulkT6kIiUavT5GaNM63maZ/KOlujxp4QZ"
    "Mnva1XIWVsV6t7/QOwIDAQAB";

std::string GenerateMessage(std::size_t len) {
    std::random_device rd;
    std::uniform_int_distribution<int> dist(0, 255);
    std::string result(len, '\0');
    for (std::size_t i = 0; i < len;   i) {
        result[i] = static_cast<char>(dist(rd));
    }
    return result;
}

bool Test() {
    auto pubKey = fromBase64(PublicKey); // CryptStringToBinaryA with flags = CRYPT_STRING_BASE64
    BCRYPT_KEY_HANDLE key = nullptr;
    CERT_PUBLIC_KEY_INFO* pubKeyInfo = nullptr;
    DWORD keyLength = 0;
    ULONG size = 0;
    ::CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO,
        pubKey.data(), static_cast<DWORD>(pubKey.size()),
        CRYPT_ENCODE_ALLOC_FLAG, nullptr,
        &pubKeyInfo, &keyLength);

    ::CryptImportPublicKeyInfoEx2(X509_ASN_ENCODING, pubKeyInfo, 0, 0, &key);
    std::vector<std::uint8_t> encrypted(128);
    std::string randomData = GenerateMessage(128);
    NTSTATUS status = ::BCryptEncrypt(key, (PUCHAR)randomData.data(), randomData.size(), nullptr, nullptr, 0, encrypted.data(), 128, &size, 0);
    return BCRYPT_SUCCESS(status) && (size == 128);
}

int main() {
    for (int i = 0; i < 1000; i  ) {
        std::cout << (Test() ? ' ' : '-');
    }
    std::cout << std::endl;
}

This code shows every time the Test is success and - when it's not. and the output I get is:

--   -            -             -                 --        -    -- -             -  -   - --         -  -   -   - --         -       - -                    -- -  -   -   -- -    -   -  --       - -            -        -        -   --     --  --  --         - -  - -          -        -  ---    - -     --       --       - -   -  -      -    -  -     -          -  -      -     -  -     -   -   --    -          -- -    -      -   - -   -          - -  --  -   -  - -    -- -  -   -     -  - -          -   --   -- --    -  - -      -  -     --     -  -    ---      -   -  -    -  -         -    -    -- - -    -   - -  - -      -      --   -    - -         - ---    - -     -            -               -   -----  ---   -               - -        -  --   -  -                 -      -      -     -    -  -       -  -     -  --       -   -  -    -     -      -        -     -          - -            -  --              --     -    - -  - -  - -      -   --  -        -         -- -   -- --       -

So it's quite random to fail. But if I change this part:

std::uniform_int_distribution<int> dist(0, 255);

with

std::uniform_int_distribution<int> dist('a', 'z');

It works well every time. No matter how many times I run it. Before the ::BCryptEncrypt there was ::CryptEncrypt and looks like it was working.

So the question: What do I wrong? Or maybe does RSA have some limits I don't know about?

Thanks.

UPD: status is always 0xC000000D (STATUS_INVALID_PARAMETER) in these cases

CodePudding user response：

RSA cannot encrypt data that is (when interpreted as a large integer) numerically greater or equal to the RSA modulus. Given that your key is 1024 bits and you're encrypting 128 bytes (1024 bits) the failing cases are most likely when the random data is numerically larger than your modulus. If you need to encrypt 128 bytes of data reliably, use a larger RSA key, otherwise you must reduce the amount of data being encrypted.

It works reliably when you change your distribution to only be a-z since the numeric values of these characters do not have their upper bits set, and so your message will be < 1024 bits when leading zero bits are ignored.

It would be remiss not to mention that "textbook RSA" as used here is not secure. See e.g.: https://crypto.stackexchange.com/a/1449/.