What might make CryptStringToBinaryW crash?-CodePudding

I want to decode a string (which I got via readAsDataUrl()) to bytes.

At first, I remove the data:*/*;base64, then I call the following:

Option Explicit

Private Const CRYPT_STRING_BASE64 As Long = &H1&

Private Declare Function CryptStringToBinaryW Lib "Crypt32.dll" ( _
    ByVal pszString As Long, _
    ByVal cchString As Long, _
    ByVal dwFlags As Long, _
    ByVal pbBinary As Long, _
    ByRef pcbBinary As Long, _
    ByVal pdwSkip As Long, _
    ByVal pdwFlags As Long) As Long

Public Function DecodeBase64(ByVal strData As String) As Byte()
    Dim Buffer() As Byte
    Dim dwBinaryBytes As Long
    dwBinaryBytes = LenB(strData)
    ReDim Buffer(dwBinaryBytes - 1) As Byte

    'within the following call, VB6 crashes:
    If CryptStringToBinaryW(StrPtr(strData), LenB(strData), CRYPT_STRING_BASE64, _
        VarPtr(Buffer(0)), dwBinaryBytes, 0, 0) Then
        ReDim Preserve Buffer(dwBinaryBytes - 1) As Byte
        DecodeBase64 = Buffer
    End If
    Erase Buffer
End Function

Now I call this:

Dim s$
'code to get base64 string
'code to strip for example "data:image/jpeg;base64,"

Dim nBytes() As Byte
nBytes = DecodeBase64(s) 'Here VB6 crashes

Edit:

I am using the following alternative version now, and it works, but I wonder what the error is:

Public Function DecodeBase64(ByVal sBase64Buf As String) As Byte()
    
    Const CRYPT_STRING_BASE64 As Long = 1
    Const CRYPT_STRING_NOCRLF As Long = &H40000000
    Dim bTmp() As Byte
    Dim lLen As Long
    Dim dwActualUsed As Long
    'Get output buffer length
    If CryptStringToBinary(StrPtr(sBase64Buf), Len(sBase64Buf), CRYPT_STRING_BASE64, StrPtr(vbNullString), lLen, 0&, dwActualUsed) = 0 Then
        'RaiseEvent Error(Err.LastDllError, CSB, Routine)
        GoTo ReleaseHandles
    End If
    'Convert Base64 to binary.
    ReDim bTmp(lLen - 1)
    If CryptStringToBinary(StrPtr(sBase64Buf), Len(sBase64Buf), CRYPT_STRING_BASE64, VarPtr(bTmp(0)), lLen, 0&, dwActualUsed) = 0 Then
        'RaiseEvent Error(Err.LastDllError, CSB, Routine)
        GoTo ReleaseHandles
    Else
        'm_bData = bTmp
    End If
ReleaseHandles:
    DecodeBase64 = bTmp

End Function

Edit:

In version 1, dwBinaryBytes is 156080 in this line:

dwBinaryBytes = LenB(strData)

and in version 2, lLen is 58528 in this line:

ReDim bTmp(lLen - 1)

Why the discrepancy, and why didn't the author notice that?

CodePudding user response：

The "CryptStringToBinaryW" requires the number of characters in the string as a parameter. That is returned by the "Len" function. You used the "LenB" function which returns the number of bytes in the string which is larger than the number of characters in the string so the function attempted to access memory past the end of the string which caused the crash.