Unable To Generate Azure CustomerProvidedEncryptionKey (cpk)-CodePudding

I am attempting to move to Azure Blob Storage from AWS S3, but am facing some issues with generating the customer provided encryption key. For reference, in AWS, it is possible to have server-side encryption enabled without too much trouble (AWS Server-Side Encryption). In Azure, the same should be possible using a CustomerProvidedEncryptionKey.

The requirements from Microsoft to create CustomerProvidedEncryptionKey are as follows (Microsoft Docs on CPK):

key_value: str (Required)
Base64-encoded AES-256 encryption key value.

key_hash: str (Required)
Base64-encoded SHA256 of the encryption key.

However, when I create any 32 character long AES256 encryption key and the SHA256 hash of it, no matter how I generate or encode them I am unable to get the python SDK to accept them.

My work so far:

import hashlib
import base64
from azure.storage.blob import (
    ContainerClient,
    __version__,
    CustomerProvidedEncryptionKey,
)
import os

local_file_name = "sample.txt"
target_file_name = "sample-encrypted.txt"
container_name = "test"
connection_str = "<redacted>"

# Key and its Hash
key_value = b"32byteslongsecretkeyshallbegiven"
key_hash = hashlib.sha256(key_value).hexdigest().encode()

# Encode the key and its hash into base64
key_value_encoded = base64.b64encode(key_value)
key_hash_encoded = base64.b64encode(key_hash)

# Create cpk object to provide to Azure
cpk = CustomerProvidedEncryptionKey(key_value_encoded, key_hash_encoded)

# Create the BlobServiceClient object which will be used to create a container client
container_client = ContainerClient.from_connection_string(
    connection_str,
    container_name=container_name,
)

# Upload the blob
with open(local_file_name, "rb") as data:
    container_client.upload_blob(
        data=data, name=target_file_name, overwrite=True, cpk=cpk
    )

This will fail with the message:

ErrorCode:InvalidHeaderValue
headername:x-ms-encryption-key
headervalue:b'MzJieXRlc2xvbmdzZWNyZXRrZXlzaGFsbGJlZ2l2ZW4='

I was able to get the server side encryption with the client provided key to work ONLY when using the hard coded key and hash values that are in the test cases created by microsoft for the SDK (Azure SDK Test Case with hardcoded key).

TEST_ENCRYPTION_KEY = CustomerProvidedEncryptionKey(
key_value="MDEyMzQ1NjcwMTIzNDU2NzAxMjM0NTY3MDEyMzQ1Njc=",
key_hash="3QFFFpRA5 XANHqwwbT4yXDmrT/2JaLt/FKHjzhOdoE="
)

Note that their hard coded key value and key hash are 44 characters long, which means that in this case it is a 32 character string that is encoded into base64. This is curious for the hash, as a true SHA256 hash is 64 characters long, which would be even longer when encoded in base64. This means that some other (unknown) hashing algorithm was used to generate the key_hash.

If I don't provide a key in my code snippet above, the upload is successful, and if I provide the hardcoded key and its hash, the upload is successful and is encrypted. However, I am unable to find any way to generate a key and its hash to successfully upload and encrypt any blob.

Any direction would be appreciated!

CodePudding user response：

The issue with the code snippet was the encoding of the key hash. Since the hexdigest of the hash is a python string object that represents a hex string, we must take special care to decode it and treat its type as hex. Additionally, we must re-encode the base64 encoded strings into python string objects before passing it to the CustomerProvidedEncryptionKey.

See https://gist.github.com/CodyRichter/a18c293d80c9dd71a3905bf9c44e377f for the complete working code