step1 : Written data to data lake gen2 with pyspark

parquet format :

Item.write.format("parquet").mode("overwrite").save("abfss://[email protected]/parquet/Item/")

csv format:

Item.write.format("csv").mode("overwrite").save("abfss://[email protected]/csv/Item/")

Step2:

I see that blobs were created successfully

But event are not being sent to the event hub topic when these blobs are created.

The event is getting triggered when we manually uploaded a file or renamed existing file in the storage.

Event which got triggered for this

Summary: Events are not getting generated when an external application like spark writes into Blob. They are only triggering when file is uploaded/renamed/ deleted manually.

CodePudding user response：

Answering my own question,

This problem happened because the storage account was a Data Lake Gen1. I've mistakenly created Gen 1 instead of Gen2.

Upgrading the account to Gen2 resolved the issue.