I am trying to build an easy infra for the following scenario:
We have a remote edge device that collects data from IoT sensors and save the data as JSON payload (this will happen every 10 minutes) and generate a web URL - so the payload can be downloaded).
So, I want this data to be stored in Azure for further analytics and need to visualize the reports by using any visualization tools (Grafana is the most preferred one).
So, I need to stick with Grafana as the visualization tool, I think, I need to go with Azure data explorer to collect the data as Grafana has plugin for Azure data explorer only.
We have the same setup in AWS. And we are using Amazon Timestream to collect data and Grafana as data visualization tool. Grafana has plugin to connect to Timestream.
1. Anyone has any suggestions regarding the infra?
If I have to use Azure data explorer itself - I need to store the data to Azure blob storage first and then ingest the data from it to data explorer (for continuos ingestion) Does anyone know other ways to directly ingest the data from the web URL to data explorer?
If I have to use Azure blob storage - Does anyone knows how to automatically push the data from the web URL to the blob storage?
Thanks for all your suggestions in advance!
CodePudding user response：
Azure data explorer does not support direct ingestion from URLs, here are a few options:
- As you mentioned, write the data to Azure storage and setup an EventGrid connection (you can set it up in the portal or from code)
- Create an Azure function that will read the data and write it to Azure data explorer using one of its SDK (for example .Net, Python, etc.)
- Using Azure Data Factory copy data from HTTP activity, from there you can use Azure Data Explorer as the destination
CodePudding user response：
If your data lands in Azure blobs, you can set up an Event Grid data connection on your ADX instance that will ingest any newly created blob. Read about this here: Event Grid data connection.
To push your data to Azure blob storage, you can use FluentBit Azure blob output connector