Parsing files DAT files, CSV files and Image files using Azure services-CodePudding

I have 5 types of EDI files namely - *.DAT, *.XML, *.TXT, *.CSV and Image files which contains data in them whose data are not in standard format. I need to parse them and extract required data from them and persist them in SQL Database. Currently, I'm spending time writing parser class libraries for each type of EDI file and not scalable .

I need to know if there are any azure services which can do the parsing work for me and is scalable.

Can I expect a solution on this regards?

CodePudding user response：

Consider using Azure Data Factory. It supports a range of file types.

CodePudding user response：

I need to parse them and extract required data from them and persist them in SQL Database.

Yes, You can use Azure Functions to process a Files like CSV an import data into Azure SQL Or Azure Data Factory is also helpful to read or copy many file formats and store them in SQL Server Database in specified formats, there is an practical example provided by Microsoft, Please refer here.

To do with Azure Functions, the following steps are:

Create Azure Functions (Stack: .Net 3.1) of type Blob Trigger and define the local storage account connection string in local.settings.json like below:

enter image description here

In the Function.cs, there will be some boilerplate code which gives the logic of showing the uploaded blob name and its size.

enter image description here

In the Run function, you can define your parsing logic of the uploaded blob files.

Create the Azure SQL Database, configure the server with location, pricing tier and the required settings. After that, Select Set Server Firewall on the database overview page. Click Add Client IP to add your IP Address and Save. Test the database whether you're able to connect.
Deploy the project to Azure Functions App from Visual Studio.
Open your Azure SQL Database in the Azure portal and navigate to Connection Strings. Copy the connection string for ADO.NET.
Paste that Connection String in Azure Function App Settings in the portal.
Test the function app from portal and the remaining steps of uploading files from storage to SQL Database were available in this GitHub documentation

Also for parsing the files like CSV, etc. to JSON Format through Azure Functions, please refer here.