I am getting error while I am reading Parquet file created by Databricks on ADLS. While I read these files using Databricks it works perfectly fine and I am able to read and write data into these files from Databricks. However with DataFactory it is giving below error.

Error: Parquet file contained column 'txn', which is of a non-primitive, unsupported type.

However there is no txn column create by me from Databricks.

CodePudding user response：

This error mainly happens because of unsupported data type. When you pass to the column in parquet file make sure you are using the supported datatype.

Supported datatype mapping for parquet file refer this Microsoft document .

CodePudding user response：

When you have complex data types in your parquet source files in ADF, you need to use a data flow without a dataset schema. Then you can work with structs, maps, arrays, etc: https://www.youtube.com/watch?v=Wk0C76wnSDE