I am trying to build a fully parametrised pipeline template in ADF. With the work I have done so far, I can do a full load without any issues but when it comes to delta load, it seems like my queries are not working. I believe the reason for this is that my "where" statement looks somewhat like this:
SELECT @{item().source_columns} FROM @{item().source_schema}.@{item().source_table}
WHERE @{item().source_watermarkcolumn} > @{item().max_watermarkcolumn_loaded} AND @{item().source_watermarkcolumn} <= @{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}
where the 'max_watermarkcolumn_loaded' is a datetime format and the 'activity' output is obviously a string format.
Please correct me if my assumption is wrong and let me know what I can do to fix.
ADF is picking a date from SQL column 'max_watermarkcolumn_loaded' in this format '"2021-09-29T06:11:16.333Z"' and I think thats where the problem is.
CodePudding user response:
I tried to repro this error. I gave the parameter without single quotes to a sample Query.
Wrap the date parameters with single quotes.
Corrected Query
SELECT @{item().source_columns} FROM
@{item().source_schema}.@{item().source_table}
WHERE @{item().source_watermarkcolumn} >
'@{item().max_watermarkcolumn_loaded}' AND
@{item().source_watermarkcolumn} <=
'@{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}'
With this query, pipeline is run successfully.