I'm trying to do an activity in Azure Data Factory. It consists of reading data from an API, and at each reading, I would add 1 in the value of the variable until I reach a certain value, as shown in the screenshot.

URL:

However, the variable never goes beyond the value 2, so I get stuck in a loop. Does anyone have any idea how to fix that?

CodePudding user response：

You might want to try to put the Web Activity inside the Until Activity. As the webservice is open and can be accessed without authentication, I made an example that worked in my environment.

{
    "name": "Get_NBA_Stats",
    "properties": {
        "activities": [
            {
                "name": "Retrieve all NBA results",
                "type": "Until",
                "dependsOn": [
                    {
                        "activity": "Set startpage",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "expression": {
                        "value": "@equals(activity('Get page from BallDontLie_io').output.meta.total_pages, activity('Get page from BallDontLie_io').output.meta.current_page)",
                        "type": "Expression"
                    },
                    "activities": [
                        {
                            "name": "Get page from BallDontLie_io",
                            "type": "WebActivity",
                            "dependsOn": [],
                            "policy": {
                                "timeout": "7.00:00:00",
                                "retry": 0,
                                "retryIntervalInSeconds": 30,
                                "secureOutput": false,
                                "secureInput": false
                            },
                            "userProperties": [],
                            "typeProperties": {
                                "url": {
                                    "value": "@Concat('https://www.balldontlie.io/api/v1/games?per_page=100&page=',variables('page'))",
                                    "type": "Expression"
                                },
                                "method": "GET"
                            }
                        },
                        {
                            "name": "Set next page",
                            "type": "SetVariable",
                            "dependsOn": [
                                {
                                    "activity": "Get page from BallDontLie_io",
                                    "dependencyConditions": [
                                        "Succeeded"
                                    ]
                                }
                            ],
                            "userProperties": [],
                            "typeProperties": {
                                "variableName": "page",
                                "value": {
                                    "value": "@string(activity('Get page from BallDontLie_io').output.meta.next_page)",
                                    "type": "Expression"
                                }
                            }
                        }
                    ],
                    "timeout": "7.00:00:00"
                }
            },
            {
                "name": "Set startpage",
                "type": "SetVariable",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "variableName": "page",
                    "value": "1"
                }
            }
        ],
        "variables": {
            "page": {
                "type": "String",
                "defaultValue": "1"
            }
        },
        "annotations": [],
        "lastPublishTime": "2021-10-25T13:32:04Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

1. Set page to the default ( webservice start pagination at 1 ).
2. Set condition when the loop should stop (meta.total_pages equals meta.current_page).
3. Retrieve all pages in a loop.
4. Update the page variable with meta.next_page

You might want to check the Expression if it stops in time, and does not continue looping and rack up your Azure bill.

CodePudding user response：

The trouble with the

Here I use a single call with the Web activity to get the total_pages, but the Copy activity is doing the main work.

@string(activity('Web Get Ballgames API').output.meta.total_pages)

I also set the Pagination rules to pick the next page automatically:

AbsoluteUrl:  $.meta.next_page