I am having a below API

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=0

Have to extract more data by increasing the offset into 2000

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=2000

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=4000

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=6000

I have a maximum of 6000 records. I don't know how to pass the offset value of every 2000 in the data factory

I can able to extract individually with the above links but wanted to do it automatically for all 6000 records.

Can anyone show me some documentation or advise how to execute this in the datafactory?

I saw the pagination documentation, but no success

CodePudding user response：

Extracting data using copy activity from rest API

Step1: Create a new pipeline and add a Copy Data activity.

Step2: Configure the source of copy activity, adding a pagination rule configured as below

Follow this sample

Pagination rule either select option1 or option2 .In my case I selected Range:0:8:2

In your scenario you can follow the range as below :

• Option1: QueryParameter.{offset}: Range:0:6000:2000

• Option2: AbsoluteUrl.{offset}:Range:0:6000:2000

Range option are using 0 as the start value,6000 as the max value, and increasing by 2000 each time.

Storage account

For more detail information refer this official article.