Best design with fault tolerant for retrieving updates periodically invoking an API-CodePudding

I am a client say clientA, I will need to call a server say serverA in a periodic manner every 15 minutes to get new updates from serverA.

ServerA is only capable to expose an API which intakes 2 parameters - Start Time and End Time and returns list of newly created account details.

Example serverA exposes: REST GET Api looks something like this

http://somedomainname.com/getAccounts?startTime={say currentTime-15minutes}&endTime={say currentTime}

Response:

{[
123,
456,
789
....
]} // List of account numbers that was created in serverA between supplied start time and end time.

As clientA, I will have to call above API periodically (say every 15 minutes) and store the updates in my client database. What would be the best approach to design my clientA system invoking serverA API, retrieving updates and storing in my client database considering the fault tolerance like if serverA is down for any reason or clientA itself is down or any other un-expected error scenarios. I wanted to as well avoid duplicate calls in the retry mechanism.

It would be great to hear your opinion on this. My clientA to be written in Java language with Spring Boot framework

CodePudding user response：

In my experience if it's possible to sync data between systems better to use push instead of pull approach. Pulling updates from API especially by multiple clients can quickly lead to problems with API performance. Contrary push approach scale better - any number of clients can subscribe to update and publisher of updates can send them in convenient time.

But coming back to you questions. I any case you need some sort of anchor to keep track when was last successful retrieval of updates. In very simple scenario - store in the DB table with one field e.g. LastSuccesfullAccountUpdateCallDatetime.If call succeed update this field. If call fails you don't update it. Next time you client knows where to start. In more complicated scenario in that table you can store all the time periods used to retrieve account update. E.g.

ID StartTime           Success
1  01-01-2021 00:00:00 true
2  01-01-2021 00:15:00 false
3  01-01-2021 00:30:00 true

Those time periods could be generated upfront or just inserted by client as a result of call.

Then you can link this Id to Account table and perform reconciliation for every time period if needed. Let's for TimePeriod.Id = 1 you have in the DB accounts : 123,456 but later turned out that there were more. You make a call with time stored in the DB, got e.g. 123,456,789 and you see that 789 should be added.