How long can an Azure Durable Function run for?-CodePudding

I have multiple ETL kind of tasks, that I plan to perform serverless. The execution time for the tasks vary from 5 to 30 minutes (depending on the amount of data coming at an instance). Since functions have a timeout of 10 minutes, these tasks cannot be performed together in one single function. I recently came across Durable Functions in Azure for orchestration of different functions. I wanted to know if Durable functions altogether also have a timeout of 10 minutes, or I can have multiple functions in it (which run from 3-5mins each).

For example, task 1 takes 3mins, task 2 takes 5 mins, task 3 takes 7 minutes, task 4 takes 3minutes and task 5 takes 2mins. Can I have all these tasks orchestrated in a single durable function?

My current approach is to have a queue trigger function for each of the tasks separately, but this kind of workflow is quiet a mess. I feel durable functions will be best for making a streamlined workflow.

CodePudding user response：

Can I have all these tasks orchestrated in a single durable function?

In short: yes. But here's some context.

Durable Functions is an extension of Azure Functions that lets you write stateful functions in a serverless compute environment. The extension lets you define stateful workflows by writing orchestrator functions and stateful entities by writing entity functions using the Azure Functions programming model. Behind the scenes, the extension manages state, checkpoints, and restarts for you, allowing you to focus on your business logic.

It sounds like your scenario would best fit the Function Chaining pattern.

Function chaining

In the function chaining pattern, a sequence of functions executes in a specific order. In this pattern, the output of one function is applied to the input of another function.

The code for this is relatively simple:

[FunctionName("Chaining")]
public static async Task<object> Run(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    try
    {
        var x = await context.CallActivityAsync<object>("F1", null);
        var y = await context.CallActivityAsync<object>("F2", x);
        var z = await context.CallActivityAsync<object>("F3", y);
        return  await context.CallActivityAsync<object>("F4", z);
    }
    catch (Exception)
    {
        // Error handling or compensation goes here.
    }
}

Each separate piece of functionality in this example (F1 to F4) is it's own Function. Which means it has its own timeout.

Please take into account that a different type of hosting plan might be a better suited solution for your problem. Have a look at Azure Functions hosting options - Service limits to see what the options and their respective (timeout) limits are.

CodePudding user response：

By default, functions running in the Consumption plan have a timeout of five minutes. If this limit is exceeded, the Azure Functions host is recycled to stop all execution and prevent a runaway billing situation. The function timeout is configurable.

Reference for Durable functions.

Azure function that runs for less that 60 minutes. azure Function in premium plan supports guaranteed 60 minutes.However for long running scenarios you can use Durable Function which are intented to solve complex scenarios which lets you split up your jobs into smaller junks. Functions are designed to be short-lived and run for a limited period of time. Functions excel in short-duration executions with low or unpredictable throughput.

Refactor huge functions into smaller function sets that work together and produce replies quickly wherever possible. A webhook or HTTP trigger function, for example, may need an acknowledgment response within a specific time limit; webhooks frequently demand an immediate response. The HTTP trigger payload can be placed in a queue and processed by a queue trigger function. This method allows you to postpone the actual task and respond quickly.

Take a look at the following:

https://docs.microsoft.com/en-us/azure/azure-functions/functions-best-practices#avoid-long-running-functions

With Durable Functions you can easily support long-running processes, applying the Async HTTP APIs. When in case you are dealing with functions that require some time to process the payload or request, running under an 'App Service Plan, WebJob, or Durable Functions' is the right way.