A Practical Guide to Azure Durable Functions — Part 4: Retry
Everything you need to know to create a production-ready function app via Azure Durable Functions.
This part 4 of the series is all about retry mechanisms of Azure Durable Functions. The sample code in this story takes a very simple approach to set configurations. If you’re interested in a more sophisticated configuration set up, you may want to check out the previous story.
There are mainly 3 retry mechanisms available for durable functions.
- Retry via service bus.
- The build-in activity/sub-orchestrator invocation with retry.
- The build-in sub-orchestrator retry.
We haven’t talked about sub-orchestrators at all in series before. It’s a very powerful feature of durable function. We will get to it, bear with me.
Our demo app is still the good old GitHub view counter. In a real application, you probably want to retry any web API calls because:
- the API might be rate-limited to protect itself from DDOS attacks.
- the network might not the stable at times.
To simulate a rate-limited API endpoint, I made the GetRepositoryViewCount activity function randomly throw a custom exception.
If we run it locally with this code change, it sometimes will fail.
That’s good. Now we need to retry the API call.
Service Bus Retry
Strictly speaking, this has nothing to do with Azure durable functions. The retry happens on the Azure Service Bus.
The code to implement it is the easiest of the three: do nothing. But is an easy approach always a good approach?
Retry mechanisms via message queues have been around for many years and are proven to be reliable. However, I do not recommend to use a service bus trigger to add resiliency to the Azure durable function apps.
Keep in mind that the service bus trigger is just one kind of Azure Function triggers. If your app’s resiliency relies on it solely, other triggers have to relay their requests to the service bus. The orchestrator effectively only takes real requests from the service bus trigger.
This might be the only option, and a limitation really, for a simple Azure Function app. But we certainly don’t have to rely on the service bus for resilience with durable functions. There are better build-in retry mechanisms.
Function Invocation With RetryOptions
Thus far, in the sample code of this series, we had been using CallActivityAsync to call the activity functions. When the activity function throws an exception, the orchestration fails and exists.
But there are two other methods on the IDurableOrchestrationContext interface available with RetryOptions:
We will get to the sub-orchestrator a bit later. Let’s improve our existing code by replacing the CallActivityAsync calls with CallActivityWithRetryAsync calls.
In this simple demo app, we can define a single RetryOptions and let all activity function share it.
This means that the first retry should happen 2 seconds after the exception, the maximum number of retry should be 5, and we only want to retry on the TooManyRequestsException exception type.
We can now replace the activity function call with the retry version in the orchestrator.
If we run the app now, and if we are not extremely unlucky (there’s a small chance that the random number is always greater or equal to 7 with these many retries), the exception still happens, but the orchestrator should finish successfully with retries.
This is, IMHO, better than the service bus retry. It doesn’t need another Azure resource (Azure Service Bus) and it can be configured just the way you want it.
The sub-orchestrator call with retry works very similarly to the activity call.
However, if you need to implement even more complex logic, there’s another retry option with sub-orchestrators.
Sub-orchestrator Retry
This option is only available via sub-orchestrators. So what is a sub-orchestrator?
You might have guessed it from the name. When an orchestrator A can call another orchestrator B, we start to call B a sub-orchestrator.
From the coding’s perspective, there’s nothing special about orchestrator B. It’s just a regular orchestrator function. It only becomes a sub-orchestrator when called from A.
Now interesting thing happens when you chain the orchestrators like this:
A => B => C => AYes, durable functions can be recursive! Now you should have no problem dealing with the tree data structure with durable functions. Of course be careful when you design something like this, just like designing any other recursive methods.
Sub-orchestrators not only make durable functions very powerful and flexible but also provide another retry mechanism with better control in the code.
Firstly, the main orchestrator in the demo app should not call the activity directly anymore. It calls a sub-orchestrator instead.
Note we do not call the sub-orchestrator with the WithRetry method. The retry happens inside the sub-orchestrator.
The newly created sub-orchestrator will call the activity function.
The most important line is context.ContinueAsNew(updatedRepoName);
ContinueAsNew lets the orchestrator reset current state (durable functions are stateful, remember?) and restart with the given input parameter.
In the code above, a random delay between 10 and 30 seconds is added before the retry. A method to update the input parameter is called to demonstrate that you can update the input before the retry.
Compare this option with the previous one, we can see that this allows much more complex retry logic to be implemented. And it does not need a max number of attempts. You can retry forever.
To verify it works, we run the updated app locally.
Summary
This story shows three retry options of Azur durable functions. And why you probably shouldn’t rely on the service bus for resilience. Please do not rush to the conclusion that the Azure service bus is useless. It’s still a good option to integrate with other parts of your system.
When your requirements are relatively simple, go for the function invocation with retry approach. It’s simple to set up and clear to read.
However, if your scenario requires complex logic to be implemented with the retry, the sub-orchestrator approach might be more suitable.
Series Summary
Does this series fulfill the promise made on the subtitle?
Everything you need to know to create a production-ready function app via Azure Durable Functions.
Almost. The series lays out the foundations for a developer or software engineer to create a complex piece of serverless software that meets business needs.
The only thing left for the readers to explore before it’s really production-ready is the devops aspect. The series touched it a bit but didn’t go into depth. It doesn’t feel right to cover a full Azure Devops pipeline set up here. It deserves its own series. And the reader might have other preferred devops tools. So I’ll leave it for you to figure out how it should be done.
This is the last story of the Azure Durable Function series. I probably will write about Azure Function again in the future. Follow me for new stories.
