Scaling an ECS Service Based On AWS SQS Queue Items
This article will help you scaling your ECS Service based on the number of items in an AWS SQS items.
Scaling an ECS service might be necessary when you have a script or service processing a queue that has to deal with peak moments. You might want to have more processes picking items from the queue in parallel, so the queue is processed faster.
That was something I had to get done recently and it was not without a struggle, hopefully you will have less trouble with it after reading this article. After all it wasn’t that difficult.
I will be using Terraform in this article, but you can of course create these resources in the AWS console manually. If you are not familiair with it (yet), Terraform is an Infrastructure as Code (IaC) software tool, allowing you to define and setup all resources that you need in a cloud environment as code, without having to click through the console of the specific cloud provider.
Prerequisites
- The ECS Service you want to scale
- A SQS Queue
What We Are Going to Do
We are going to scale our service based on the number of items in the queue.
- If there are 0 items in the queue, the number of tasks has to be set to 0.
- If the number of queue items is > 0 and ≤ 500, the number of tasks has to be set to 1.
- If there are > 500 and ≤ 1000 queue items, scale up to 2 tasks.
- If there are > 1000 queue items, scale up to 3 tasks.
The Terraform Script
From now on I will be discussing different parts of our Terraform script. You can find my complete Terraform script at the end of the article, which will also include resources for a task definition, ECS service et cetera.
Autoscaling target
You have to add an auto scaling scaling target pointing to your existing ECS service. Like I stated, this service can also be set up using Terraform, but you could also create this manually using the AWS console.
In the autoscaling target, you can provide the minimum and maximum capacity of the dimension you would like to scale. In our case, we want to scale the ecs:service:DesiredCount which indicates how many tasks should be running within our service. If the number of running tasks does not meet the desired count, ECS will launch the required amount of tasks.
Autoscaling Policies
After setting up the autoscaling target, we can begin defining our scaling policies.
Policy for Scaling up
In the autoscaling policy for scaling up our service, we define that we want adjust the ExactCapacity of the DesiredCount (we refer to the scalable dimension of the autoscaling target defined above) when a particular metric meets certain conditions.
We want our desired count to be 1 when there are between 1 and 500 items in our queue, 2 when there are between 500 and 1000 items in our queue and 3 when there are more than 1000 items in the queue. Please note: lower bounds are ‘greater than’ and upper bounds are ‘less than or equals’.
Policy for Scaling down
We also want to scale down the desired count to 0 when there are no items in our queue, so we also add an autoscaling policy for scaling down.
CloudWatch Metric Alarms
We now have defined our policies, but there is no trigger for these policies yet. We have to trigger the policies with CloudWatch metric alarms. In our case, these metric alarm constantly look at the number of items in our queue and go in alarm when it passes the threshold we defined.
We can set an action when this metric alarm triggers and it will pass through the value of the metric to the action. The metric we are watching is the ApproximateNumberOfMessagesVisible metric of our SQS queue. This is a metric automatically provided by AWS when you create a SQS queue.
Scale Out
We create a scale out metric alarm that goes in alarm when the ApproximateNumberOfMessagesVisible of our queue is 1 or above. The alarm action is our scale up policy, which will then determine what to do based on the value.
Scale In
We do the same for scaling in, we want to trigger our scale down policy when the number of items in our queue is 0.
That Is Quite About It!
This is what I have done to scale an ECS service based on the number of items in a SQS queue. I did not explain every setting and you may have to tweak the scripts to your needs. You can also look into dynamically create the step_scaling_policy_configuration, so you do not have to provide a whole list of step adjustments in the autoscaling policy.
Questions, Suggestions or Feedback
If you have any questions, suggestions or feedback regarding this article, please let me know!
Final Words
I want to thank Redditor JohnPreston72, who was so kind to provide some CloudFormations scripts. Based on the scripts provided by u/JohnPreston72, I was able to set up the scaling myself and write my own Terraform script (see the complete script below).
