Trigger Events at a Specific Timestamp
Using AWS Step Functions, API Gateway, and Lambda, schedule events to be executed at a particular timestamp without the need to do any polling — No Polling At All!
An extremely common use case for a lot of applications is the need to have scheduled events. There are majorly two kinds of scheduled events — ones that run as CRON jobs, and others that need to be triggered at a specific time.
CRON jobs are software utilities (generally commands or shell scripts) that run periodically at fixed times, dates or intervals. They can be one-off or repetitive. Scheduled Triggers on the other hand are events that need to be fired at a very specific time for which the time is generally expected to be dynamic or configurable based on input.
The most common approach to solve the scheduled trigger problem is by doing something called — Polling. Polling is when we trigger the job every x units of time and check whether the execution time is the current time or if it has already passed. While this may work for simpler, smaller use cases, it is extremely difficult to scale. Immediately it is evident that the precision of this job depends on the interval it is called at. Lower the gap, more accurate and close to the scheduled time the function is fired. But this brings with itself the problem of wastage of resources and computing power, and if there’s a database involved, unnecessary reads from the database extremely frequently.
We need a way that whenever we receive an event with a future timestamp of execution, it automatically enters a WAITING state and is triggered when the timestamp has occurred without our intervention. This fortunately is the exact use case for which Step Functions were created a.k.a. State Machines in the Cloud.
AWS Step Functions is a low-code visual workflow service used to orchestrate AWS services, automate business processes, and build serverless applications. Workflows manage failures, retries, parallelisation, service integrations, and observability so developers can focus on higher-value business logic.
In this example, we’ll learn how to create a Step Function that receives events through an API Gateway, waits until the execution time and then triggers a Lambda function to perform the task.
This tutorial uses Node.js and the Serverless Framework and in order to follow along you will need to have Node.js and NPM installed on your system. Along with that, make sure that you also have the serverless NPM package installed globally. If not, run the following command and setup your AWS profile post installation using this link —
npm install -g serverlessIf you have worked with Lambda Functions and AWS in the past using Node.js, this initial part of the setup and codebase might be familiar to you. We’re just going to extend the standard Lambda Functions template to include support for Step Functions.
To get started we need to install the dependency for step functions —
npm i --save-dev serverless-step-functionsThis package allows us to use the stepFunctions keyword in our serverless.yml which looks something like this after all the configuration is complete.
Let’s walk through the sections in the serverless.yml file —
- provider — This holds all the configuration regarding deploy region, cloud provider, runtime details, stage, etc.
- package — This section includes information about how our code should be packaged and built and includes options for individually packaging in case you have multiple functions, include/exclude options, etc.
- functions —
The definition for our Lambda functions sits here. It needs the handler value to point to the file in our directory. There are additional options like memory, timeout, retryAttempts which allow for further customisation of our Lambda functions.
- stepFunctions —
This is a custom property that we are able to use because of the package we installed previously. This holds the definition of our Step Function written in YAML. The events object defines the triggers for our Step Function which in our case is an HTTP POST request.
The definition of the Step Function holds the different states and connections of the Step Function. This comes from the Amazon States Language (ASL) which is JSON-like language used by Amazon to define Step Functions. A detailed write-up and available states and choices can be found here. Recently, Amazon launched Workflow Studio which is a drag-and-drop interface used to create Step Functions which makes the process of creating Step Functions many times easier that before.
- plugins — This section contains packages that help streamline or include/extend functionality of the serverless framework
Since our solution is triggered using a POST request, as soon as we send a POST request to the API Gateway endpoint with a required data, API Gateway will that data forward to the Step Function. The Step Function will then enter the START state and soon transition into the WAITING state and remain there until the timestamp received in the POST request body has arrived. Once the timestamp is reached, the Step Function moves into the next state called ACTION where our Lambda function is triggered and after which the Step Function goes into END state thus completing the execution cycle. Our Step Function here depends on a specific parameter in the request body called dueDate which has to be passed in the POST request along with any other data that we may need.
curl --location --request POST 'https://...../execute' \
--header 'Content-Type: application/json' \
--data-raw '{
"dueDate": "2021-08-28T15:00:00Z",
"data": { .... }
}'
dueDateparameter is in UTC Timestamp format containing date and time of the event
To deploy all these services, just run — serverless deploy from the terminal and that’s it! Your application is now deployed and you’ll be shown the POST endpoint in the console output once the deployment is complete.
Although this is a great solution, there are limitations to this approach which are highlighted very well along with useful metrics in the following blog post by AWS Guru, Yan Cui —
And finally, here’s the link to the finished code repository —
Thank you so much for reading and getting this far. I hope this helps someone in need and if it did, make sure you help someone else! Cheers!
References
[1] AWS Step Functions — https://aws.amazon.com/step-functions/?step-functions&step-functions
[2] Serverless Event Scheduling — https://medium.com/serverless-transformation/serverless-event-scheduling-using-aws-step-functions-b4f24997c8e2
[3] Serverless Functions as an ad-hoc scheduling mechanism — https://blog.smirnov.la/step-functions-as-an-ad-hoc-scheduling-mechanism-ed1787e44bb1
[4] Invoke Step Function from AWS Lambda — https://iamnoah1.medium.com/invoke-aws-step-function-from-aws-lambda-using-the-serverless-framework-21b7fde38a42
