Open in app

Sign up

Sign in

Write

Sign up

Sign in

Building URL Shortener from Scratch

Shrishty Chandra
Level Up Coding
Published in
7 min readMay 21, 2023

--

Implementing a Serverless URL Shortener with AWS DynamoDB and the Serverless Framework

Photo by NASA on Unsplash

Hello friend,
In this article let's see how you can create a fully functional URL shortener and deploy it in AWS. We will be using the Serverless Framework for the implementation.

I will divide the full implementation into 5 parts.

  • Part 1: I created a functional URL shortener.
  • Part 2: Building the UI for the URL shortener and deploying it in AWS.
  • Part 3: Monitoring and Logging.
  • Part 4: Adding a domain name and mapping it with DNS.
  • Part 6: Scale considerations and implementation according to that.

So let’s get started.

High-Level System Design (Part 1)

Image created by author using FigJam

Here is how the data will flow when the user pastes the URL to be shortened in a text box and presses shorten button.

  • The request will hit the API Gateway, where API Gateway will figure out based on the URI, where to forward the request to. In this case, since the user wants a shortened URL, it's gonna be a POST request with URI (tinyurl/shorten). So the request will be forwarded to Create Short URL Lambda
  • Create Short URL Lambda will fetch the longUrl from the request and generates a random string of length 7, which gets used to generate the shortURL by appending the random string to our BaseURL. Stores them in the URL’s table and returns the same as a response.
  • When a user makes a GET call on the shortened URL then, API Gateway forwards the request to Fetch Long URL Lambda, which makes a db lookup and returns the response with status code 302, and the location as the longURL in the header which helps in redirecting the request to the long URL.

APIs

We will have just two APIs one to shorten the long URL and the second to get the long URL from the short URL.

POST tinyurl/create

  • Method: POST
  • URI: tinyurl/create
  • Request Body:
{
    "longUrl": "https://medium.com/@sheril"
}
  • Status Code: 200
  • Response Body:
{
    "baseUrl": "https://shrishty.me",
    "shortUrl": "https://shrishty.me/tinyurl/pvggw",
    "longUrl": "https://www.linkedin.com/in/shrishty-chandra-49b1585a/",
    "expirationTime": "1683564336",
    "creationTime": "1683564336"
}

GET tinyurl/{short_id}

  • Method: GET
  • URI: tinyurl/pvggw
  • Request Body: None
  • Status Code: 302 (Redirect)
  • Header: Location: {long_url}

Database Design

The database design is simple, and I will use Dynamodb to store tinyurls. We will have one table named URLTable where each row will have the following schema.

{
    "PK": "<short_id>",
    "SK": "<short_id>",
    "longUrl": "https://www.linkedin.com/in/shrishty-chandra-49b1585a/",
    "shortUrl": "https://shrishty.me/tinyurl/pvggw",
    "creationTime": "1683564336",
    "TTL": "1683564336"
}

We will talk about PK, SK, and TTL in detail later in this article.

Implementation

Setup

  1. Install Serverless framework.
npm install -g serverless

2. Create a new serverless project. After running the below command, a new folder named url-shortner-backend will be created, containing multiple files eg. serverless.yml, handler.py, etc.

serverless create --template aws-python3 --path url-shortener-backend

Serverless.yml

In serverless.yml we define all the resources that are going to be used in our application. Let's go through each section one at a time:

Provider — The provider section is used to configure the cloud provider, that will be used to deploy the application. In our case, it's gonna be AWS. It also specifies the necessary credentials and settings for the chosen cloud provider. Eg. stage - specifies the deployment stage of the application, region — specifies the cloud region where the cloud function is deployed, environment — specifies the environment variables that will be used in the cloud functions, and iamRoleStatements — specifies the permissions required by the serverless functions.

Functions — I have defined two functions shortenUrl and getLongUrl which just handles the POST and GET, HTTP endpoints respectively.

Resources — I have also defined 1 resource, the URLTable which stores the information about the shortened URL. We also export the table name and its ARN as it will get used by our functions to get or add data.

Custom — The custom section contains all the variables whcih we may need during the implementation of tinyurl app, eg. URLTable, BaseURL etc.

service: url-shortener-backend
frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.9
  stage: ${opt:stage, 'dev'}
  region: 'us-east-1'
  environment:
    URL_TABLE_NAME: ${self:custom.URLTable.name}
    REGION: ${self:provider.region}
    SERVICE_NAME: ${self:service}
    BASE_URL: ${self:custom.BaseURL}
  iamRoleStatements:
      - ${file(src/iam/URLTableIAM.yml):URLTableIAM}
      - ${file(src/iam/CloudWatchIAM.yml):CloudWatchIAM}

functions:
  shortenUrl:
    handler: src/handlers/shorten_url.create_short_url
    events:
      - httpApi:
          path: /tinyurl/create
          method: post
  getLongUrl:
    handler: src/handlers/shorten_url.get_long_url
    events:
      - httpApi:
          path: /tinyurl/{id}
          method: get

resources:
  Resources:
    URLTable: ${file(src/resources/URLTable.yml):URLTable}
  
  Outputs:
    URLTableArn: ${file(src/resources/URLTable.yml):Outputs.URLTableArn}
    URLTableName: ${file(src/resources/URLTable.yml):Outputs.URLTableName}

custom:
  URLTable:
    name: !Ref URLTable
    arn: !GetAtt URLTable.Arn

  ShortURLLambda:
    arn: arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:user-service-${self:provider.stage}-shortenUrl

  BaseURL: !Sub 'https://${HttpApi}.execute-api.${aws:region}.amazonaws.com'

Dynamodb Configuration for URLTable (URLTable.yml)

I have kept the base implementation of the dynamodb table as simple as possible and is limited to PK, SK, and TTL. Other entries like shortURL, longURL, etc. will be written directly into the table, we don’t need to define attributes for them. So, let's understand the role of PK, SK, and TTL.

PK — It is the partition key; in our case, it would just be the 7-digit shortId generated by the create function.

SK — SK, also called Sort Key, is mainly used for sorting and ordering data. For the tinyurl example, we actually don’t need SK. But I have kept it so that I leave room for expanding the application in the future. Currently, SK is also the 7-digit shortId generated by the create function.

The Partition Key and Sort Key combine together to make the Primary Key.

TTL — The TTL, also known as TimeToLive, is a configuration that helps us specify the table row's expiration time (in timestamp format). Post expiration the row gets deleted, freeing up the ShortId. We will do better management of shortId in the next post so, stay tuned.

URLTable:
  Type: AWS::DynamoDB::Table
  Properties:
    TableName: URLTable-${self:provider.stage}
    BillingMode: PAY_PER_REQUEST
    KeySchema:
      - AttributeName: PK
        KeyType: HASH
      - AttributeName: SK
        KeyType: RANGE
    AttributeDefinitions:
      - AttributeName: PK  # SHORT ID
        AttributeType: S
      - AttributeName: SK  # SHORT ID
        AttributeType: S
    TimeToLiveSpecification:
      AttributeName: TTL
      Enabled: true

Outputs:
  URLTableArn:
    Value: 
      "Fn::GetAtt": [ URLTable, Arn ]
    Export:
      Name: URLTable-${self:provider.stage}-Arn
  
  URLTableName:
    Value: 
      Ref: URLTable
    Export:
      Name: URLTable-${self:provider.stage}-Name

IAM Permissions (iam/URLTable.yml)

We need to define the IAM (Identity and Access Management) permissions for the URLTable resource.

Action — Tells what actions or operations can be performed on the AWS resource (URLTable)

Resource — Tells the resource on which the actions can be performed.

Effect — Tells whether the resource is allowed or denied to perform the specified action

For the below example, we are allowing the URLTable to perform the following actions: PutItem, Scan, GetItem, UpdateItem, and Query.

URLTableIAM:
  Effect: Allow
  Action:
    - dynamodb:PutItem
    - dynamodb:Scan
    - dynamodb:GetItem
    - dynamodb:UpdateItem
    - dynamodb:Query
  Resource:
    - ${self:custom.URLTable.arn}

Create Short URL Lambda Implementation

Let's go through the implementation of CreateShortUrl and GetLongUrl one by one. Here are the main responsibilities of both functions:

Create Short URL

  • Fetch the LongURL from the request body
  • Generate a Unique Short Id
  • Calculate the Expiration time.
  • Write everything to dynamodb

Get Long URL

  • Get the short id from the query param.
  • Fetch the data from dynamo db.
  • Create a response with Status code 302.

Checkout the code below to understand in more detail.

import json
import boto3
import os
import datetime
import random
import string
from botocore.exceptions import ClientError

from typing import Any, Dict, Optional

URL_TABLE_NAME = os.getenv('URL_TABLE_NAME')
BASE_URL = os.getenv('BASE_URL')
dynamodb = boto3.resource('dynamodb')
url_table = dynamodb.Table(URL_TABLE_NAME)


def get_item(short_id: str) -> Optional[Dict]:
    try:
        response = url_table.get_item(
            Key={
                'PK': short_id,
                'SK': short_id,
            }
        )
    except ClientError as e:
        print('Error:', e.response)
        return {
            "statusCode": 500,
            "body": e.response['Error']['Message']
        }

    item = response.get('Item', {})
    return item


def get_unique_short_id(size=7, max_iter=10) -> str:
    short_id = ''
    curr_iter = 0
    while (short_id == '' or get_item(short_id) and curr_iter < max_iter):
        short_id = ''.join(random.choices(string.ascii_lowercase, k=size))
        curr_iter += 1

    return short_id


def create_short_url(event: Dict[str, Any], context) -> Dict[str, Any]:
    request_body = json.loads(event['body'])
    long_url = request_body['longUrl']
    short_id = get_unique_short_id()
    short_url = f'{BASE_URL}/tinyurl/{short_id}'
    creation_time = datetime.datetime.now()
    ttl = int((creation_time + datetime.timedelta(days=7)).timestamp())
    creation_timestamp = int(creation_time.timestamp())

    try:
        _ = url_table.put_item(
            Item={
                'PK': short_id,
                'SK': short_id,
                'longUrl': long_url,
                'shortUrl': short_url,
                'creationTime': creation_timestamp,
                'expirationTime': ttl,
                "baseUrl": BASE_URL,
                'TTL': ttl
            },
            ConditionExpression='attribute_not_exists(PK)'
        )
    except ClientError as e:
        print('Error:', e.response)
        body = f"something went wrong please try again: {str(e)}"
        return {
            "statusCode": 500,
            "body": body
        }

    body = {
        "baseUrl": BASE_URL,
        "shortUrl": short_url,
        "longUrl": long_url,
        "creationTime": creation_timestamp,
        "expirationTime": ttl,
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response


def get_long_url(event: Dict[str, Any], context) -> Dict[str, Any]:
    short_id = event['pathParameters']['id']
    item = get_item(short_id=short_id)
    if not item:
        return {
            "statusCode": 404,
            "body": "The short URL does not exist"
        }

    return {
        "statusCode": 302,
        "headers": {
            "Location": item['longUrl']
        },
        "body": json.dumps({
            'longUrl': item['longUrl'],
            'shortUrl': item['shortUrl'],
            'creationTime': int(item['creationTime']),
            'ttl': int(item['TTL'])
        })
    }

Deploy

To deploy your serverless API, run the following command.

serverless deploy --aws-profile <ProfileName>

Once the deployment completes you will get an output containing the API Gateway URL for the service you just created. For my deployment, the API Gateway URL is: https://5eur2xset6.execute-api.us-east-1.amazonaws.com

Image By Author

To see debug logs for the APIs you have created you can run the following command:

serverless logs -f getLongUrl --aws-profile <ProfileName>

Test

Congratulations on reaching this far. We can test the code, I prefer testing by Postman but I am pasting the curl commands so that you can simply import it into your Postman account and run it.

Create Short URL

curl --location 'https://5eur2xset6.execute-api.us-east-1.amazonaws.com/tinyurl/create' \
--header 'Content-Type: application/json' \
--data '{
    "longUrl": "https://www.linkedin.com/in/shrishty-chandra-49b1585a/"
}'

GET Long URL

curl --location 'https://5eur2xset6.execute-api.us-east-1.amazonaws.com/tinyurl/qeqvemn'

Conclusion

I enjoyed writing this article and I hope you find it interesting too. Please stay tuned for the next article where I will implement the UI for URL Shortener and deploy it in AWS.

Find the full implementation in my GitHub repository: https://github.com/shrishty/tinyurl

If you like this article, I would love to get a few claps (It makes me happy 😅).

See you soon. Bye!

~~~
Connect with me on LinkedIn. You can also book a free 1:1 with me on topmate.io.
~~~

AWS
API
Serverless
Cloud Computing
Computer Science

--

--

Shrishty Chandra
Level Up Coding
Follow

Written by Shrishty Chandra

399 Followers

Software Engineer at Google | https://www.linkedin.com/in/shrishty-chandra-49b1585a/

Follow

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams