Serverless Architecture Layers 🚀
How and why I believe enterprise organisations should use domain-driven design with the five serverless architecture layers, inspired by the great work of Eric Evans. We will talk through a fictitious company using architecture layers, with visuals and descriptions.
Contents
⚪ Experience Layer
⚪ Cross-cutting Layer
⚪ Domain Layer
⚪ Data Layer
⚪ Platform Layer
Introduction
In enterprise architecture in the Serverless World it is imperative that we think conceptually and holistically about our overall solutions as a set of patterns, templates and guardrails to ensure that we don’t end up with an anaemic lambda pinball architecture, which over time becomes a big ball of mud, and most probably a distributed monolith with lots of duplication of business logic and wasted team effort. (shown below)
“Lambda pinball architectures characteristically lose sight of important domain logic in the tangled web of lambdas, buckets and queues as requests bounce around increasingly complex graphs of cloud services.” — Thoughtworks
This can easily happen as enterprise organisations start their Serverless transformations with many new teams spinning up new solutions in their own silos over time, extrapolating this out quickly as many organisations are also global and geographically distributed. Couple this with Conways Law and we can very quickly have large issues and mass rework at scale.
This article discusses Layered Architecture, which was made prominent by Eric Evans in the book Domain-driven Design. (I encourage all architects and serverless developers to buy this book), and what this means in the Serverless paradigm for enterprises.
When it comes to the Serverless World in enterprise organisations I believe it looks more like the diagram below which shows the “Five Serverless Architecture Layers”:
The following layers have cross over from the old World and new Serverless World (we will quote from the book as we go):
✔️ User Interface (Presentation) & Application → Experience
✔️ Domain → Domains
✔️ Infrastructure → Platforms
During this article we are going to go into the five serverless architecture layers, whilst showing how a fictitious companies ‘(Lee James Mountain Wear’) Serverless architectures looks when using this approach or not.
We can also couple this approach with Serverless TACTICAL DD(R) and an enterprise wide Tech Radar to ensure that we have the relevant guardrails, patterns and architectural governance in place as teams move to the Serverless World during an enterprise wide transformation; whilst also still allowing for innovation and evolutionary architectures at the same time:
Let’s start by going through the various layers below in the next section.
Experience Layer Overview 🏂
For a deep dive into the Experience Layer see the link below
In the book, the User Interface or Presentation Layer is described as “Responsible for showing information to the user and interpreting the users commands” and the Application Layer is described as “Defines the jobs the software is supposed to do and directs the expressive domain objects to work out problems”.
“Defines the jobs the software is supposed to do and directs the expressive domain objects to work out problems” — Eric Evans (Domain-driven Design)
It goes on to say about the Application Layer “This layer is kept thin. It does not contain business rules or knowledge, but only coordinates tasks and delegates work to collaborations of domain objects in the next layer down”
“This layer is kept thin. It does not contain business rules or knowledge, but only coordinates tasks and delegates work to collaborations of domain objects in the next layer down” — Eric Evans (Domain-driven Design)
We can see an example on the diagram below of the Serverless Experience Layer made up of both Presentation and Application Layers, which is typically made up of experiences such as Micro-frontends, Alexa Apps, Websites (Perhaps React app stored in Amazon S3), Mobile apps etc; as well as specific Backend for Frontend (BFF) thin APIs, perhaps using Amazon API Gateway or AWS AppSync.
Create separate backend services to be consumed by specific frontend applications or interfaces. This pattern is useful when you want to avoid customising a single backend for multiple interfaces. This pattern was first described by Sam Newman. — https://learn.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends
These thin APIs can have their own authentication needs, for example an internal employee application may use SSO federated with Azure AD, but the Mobile and Alexa Apps may use Amazon Cognito for external users of the systems.
There should be no business logic in these thin API’s (we will go on to discuss why later), but should only be concerned with the orchestration of business functionality required for the experience by utilising the Domain Layer.
These experience teams align to the ‘Stream Aligned’ teams in Team Topologies. These are described as “aligned to a flow of work from (usually) a segment of the business domain”
Let’s see an example for our company
We can see from the diagram below that the Stock team who deals with the mountain wear stock has:
- Allowed business logic to leak into the frontend code for the internal stock check system. This logic is not accessible to the rest of the systems.
- An Alexa app needs to add functionality for ‘Check stock online’; however that logic has already been duplicated in the two backend for frontend APIs for the internal stock check system and the websites stock check functionality.
If the business logic had remained in the Domain Layer where it should have been, then all three experiences could have utilised it properly.
Summary
✔️ Ensures business logic is not leaked through an organisation. (Lambda Pinball)
✔️ Ensures that business logic is reusable across the organisation and experiences as it doesn’t sit in BFFs or frontend code.
✔️ Allows us to use different authentication requirements per experience.
✔️ The experience i.e. UI/Voice/UX should contain no shared business logic.
✔️ The backend APIs for the experience should be stateless.
Cross-cutting Layer Overview 📐
There is no equivalent in the book for the Cross-cutting Layer, but this is something which in my opinion is imperative to think about when architecting enterprise Serverless solutions, and gaining governance through a Tech Radar.
These cross cutting concerns are typically progressed through ‘Enabling’ teams in Team Topologies. These are described as “helps a Stream-aligned team to overcome obstacles. Also detects missing capabilities”
Examples of cross-cutting concerns are component libraries for frontend development, the sending of communications such as emails and SMS, and aggregated logging and tracing, allowing us to correlate logs across a myriad of services and calls. Another would be authentication, as our customers should have an SSO experience regardless of using an Alexa app, a Chatbot or our Website.
We may also want to consider the creation of a company SDK for code reuse and to increase developer speed, and reduce cognitive load. In the AWS World this could be through the use of AWS CodeArtifact.
A software development kit (SDK) is a collection of software development tools in one installable package — Wikipedia
As CTO’s and architects if we don’t think about this early as a group as we start to architect our solutions, we find that individual teams start to tackle and solve the same issues over and over; for example more than one team looking at customer authentication. (shown below)
This leads to teams choosing different SasS products within their own silos for example, increasing cognitive load on new teams which are spun up (solving the same problems again and again), and reducing the chance of us gaining better contracts with suppliers at scale. If we then extrapolate this out with logging, comms, component libraries and more, across a large number of teams, it leaves us in a poor state.
Let’s see an example for our company
In our example for Lee James Mountain Wear, we find that our Stock and Orders teams are both building out their own React Component Libraries in their own silos as there is a lack of communication at a strategic level, without thinking about this globally at an enterprise level.
At the same time, our Customer and Orders teams are both doing POC work around observability, and speaking to two different SaaS suppliers. Extrapolate this out as the number of teams grow, as well as the global aspect, and you have some serious issues at scale (shown below).
Summary
✔️ Prevents complexity, duplication, and cognitive load in each team solving the same issues.
✔️ Allows people to move more easily between teams as we have standards.
✔️ Increases speed and agility in teams without re-inventing the wheel.
✔️ Allows us to gain better deals with suppliers on SasS products due to increased usage at an enterprise scale.
Domain Layer Overview 🛍️
For a deep dive into the Domain Layer see below
The Domain Layer in my opinion is the most important to get right and focus on. In the book Eric Evans describes this as “Responsible for representing concepts for the business, information about the business situation, and business rules.”.
“Responsible for representing concepts for the business, information about the business situation, and business rules” — Eric Evans (Domain-driven Design)
In a Serverless World, this would be represented typically using Private API Gateways backed by AWS Lambda or AWS Fargate, specifically with its own datastore such as AWS DynamoDB. We would also want to drive this through Domain-Driven Design (DDD), which is discussed below at a conceptual level:
These domain teams align to the ‘Stream Aligned’ teams in Team Topologies. These are described as “aligned to a flow of work from (usually) a segment of the business domain”
The API should be well defined and versioned using Open API (Swagger), and should become the aggregate root and main interface for the domain. In event-driven systems it is also imperative that we use well defined versioned events to interface with the domain, for example using Amazon EventBridge with the Schema Registry. This ensures that we can change the internals of our domain service without effecting consumers.
The internals of the domain service should be incapsulated so we don’t have other domains and teams reaching out directly to our data stores for example, bypassing the governance of our business rules, logging and logic. (as shown above)
This now means that we can reuse domain business logic across many experience layers and other domains, calling through from the thin backend for frontend APIs to our Domain APIs using IAM auth and SigV4.
“This now means that we can reuse domain business logic across many experience layers and other domains, calling through from the thin backend for frontend APIs to our Domain APIs using IAM auth and SigV4.” — Lee Gilmore
We need to ensure that the domains are well encapsulated and that domain services don’t start interacting with the internals of other domains, whether that be data stores or internal AWS SNS topics or Amazon SQS queues (as shown above). This undoes all of the good work that we have done in defining the domains and creating our versioned interfaces.
We should also ensure that our domain services are not accessible to the outside World, even when they are tied down using authentication. (as shown above).
As the APIs should be tied down using a machine to machine flow, there is no reason that they should be accessible via Postman, curl, mobile, webpages etc. They will never be called directly from a frontend for example, so for security reasons they should be tied down accordingly.
So what would Domain-Driven Design look like at the code level?
The link above discussed DDD, and how we could apply the concepts at purely a conceptual and service level. The link below takes this one step further and shows how we can do this at the code level:
Summary
✔️ Well-defined versioned APIs and events ensure consumers understand how to interact with our domains.
✔️ Domain interfaces mean we can change the internals without affecting consumers.
✔️ Our domain services are not externally accessible to the World, and only accessed using machine to machine flows. (i.e. using private API Gateways).
✔️ We ensure that consumers don’t interact with the internals of our domains, only through our well defined APIs and events.
✔️ The domain logic can be utilised by other domain services or experiences, preventing anaemic domains being split across a distributed monolith.
Data Layer Overview 📊
For a deep dive into the Data Layer see the following article:
The next Serverless layer is the Data Layer, which is not represented in the book; however in todays data and event-driven world, we need to think about this from an enterprise level.
The data layer is typically made up of three key areas:
- Enterprise Service Bus. This for example would typically be Amazon EventBridge or Amazon MSK (Kafka).
- Reporting and BI. This for example would typically be Amazon QuickSite and Amazon Athena.
- Data. This would typically be Amazon Redshift or S3 data lakes for example.
The data layers are typically progressed through both ‘Complicated Subsystem’ and ‘Platform’ teams in Team Topologies. Complicated Subsystem teams are described as “where significant mathematics/calculation/technical expertise is needed”. Platform teams are discussed further on.
We should govern this at an Enterprise Architecture level to ensure we have a way for our domain services to interact with each other in an event-driven manner across the organisation, and don’t have different teams using different technologies (SNS, Kafka, EventBridge etc). This is covered in the article below:
Utilising an enterprise service bus such as Amazon EventBridge now means that domains can interact with each other using well defined versioned events, as well as backend for frontend APIs in the experience layer also raising events which other domains may be interested in:
This is shown below where domains may raise events such as ‘Order Created’, ‘Price Amended’ etc; and Experience Layers may raise events such as ‘User Logged In’.
This also allows us to build up our data lakes and aggregate these events to allow us to be more data driven, so there are intrinsic links between events, data and reporting/BI. The article below discusses how to utilised streams and events to work around the issues with two phase commits in a Serverless World:
Let’s see an example for our company
In the enterprise organisation Lee James Mountain Wear we have three different teams utilising three different technologies (SQS, EventBridge and MSK), so this makes it very difficult for domain communication based on events. Standardising at an enterprise level from the outset would have made this much easier, reducing development time, reducing cognitive load on teams, and allowing for versioned event schemas and event discovery.
“Standardising at an enterprise level from the outset would have made this much easier, reducing development time, reducing cognitive load on teams, and allowing for versioned event schemas and event discovery” — Lee Gilmore
Summary
✔️ An Enterprise Service Bus allows us to coordinate across teams in a business — and Amazon EventBridge should be your go to on AWS.
✔️ Amazon EventBridge Schema Registry allows other teams to view our versioned events and consume them (easy to discover, reducing cognitive load and enhancing communication)
✔️ We can report on the data and events by streaming them to data lakes or warehouses, and consuming using tools such as Athena and QuickSite.
✔️ We can use sentiment analysis and events to build up single customer views.
✔️ Reduces cognitive load on teams using an ESB and increases their speed and agility.
Platforms Layer Overview 🌐
The final layer in the book, the Infrastructure Layer, is the equivalent of the Platforms layer. In the Serverless World the teams will be managing their own architecture and services through IaC, rather than the traditional World of server management. The book describes the Infrastructure layer as “Provides generic technical capabilities that support the higher layers”.
“Provides generic technical capabilities that support the higher layers” — Eric Evans (Domain-driven Design)
In the Serverless World for development teams this is typically made up of:
- Authentication Platforms. This is a standard way of teams performing machine to machine flows.
- Pipelines/Accounts. Serverless teams should not need to think about the differentiated heavy lifting such as spinning up pipelines or creating VPCs, networking, transit gateways etc. This should be templated and quick to spin up for new teams/domains (think vending machines).
- Security. Security should be baked in as early as possible into our templates when new teams and solutions are spun up — shifting this left as quickly as possible.
- API Specs/Event Schemas. These should be centralised and easy for teams to discover, increasing agility and speed. This is where we should have a key focus on Developer Experience.
The platform layer is typically progressed through the ‘Platform’ teams in Team Topologies. Platform teams are described as “a grouping of other team types that provide a compelling internal product to accelerate delivery by Stream-aligned teams”.
This is shown in the diagram below:
The most important of these in my opinion is Pipelines and Accounts, as Serverless teams should have the ability to utilise templates and reference architectures, with the platform essentially removing the differentiated heavy lifting and reducing cognitive load. These templates should also have security baked in, shifting security left as early as possible.
There are many products such as AWS Proton and Backstage IO which perform this at an enterprise level.
Summary
✔️ Developer portals and platforms should increase speed and agility for Serverless teams.
✔️ Differentiated heavy lifting such as networking, setting up VPCs, creating pipelines etc should be provided by the platform.
✔️ Developer portals should make it easy for teams to discover API and event definitions.
✔️ Amazon CDK L3/L4 constructs allow teams to reuse standard infrastructure templates.
✔️ Having a Micro-frontend strategy and Core UI allows teams to be more efficient and to embrace DDD at the UI level.
Summary
I hope you found that useful as a way too conceptually think of our Serverless architectures at an Enterprise Architecture level.
Go and subscribe to my Enterprise Serverless Newsletter here for more of the same content:
Wrapping up 👋
Please go and subscribe on my YouTube channel for similar content!
I would love to connect with you also on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)
About me
“Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect and Global Serverless Architect based in the UK; currently working for City Electrical Factors, having worked primarily in full-stack JavaScript on AWS for the past 6 years.
I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture and technology.”
*** The information provided are my own personal views and I accept no responsibility on the use of the information. ***
You may also be interested in the following:
