Circuit Breaker Pattern with Cats Effect
Recently, I have been interested in the Cats Effect which is a functional library in Scala. To improve my coding skills with Cats Effect, especially concurrency state management, I implemented a Circuit Breaker with this library. Circuit Breaker is one of the design patterns in software development. In the context of Microservices, this design pattern is popular.
In this article, I would like to explain a bit about Circuit Breaker Pattern and how to implement it with the Cats Effect. All entire code is on my Github repositories.
Circuit Breaker Pattern
Circuit Breaker Pattern is a design pattern of software development.
In a distributed system, each service communicates with the other service. However, at times a service keeps sending a request to the other service until the request times out, even if the service has something wrong and doesn’t respond. This situation may lead to exhaustion of threads, then eventually the entire system and the others which depend on it could be stopped.
Circuit Breaker is a solution for this case. It works as a proxy between services. This proxy supervises the request and decides if the request should be sent or not based on the recent number of failures. So, Circuit Breaker is a state machine that follows these states:
- Closed
A request is sent directly to the service. If the request fails, the breaker adds up the number of failures. When the number reaches out to the defined threshold, the breaker opens.
- Open
A request returns the error immediately. After a certain amount of time, the breaker becomes half-opened.
- Half-Open
A request is sent directly to the service. If the request succeeds, the breaker is closed again. Otherwise, the breaker remains open.
Implementation with Cats Effect
As I explained previously, the Circuit Breaker has an internal state. In terms of implementation, this state must be managed properly, even if a request is sent asynchronously. In other words, Circuit Breaker has a mutable reference internally and this reference must be atomic.
To achieve this requirement, Cats Effect provides some goodies for concurrent programming in cats.effect.concurrent module. In the module, Ref is suited for this state management. It has a purely functional, concurrent, lock-free mutable reference. This mutable reference is kept as AtomicReference. As you can see from the code below, their operations such as get, modify are executed atomically.
final private class SyncRef[F[_], A](ar: AtomicReference[A])(implicit F: Sync[F]) extends Ref[F, A] {
def get: F[A] = F.delay(ar.get)def set(a: A): F[Unit] = F.delay(ar.set(a))def update(f: A => A): F[Unit] = modify { a =>
(f(a), ())
}def modify[B](f: A => (A, B)): F[B] = {
@tailrec
def spin: B = {
val c = ar.get
val (u, b) = f(c)
if (!ar.compareAndSet(c, u)) spin
else b
}
F.delay(spin)
}
}
Next, let’s take a look at this implementation step by step. This implementation was inspired by these two modules: Glue.CircuitBreaker, which is a module for Haskell, and circuit-breaker-monad, which is a module for TypeScript.
Interface
The Circuit Breaker trait has run and getStatus Methods.
- run
This is the main method a client uses when a request is sent with a Circuit Breaker. The argument is an execution that sends a request to the other microservice. In the method, it’s decided whether the request should be sent based on the state of Circuit Breaker. When the breaker is open, this method returns the error immediately wrapped in IO.
- getStatus
This method returns the internal state atomically. This state can be read, but can’t be changed from external.
The CircuitBreaker object can be created by the create method of the companion object. In this method, the state is represented as Ref.of[F, BreakerStatus]. The create returns the Circuit Breaker object wrapped in type constructor F[_] because the mutable state is created internally, which is an effect. In the example below, it’s wrapped in IO.
The CircuitBreaker allows a client to use the same breaker or separated breakers. As you can see in the example below, in the context created by calling flatMap, the breaker is shared. Otherwise, each state is created and shared separately. It is because of referential transparency which the Ref has. This specification makes it easy to understand where a breaker state is shared. In Fabio’s presentation, this referential transparency of Ref was explained in detail.
val circuitBreaker: IO[CircuitBreaker[IO]] =
CircuitBreaker.create[IO](breakerOptions)
def toServiceA1(circuitBreaker: CircuitBreaker[IO]): IO[String] = ???
def toServiceA2(circuitBreaker: CircuitBreaker[IO]): IO[String] = ???
def toServiceB(circuitBreaker: CircuitBreaker[IO]): IO[String] = ???def separatedBreaker =
circuitBreaker.flatMap(toServiceA1) >> circuitBreaker.flatMap(toServiceB)
def sharedBreaker = circuitBreaker.flatMap { c =>
toServiceA1(c) >> toServiceA2(c)
}
On the other hand, this CircuitBreaker object doesn’t consider a use case of a multi-node server. It means a breaker state isn’t shared among servers. To satisfy this requirement, persistent storage should be used.
In a multi-node (clustered) server, the state of the upstream service will need to be reflected across all the nodes in the cluster. Therefore, implementations may need to use a persistent storage layer, e.g. a network cache such as Memcached or Redis, or local cache (disk or memory based) to record the availability of what is, to the application, an external service.
Breaker Closed
When the breaker is closed, the callIfClosed method executes the body which a client takes. If the execution succeeds, the response is returned. If the execution fails, the number of failures adds up with the modify that updates the state and gets the result atomically. When the number of failures reaches the threshold, the breaker opens.
Breaker Open
When the breaker is open, the callIfOpen method checks if a specific time passes after the breaker was open. After a specific time passes, it means the breaker is half-open and the body is taken the canaryCall method. Otherwise, the breaker would be kept open and the error is returned immediately.
Breaker Half-Opened
When the breaker is half-open, the canaryCall method attempts to execute the body. If the execution succeeds, the breaker would close again.
Running CircuitBreaker
This example below shows how to use the CircuitBreaker with http4s. http4s also depends on the Cats Effect, so it’s easy to use this library with the CircuitBreaker. In the example below, one CircuitBreaker object is shared. You can also use a CircuitBreaker object separately in each context by calling flatMap as I explained previously.
Moreover, you can see the behavior of the Circuit Breaker object through the test code I implemented. This test code is implemented with cats-retry to realize a retry mechanism, which is a library for retrying actions that can fail. I’ve written about this library in my previous article.
The code below is one of the test cases in the test code. It proves that the breaker opens after the execution failed once, which is defined as maxBreakerFailures.
Conclusion
Many libraries such as doobie, http4s are using the Cats Effect internally. So by using these libraries, you can implement an entire system with functional programming. In addition, in terms of implementing a module by yourself, this library is very useful because it provides powerful modules such as concurrent programming, state management, etc.
If there is something wrong with my code or you have another way of implementing Circuit Breaker with Cats Effect, please let me know via hirokifujino0108@gmail.com.
Thank you for reading!
