How to Write Data to a File with FS2

Last week, I tried to investigate raw data from Dead Letter Queue (DLQ) at work. One of the functionalities that I want to do is poll sources from the SQS DLQ and write those data in a file for further investigation. I thought using FS2 will be a great use-case for this.

FS2 is a light-weight streaming library that can help you do a lot of data processing functionality. Often, we want to get some data from upstream and write them to a file for any investigation.

I want to share how we can create a simple type class for writing data to a file system with FS2.

We will break it down by the simplest use-case of writing any value to a file. We will then explore how we can incorporate Queue to decrease the back-pressure of writing to a file.

Let’s start by defining our function.

In the fs2 guide, there is already a sample example of how we want to write to an external file system.

The code specified that we will read all from the file and do filtering on some function fahrenheitToCelsion. Then, encode it into a byte and write it to the testdata/celsius.txt.

through will combine one stream to another, and text.utf8Encode returns a Pipe[F[_], I+, O-] which is equivalent to Stream[F,I] => Stream[F,O].

Therefore, we got our first questions answered!

We can write our initial function that takes in an upstream and write it to file. Let’s create toFile(fileName:String, upstream:Stream[IO, String]): Stream[IO, Unit]:

We need a blocker writeAll because it is an operation that will block the thread. Therefore, cats-effect provide a dedicated thread pool Blocker[IO] to explicitly handle blocking operations.