From DynamoDB, ElasticCache, and OpenSearch to RavenDB
How to save a couple thousand euros and contribute to OpenSource while doing it
Written by Ygor Castor, Software Architect and Elixer Tech Lead
Scaling your application in the cloud is a challenge, albeit it’s simple to just let your services scale on demand. However, we often forget that some cloud solutions can really startle your budget if you don’t pay attention to the usage, and that’s what happens with AWS Elastic Databases like DynamoDB, OpenSearch, and ElasticCache.
Four databases, one use case
In our team, we used those different databases for very specific use cases:
- In Dynamo, we stored the user high scores and puzzle states.
- In OpenSearch, we stored production data for textual search.
- In ElasticCache, the most used playlists.
- Postgres for relational data
We already were using four databases, and a new use-case came by. We needed to consolidate data from our relational database to improve our API times, a problem that was not solved with indexes or views.
The data was complex, not a simple key value, so DynamoDB was out of the question. It was not text-based, so there goes OpenSearch; we needed to persist and query the data, so ElasticCache was not an option either. In the end, we needed another database.
But adding another database was against our “simplify” approach. It’s another database to maintain, another backup, and more management time. But what if we replaced one or two with a new one?
Eliminating databases
First, we identified which of the current databases we wanted to ditch, so we checked DynamoDB, where our gaming state was stored. We identified that it was not the best database for our use case: it’s expensive, does not support ad-hoc queries, and querying was a massive headache. All in all: a good candidate to get rid of.
Then we checked our OpenSearch, where we stored textual data from hundreds of thousands of productions. In our new use case, we needed this data, but not in a way that would be easy to query using OpenSearch, so it would be nice to have a database supporting full-text search.
And last, we wanted to have a quick reading cache. That’s why we used ElasticCache, so we needed a database that provides blazing-fast reading.
So we needed a blazing fast reading database, with support for quick full-text search, with guaranteed consistent data, easy to implement, and easy to query. How could we solve this conundrum? RavenDB for the rescue!
Building a driver for RavenDB
RavenDB is a multimodel database with support for conflict-free replicated data types and ACID built-in since its conception. It provides a simple query language close to SQL and full-text-search capabilities — looks like a good candidate! But sadly, it didn’t provide any drivers for Elixir, the language of our services ☹️. But we don’t give up easily.
Analyzing the RavenDB documentation, we got even more sure about our decision to use it for our use case. All the communication with the database can be done with a well-documented API. And our initial tests identified that the database held more than 50.000 read requests per second in a simple t4g.medium EC2, it could solve our problems! So why not just build our own driver and also contribute to the OpenSource community? And that’s what we did; we even gained praise for it!
With our new driver built, we implemented our database strategy and phased out three(!) databases in one go. So, how is it fairing out? Our first numbers are quite interesting!
- One of our APIs went to a median of 1~2 seconds to 180ms!
- Our costs went considerably down without the elastic databases!
- We have almost 1000 queries per second, using 20% of our capacity and reaching an average of 3ms per request!
We’re pretty excited about the results and curious to get some tips from you too. Drop them in the comments below.
