Hacking the Redis Protocol to use as an HTTP API alternative with Asyncio in Python.

Normally when you think of APIs, your mind goes directly to HTTP. While the protocol has dominated for good reason, it was primarily built for the web and can fit awkwardly in a server-to-server API where long-lived and authenticated connections are common. HTTP2 addresses some of the problems with multiplexing, header compression and more, but these added features make the protocol unwieldy. An alternative should at least be considered.

Why Redis Protocol?

It's hard to find a Request-Response protocol written in as many languages as Redis. Nearly every language has a client, either officially supported or not, and the protocol is simple enough that a basic client can be written in an hour. Some potential benefits of using RESP as an API include:

• Long lived connections.
• Don’t need to re-authenticate every request. This includes both sending headers and checking API keys in storage or processing JWTs.
• Clients have already been battle tested, written for high performance and are available in most languages.
• Clients have built in authorization and SSL.
• Simple Protocol. HTTP2’s Protocol is very complex involving state-machines and implementations end up sprawling. RESP is simple enough to read raw on the wire.
• Good enough for one of the most prolific memory stores used in production.

Building the Server

The Redis protocol is made up of 5 types. These types are prefixed by a byte to signify what the subsequent bytes represent. Redis’ documentation succinctly details the header byte of objects.

• For Simple Strings the first byte of the reply is “+”
• For Errors the first byte of the reply is “-”
• For Integers the first byte of the reply is “:”
• For Bulk Strings the first byte of the reply is “$”
• For Arrays the first byte of the reply is “*"

This simplicity makes Redis responses easily constructed with bytes formatting like so: b"-ERROR %b\r\n" % err_text (this is a serialized error).

Our compact server is made possible by one simple trick: since Redis commands are encoded as arrays and a client can receive an array as a response, the parser for clients is the exact same as the parser for the server.

A minimal Redis Server that implements the QUIT command would look like this:

Highlights of our API server

Python does not recommend you use StreamReader directly and instead use loop.start_server however, aioredis implements their own StreamReader that directly parsers Redis objects from the socket. In order to use it, part of the start_server function is copied over and aioredis’ StreamReader is injected.

This is the main loop for a connection. It loops indefinitely parsing a Redis object and calling process_command for each call to the server.

Our business logic is isolated to the process_command function. The only information the loop needs back is whether or not to break and close the connection.

Implementing Commands

Redis commands come in as an array of binary strings. You can easily encode and decode your arguments using JSON to produce commands that provide rich functionality:

We can use these commands by starting up a Redis client and manually executing them. Wrapping these commands in a client class cleans things up and compartmentalizes things.

Final Thoughts

Since we are using the Redis client parser, if we install hiredis-py our server will automatically start using the faster parser written in C. Another performance improvement would be to serialize using msgpack for an extra edge of performance. Because Redis clients typically support TLS, we can experiment throwing this behind a load balancer with SSL to see how production ready this implementation is. Security of the parser should also be considered to ensure that it isn’t possible to instantiate a DDOS attack by sending large strings for example.

Performance should be very good. All the code is rather low level and objects are parsed directly from the wire. It would be great to be able to benchmark this against uvicorn or other HTTP frameworks.

Overall, I was very impressed with the compact implementation that asyncio provided for creating a pretty robust server in just 50 lines of code. This method of creating APIs looks very promising and I look forward to investigating it further.