Internals of Protocol Buffers

With a rapid transition from Monolithic application development to Microservices based application development, communication between microservices has become a crucial and core component. We need some sort of common language pattern where all dependent microservices should be able to communicate irrespective of programming language.

It all started off with XML and we then moved on to JSON (Javascript Object Notation) which is a key-value pair. JSON worked well with REST and is still the go-to guy when it comes to microservices communication. But just like any other product which has its own ups & downs, there are quite a few problems with JSON when it comes to operating at scale. As this article is not intended to compare a wide range of data exchanging formats, I’ll skip this and move ahead.

Taking the problems caused by JSON into consideration, Protocol Buffers were developed, and in this article, I’m gonna discuss a few common mistakes people commit while using Protocol Buffers.

Let’s say there are two microservices that are handling the user profile section. One is the front-end service and the other is the backend service.

Scenario

Consider the app has 3 different kinds of wallets and the user can check his/her wallet balance by clicking on the wallet section under the user profile. As soon as the user clicks the check wallet balance, the frontend makes a network call to the backend to get the wallet balance. Let's now compare both JSON and proto request formats.

JSON Format

Request (can be a get with query params as well)
URL: http://localhost:8080/getUserWallet
RequestType: POST
{
    "userID":"mouryavenkat"
}
Response
[
    {
        "walletType": "wallet1",
        "walletAmount": 300,
    },
    {
        "walletType": "wallet2",
        "walletAmount": 200,
    },
    {
        "walletType": "wallet1",
        "walletAmount": 500,
    }
]

Proto Format

Request 
RPC Method: GetUserWalletDetails
message UserWalletRequest {
    string user_id = 1;
}
Response
message WalletInfo {
    string wallet_type = 1;
    int64 wallet_amount = 2;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}

The proto request and response are stored in a .proto file on both the client-side and service side. From the frontend, it fills in all the data as required and then calls the RPC Method as if it's a function call. The underlying GRPC client library then marshall the data into binary format and transfer the data over the wire to the server and the server’s GRPC client library then un-marshall the data and hand it over to the application to process the request and the same process is gonna repeat while the server is responding with wallet info back to the frontend.

One very important thing to note here is the marshalling and un-marshalling process. The numbering we had given to each field in the proto request and response plays the key role in marshalling & un-marshalling the request-response.

Marshalling process

The response contains multiple chunks of data. As of now, on level 1 it contains only one chunk which is of type repeated wallet_info. Each wallet_info contains 2 subfields with wallet type as a string with sequence 1 and wallet amount as integer type with sequence 2.

Unmarshalling process

While un-marshalling, the server takes in the proto file it had and instead of un-marshalling by key name, it tries to match the sequence order sent by the client.

• It first goes to UserWalletResponse and gets sequence 1 which is repeated wallet info. It first checks if the data type of sequence 1 which the server had is matching with the client sequence or not. If yes, it tries digging deeper until it finds a basic data type(int, double, string etc).
• In our case, it finds a slice of wallet_info and goes into each block of wallet info and then it checks the sequence of wallet_info starting from 1.
• 1 in wallet_info is a string and matching with the server’s proto.
• 2 in wallet_info is an int64 and matching with the server’s proto.

As the sequence that the client sent, matched with the sequence that the server proto had, it populates the request data accordingly and pass the same to the application to process the request.

Now that we got a brief about un-marshalling happens, let’s now derive a couple of cases.

Case-1 (Client, Server having the same sequence with different field names)

Response Structure Client Has
message WalletInfo {
    string wallet_type = 1;
    int64 wallet_amount = 2;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}
Response Structure Server Has
message WalletInfo {
    string type_of_wallet = 1;
    int64 wallet_balance = 2;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}

If you see the above .proto file both client and server had two different proto’s with the same sequence but different names. If it was for JSON, the un-marshalling surely results in an erratic behaviour but with proto, as the sequence and data type match the un-marshalling will be successful.

Server sends response as 
WalletInfo {
    type_of_wallet = "wallet1";
    wallet_balance = 300;
}
At client end it will be interpreted as 
WalletInfo {
    wallet_type = "wallet1";
    wallet_amount = 300;
}

Case-2 (Client wishes to change the data type of a field)

So till now, we’ve shown wallet_amount as an integer value to the client and somehow the product team want it to be shown as a float value rounded up to 2 digits after the decimal point(Ex: 300.27). For this, the backend and front end contract has to be updated to send and receive a double(float64 equivalent in proto) instead of an integer.

With rolling upgrades and independent microservices deployments, either of the services has to go live first. Let’s say the backend service had gone to production first. The proto’s now look this way.

Response Structure Client Has
message WalletInfo {
    string wallet_type = 1;
    int64 wallet_amount = 2;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}
Response Structure Server Has
message WalletInfo {
    string type_of_wallet = 1;
    double wallet_balance = 2;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}

With this change, the data type of sequence 2 in wallet info is not matching with the sequence 2 in wallet info of the client and whenever the client library tries to un-marshall the response it results in an error. Till the time client takes the updated changes, the RPC call keeps breaking in production.

Note: Never change the data type of a field, if there is any need just add a new field with a newer data type and ask the client to consume value from the newer field.

It should be this way

message WalletInfo {
    string type_of_wallet = 1;
    //deprecated field
    int64 wallet_balance = 2;
    double double_type_wallet_balance = 3;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}

The best part of the proto un-marshalling process is that it only checks for sequence numbers that the client has and if the server is sending some additional data it will be ignored. So in this way, the server can hit production first and the client can pull the recent proto, make required application changes and then hit the production.

Now comes the question, can we re-use the deprecated fields for something else? Well, you can but it might cause some erratic results based on how the client is interpreting it. The best way is to only increment the sequence and never deprecated ones. To add more meaning to deprecated ones, proto has a keyword called reserved.

message WalletInfo {
    reserved 2;
    string type_of_wallet = 1;
    double double_type_wallet_balance = 3;
}
message UserWalletResponse {
    repeated WalletInfo wallet_info = 1;
}

This reserved keyword helps us in a way that no one mistakenly reuses a sequence number that’s deprecated.

Case-3 (The classical string marshalling and unmarshalling problem. Exception of Case-2)

Consider a case where the user is trying to update his/her details and initial proto looked this way.

message UserDetails {
    string first_name = 1;
    string last_name = 2;
    string phone_number = 3;
}

Let's say both client and server holds the same proto. Now the front end thought of changing the phone_number to a repeated string wherein the user can add all the mobile numbers he/she uses.

Without notifying the frontend service, the backend updated the field type from string to repeated string.

Client Request Structure
message UserDetails {
    string first_name = 1;
    string last_name = 2;
    string phone_number = 3;
}
Server Request Structure
message UserDetails {
    string first_name = 1;
    string last_name = 2;
    repeated string phone_number = 3;
}

As discussed earlier this should fail while the server tries to un-marshall the request but that ain’t the case here. Consider below example

Client Request
UserDetails {
    first_name = "mourya";
    last_name = "venkat";
    phone_number = "99493XXXXX";
}
Server Request
message UserDetails {
    first_name = "mourya";
    last_name = "venkat";
    phone_number = ["9","9","4","9","3","X","X","X","X","X"];
}

So instead of failing to un-marshall, it tries to convert string to repeated string by splitting the string per character. I’m not completely sure of why this is designed this way but it’s one of the edge cases which I’ve encountered on production.