Comparing Java WebSockets: Jetty vs. Netty

Credits: Symphony Solutions

This article shows us a step-by-step process of comparing two WebSocket implementations, Jetty and Netty, in Java using standard testing tools.

We are going to compare the two implementations by the following attributes: average and maximum response time, memory consumption, maximum used threads for small, average, and large data.

What you need

• Java 8
• JMeter 5.1.1
• Java IDE
• Maven 3
• VisualVM or any Java profiler

What you will use

We are going to be working with two WebSocket implementations :

• Jetty from Eclipse foundation
• Netty open-source Java implementation of Socket.IO server.

Download source

First of all, you need to download the source code from https://github.com/vhavryk/compare-jetty-netty.

I have changed demos from both projects to run the load test properly using JMeter. I have added to both projects a new file DataFileLoader that loads data from the file using env variable “data-file-name”. Later on, we will use the data for sending from server to client (JMeter).

Jetty WebSocket server

I have changed two files EventServer and EventSocket:

So, we can send data from server to clients after the connection is open.
To build a project, we need to go to native-jetty-websocket-example folder and run the command:

mvn clean install

Go to the target folder and run the command:

java -Ddata-file-name=small.json jar native-jetty-websocket-example-1.0-SNAPSHOT.jar

After that, the Jetty server starts using data from small.json file.

Netty WebSocket server

I have changed one file ChatLauncher:

I added support for reusing address to fix the issue with JMeter WebSocket plugin. Also, I changed the data recipient from “all clients” to “one connected client”. To build the project, we need to go to netty-socketio-demo/server folder and run the command:

mvn clean install

After that, go to the target folder and run the command:

java -Ddata-file-name=large.json -jar demo-1.0.0-SNAPSHOT.jar

Netty server starts using data from large.json file.

Load test configuration

We are going to use the JMeter tool to load test both servers with the help of JMeter WebSocket Samplers.
I have created two different JMeter scenarios for Netty and Jetty servers due to the Jetty server requiring three frames to read data, while the Netty server requires five. You can find both scenarios in the JMeter folder. Also, please ignore the errors in the Netty scenario because the plugin we are using is not fully compatible with the Netty WebSocket implementation and it can’t handle response on close events properly.

The general load test configuration is presented in the table below:

Load test methodology

We are going to collect the following data from the load test:

• Data file size (KB) — the data that the server will send to the client
• Average response time on opening connection (ms)
• Maximum response time on opening connection (ms)
• Average response time on reading data (ms)
• Maximum response time on reading data (ms)
• Maximum heap size (MB)
• Maximum created threads

Collecting these data will be done using the next steps:

1. Run the server with -Ddata-file-name={filename} parameter
2. Run JMeter and open the corresponding scenario from the JMeter folder
3. Run the scenario and collect aggregation report data
4. Run VisualVM or any Java profiler and connect to the server
5. Run the scenario and collect monitoring data: memory and number of created threads
6. Close/Stop profiler and server.
7. Repeat the steps for the next data file.

I have already completed all the steps and collected information in the reports folder.
Next, we will analyze the results.

Analysis of the load test with small data

As you can see, Jetty has a shorter response time than Netty on small files.
Maximum heap sizes are comparable. But Netty uses fewer threads.

Winner: Jetty

Analysis of the load test with middle data

There is no clear winner in this case:

• Jetty is slightly better in opening connections but consumes more memory and uses more threads.

Analysis of the load test with large data

In this case, we have two winners:

• Jetty is better in regard to response time.
• Netty is better in regard to memory consumption and the number of threads.

General analysis

Based on the aggregation report, we can find that Netty has a longer maximum response time than Jetty but consumes less memory and creates fewer threads. Both servers struggle with maximum times, especially Netty. So, to use these servers in production you need to find solutions for handling long response from the server to the client (using timeouts and retries) or/and server.

Final words

In general, there is no clear winner but, for small data size in client-to-server data transfer (under 2KB), it is better to use Jetty. For average data size or anything over 10KB, it’s better to use Netty due to optimal usage of memory and threads.