We’re gearing up for the 1.8 release of Cubeia Firebase which will include native server-side support for HTML5 in the form of WebSockets and Comet, and naturally we want to make sure the performance is up to our normal standard. In this article we’l compare the standard Firebase binary socket communication with web sockets.
Setup
The tests were performed on Cubeia Ltd’s load test environment in Stockholm, Sweden. The participating server was configured as follows:
- Dual Core, Intel Xeon 2.4 GHz
- Linux CentOS, Kernel 2.6.18 (EPOLL enabled)
- Oracle Java HotSpot 64-Bit Server VM (16.3-b01, mixed mode)
- 1280M allocated heap memory
- Cubeia Firebase 1.7-EE Release Candidate 3
The Cubeia Firebase load test game was used, and the sample values were:
- CPU load (“cpu idle” sampled from system)
- Throughput (number of game actions executed)
- Response time (average sampled from the bots)
- Bandwidth (sampled from system)
Binary vs Web Socket
The semantics between the two sockets are similar, both are TCP-based using control characters for frame-demarcation. The main difference is that of the frame payload, whereas the standard binary socket uses a native Styx binary packaging, the Firebase implementation of web sockets uses the same packet format encoded in JSON which is a text-based encoding over UTF-8. This will impact the bandwidth reading, and also the CPU performance.
Throughput
We’ll start with throughput as it is an important factor for reading the other results. The load test uses a bot implementation that sends one event per second and bot. Thus the theoretic MAX value for 100 bots are 100 events per second.
The above chart shows the theoretical max, which equals the number of started bots, with the actual throughput in terms of events per second. We can see that the performance is very similar between the two sockets. In fact, it’s not until approx. 1500 events per second the measured difference is bigger than 1%.
For the rest of the performance results below it pays to remember that the throughput of web sockets trailed off around 1500 events per second as all results will be reported with the theoretical max events per second in order to make them comparable.
CPU Idle
The chart below show the idle percentage on the server as the load increases.
It’s interesting to see that both sockets shows a decline in CPU use around 1500 events per second. This corresponds to the decreasing web socket through put, and one can clearly see on the graph that the diminished throughput makes the CPU utilization grow in parallel between the sockets.
Response Time
Now lets look at response times. These values are sampled from the bot server and averaged over the last 1000 requests. As such it represents a sliding window over the last 1000 events.
Again we see exactly where the throughput of the web socket implementation started diverge. Still, at 2000 events per second the response times are well under 60 milliseconds which is more than enough.
Bandwidth
The bandwidth should be a fixed percentage higher in the web socket implementation than that of the binary socket. The size of the game data does not matter so much as Firebase treats the game data as a byte array, and will Base64-endcode it during transport.
At 1500 the web socket bandwidth tapers off as the throughput slows down. Up to this point we can see that web sockets consume approx 20% more bandwidth than does the corresponding binary socket. However, this test was performedwithout measuring lobby data which would probably slew the result significantly.
Conclusion
After this first glance we can conclude that web sockets performs really well compared to their binary counterparts. Even the bandwidth difference is not too concerning when the lobby data is not included. For very intensive applications binary sockets still reign supreme but in most cases web socket will prove to be more than enough.