This article looks at a simple scalability scenario using so called “singleton” configurations of Cubeia Firebase 1.7 Enterprise Edition. It is looking at high load scenarios on limited hardware, and will discuss the implications of the test result.
Topology
When a Firebase server is used stand-alone the configuration is called “singleton”, and that server will run all necessary roles complete functionality (master, client, game and tournament). Normally a multi-node Firebase Cluster is configured in a role based topology where different servers handle different roles in the cluster. For example, you may run client sessions on dedicated machines and all game play on others. However, for smaller installations it may be easier to run several singleton servers side by side.
Setup
The tests were performed on Cubeia Ltd’s load test environment in Stockholm, Sweden. The participating servers were configured as follows:
- Dual Core, Intel Xeon 2.4 GHz
- Linux CentOS, Kernel 2.6.18 (EPOLL enabled)
- Oracle Java HotSpot 64-Bit Server VM (16.3-b01, mixed mode)
- 1280M allocated heap memory
- Cubeia Firebase 1.7-EE Release Candidate 3
As load test game Cubeia Poker was used (nightly build 18th Aug 2011), and the sample values were:
- CPU load (“cpu idle” sampled from system)
- Hands per minute (poker hands executed)
- Actions per second (number of game actions executed)
Game Configuration
The game was configured using one speed for all tables, and 10 seats per table. The bots were configured to react within 2.5 and 3.3 seconds on normal actions (see “player speed and hands per minute” below”).
No external integrations such as wallet or databases were used in order to test the Firebase performance isolated. In real world scenarios the performance would be greatly affected by the latency and throughput of such integrations.
All bots were subscribing to the entire lobby. For a large installation (10 000 players or more) this is not recommended as it generates a massive load. Cubeia Firebase has the ability to partition the lobby so that players only needs to subscribe to a part of the whole lobby.
Actions Versus Hands
The actions sampled represents the total number of actions executed by the poker game. This value is directly linked to hands per minute as shown below, using samples from the single server scenario.
As the graph shows, the relationship between hands per minute and actions per second appears to be fixed. Exploring the the full sample set the relationship between these two values seems to fluctuate between 2.3 and 2.6 (events per second / hands per minute).
Player Speed and Hands per Minute
Another relationship which is expected to scale linearly is between the number of players and hands per minute. The bots used for this test reacted to requests in between 3.3 and 2.5 seconds, thus a full table with 10 players is expected to generate approximately one event every 2.8 seconds and, for example, 2 000 players (200 tables) approximately 56 events per second.
Again the graph is sampled from a single server. The relationship between the number of tables and the number of of events seems to by slightly less than assumed above, but this is to be expected when hand start/end and system latency is included in the calculation.
Performance
Limiting Factors
Each test run was ended when the CPU load of the machines reached up towards 80%. In reality the limiting factor was the CPU in relationship with the Java VM Garbage Collector (GC): as the number of player grows, the initial snapshot for lobby subscription gets larger, and when the available overhead gets too small the GC will fail to release enough memory. Large installations (10 000 players or more) should consider allocation significantly more memory to server “client” roles.
This limitation is not unique for singleton-style clusters however: even in mixed topologies large lobbies will put higher demands on the client role heap size.
CPU
The following chart shows the CPU load for 1-3 server together with the corresponding number of players connected.
We can see that the load peaked at 16 000 players for a single server, 24 000 for 2 servers and 40 000 for three servers.
Going from 1 to 2 server increases the maximum load with approximately 50%. This is because a single server does not have to replicate state for fail over, and as such will not scale linearly compared to a single server.
Hands per Minute
The following chart shows the hands per minute for 1-3 servers together with the corresponding number of players connected.
The game performance scales almost perfectly across all servers, and the peak was reached at 2 763 hand per minute when 40 000 players were connected.
The speed of the bots seems to correspond to a factor of 13-14 in relation to the hands per minute, so for example, we would expect approximately 1 400 hands per minute for 20 000 players.
Conclusion
Cubeia Firebase 1.7 Enterprise Edition shows healthy performance and scalability numbers even when configured to use singleton nodes. The hardware for the tests is a few years old and is lacking on RAM, it is recommended that very large installations (20 000 players or more) is set-up on hardware with significantly more memory available.