Cluster of 200 Nodes

Most Akka clusters will probably consist of 2 to 10 nodes, but we were curious to know how far we could push it. We are happy to share that we have successfully tested an Akka 2.2.0 cluster with 200 nodes. This is mind blowing, considering that this is the first official version of Akka cluster. This post details our experiences during the experiment.

Before we all get too excited it is important to point out that there are alternatives to one single large cluster, such as splitting up the system into several smaller clusters, or worker nodes that are not part of the core cluster. The Cluster Client can be a nice tool for this purpose.

The Test

The used test is akka.cluster.StressSpec. It performs the following steps, when configured with 200 nodes. Each node is a separate JVM running one actor system.

  1. Start and join 3 seed nodes and join 32 more nodes to the seed nodes
  2. Join nodes one by one, up to cluster size 67
  3. Join 32 nodes at the same time to one single node
  4. Join 69 nodes at the same time to the seed nodes, thereafter the cluster consists of 168 nodes
  5. Join nodes one by one, up to cluster size 200
  6. Exercise additional join/remove behavior
  7. Idle for a while
  8. Gracefully leave nodes one by one, down to cluster size 168
  9. Hard shutdown nodes one by one, down to cluster size 136
  10. Leave 32 nodes at the same time
  11. Shutdown 32 nodes at the same time, thereafter the cluster consists of 72 nodes
  12. Hard shutdown nodes one by one, down to cluster size 56
  13. Gracefully leave nodes one by one, down to cluster size 40

This test was run 4 times with different configuration of number of nodes; 50, 100, 150 and 200 nodes. The charts below includes the results for all 4 runs.

Hardware and configuration

It is not a completely realistic test environment, because we were short of physical hardware, but the test gives an indication of what is possible.

  • Five 16 core AMD Opteron 6128 and one 48 core AMD Opteron 6172 hosting up to 199 JVMs in total, and one additional 16 core AMD Opteron 6128 hosting 1 single JVM. The servers are connected on a private network.
  • OS Ubuntu 11.10
  • OpenJDK 7, version 1.7.0_09 (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1).
  • JVM settings: -Xms128M -Xmx256M -XX:+UseNUMA -XX:+HeapDumpOnOutOfMemoryError -XX:-UseBiasedLocking -XX:+UseCompressedOops -XX:+UseCondCardMark -XX:+UseParallelGC
  • Akka version: 2.2.0
  • Akka configuration deviation from default and what is specified in StressSpec:
    akka.cluster.gossip-different-view-probability=0.3
    akka.remote.log-frame-size-exceeding=5000b
    MultiJvm.akka.cluster.Stress.nrOfNodes=200
    akka.test.cluster-stress-spec.exercise-actors=off
    akka.test.cluster-stress-spec.nr-of-nodes-factor=16
    akka.test.cluster-stress-spec.infolog=on
    akka.test.cluster-stress-spec.idle-gossip-duration=120s
    akka.test.cluster-stress-spec.expected-test-duration=7200s

Results

One cluster scalability aspect is to measure how long time it takes to join one new member to an existing cluster, and compare how the size of the cluster affects that time. We measure the time it takes until all members have seen the new member as ‘Up’.

image

Normally it takes 10 to 20 seconds to join one new member, and increased size of the cluster does not drastically increase the duration.

What happens if we join many nodes at the same time. Good news. It does not take much longer than joining one node, i.e. it is more efficient to join many at the same time than it is to join them sequentially. The reason for this is that the membership changes are batched before being spread by the periodic gossip. One data point from the test is that it takes 21 seconds to join 69 nodes to a cluster of 99 nodes. Here we have used 3 seed nodes. It might be less efficient when joining to more seed nodes.

Removing nodes, including failure detection, is also important in an elastic cluster. When shutting down nodes hard—System.exit of the JVM—the time for detection, downing and until all members have seen the removal looks like this:

image

A more graceful exit can be performed if you tell the cluster that a node shall leave. For example cluster aware routers take advantage of such removal by avoiding usage of nodes that are leaving. Leaving takes longer time, since there are more steps involved.

image

It is not unrealistic that several nodes become unreachable at the same time. Hard shutdown of 32 nodes in a cluster with 104 members takes around 20 seconds. We got similar results for leaving several nodes at the same time. 24 seconds to leave 32 nodes in 136 nodes cluster.

How much resources does the cluster itself consume? To be able to answer this we used one dedicated machine running one single JVM. The charts below shows the CPU and network usage of this machine during the test with 200 nodes. Note that this test represents an active cluster and you can expect less CPU and IO usage when there are less membership changes.

image

image

Configuration Considerations

One configuration option that should be adjusted when using clusters with more than 50 nodes is akka.cluster.gossip-different-view-probability. This probability setting is used for the biased selection of the node to gossip to. When the cluster is large the default value (0.8) means a too high total probability that many nodes selects the same target, resulting in a storm of gossip messages to one node. In this stress tests we have used 0.3, but for a production cluster of 200 nodes it should be set to 0.1 or lower to reduce the risk of overloading nodes with many gossip messages. The drawback is slightly longer dissemination times when the cluster is small. In next version we plan to adjust this probability automatically with the size of the cluster.

There are some other configuration settings that can be considered to be amended for large clusters, mostly to reduce the amount of transfered data. We have used default values for the test results presented here.

  • akka.cluster.failure-detector.heartbeat-interval=2s and akka.cluster.unreachable-nodes-reaper-interval=2s - less frequent heartbeat messages, trade-off: longer time to disseminate cluster membership changes
  • akka.cluster.gossip-interval=2s - less frequent gossip messages, trade-off: longer time to not as quick failure detection
  • akka.cluster.metrics.enabled=off - if you don’t use the AdaptiveLoadBalancingRouter you can turn off metrics

Aiming for the Next Record

Of course we tested with an even larger cluster. When we tried with 250 nodes the test timed out in various places, and the failure detector reported nodes as unreachable, which is expected from an overloaded system. We learned several things from pushing the cluster to the limit and these tests have been invaluable in the effort of hardening the remoting and clustering of Akka 2.2.0.

It is great to see that it is possible to run a cluster of 200 nodes. We will continue to push the scalability of Akka cluster in upcoming versions and setting new large cluster records. Areas that can be optimized to reduce the cost of some operations have been identified. We also have several ideas of how to make the gossip protocol more efficient.

Recent comments

Blog comments powered by Disqus

2 Notes

  1. hakkers posted this