Cluster Health, State and Fault Tolerance

The Akka 2.1 Cluster offers a wealth of insight into the health and state of the cluster at any given time, and time sensitivity of this data is easily configurable. One of the immense benefits of this internal PubSub event system is that by subscribing to events of interest on the health and state of nodes, you can react and take action to negative anomalies. But even better, one can proactively manage and tune the cluster toward a highly robust system and decrease the potential of failures all together.

There are a few major players in the public-facing fault tolerance of the cluster:

  • The cluster event system
  • Cluster node membership
  • Cluster node failure detector and heartbeat

The internal cluster system keeps track of cluster state as the various internal workers gather real-time data on the cluster’s state and health, and for some of these data points, the worker nodes internally Gossip data to the other nodes to share the information. How real-time the data snapshots are, is fully dependent on your configuration, and relevant to information on the type of cluster, cluster state, and membership in / state of the node ring.

Of the many cluster configurations, here are a few of interest in failure detection and general health

Cluster nodes send heartbeats to track any nodes that may become unreachable. The AccrualFailureDetector is a part of the node heartbeat infrastructure, analysing heartbeat stats for each Akka Address. This failure detector strategy employs a suspicion level and a configurable threshold to determine whether an anomaly has occurred, based on a cumulative distribution function calculation. This is dynamically adjusted to reflect the current network conditions. Because any cluster health metric must be able to handle sudden anomalies that may not be indicative of failure, the failure detector has a configurable margin of error [see akka.cluster.failure-detector.acceptable-heartbeat-pause above].

There are many granular cluster domain events one can subscribe to, relative to node ring membership which is continuously tracked internally. Imagine knowing when a node becomes unavailable due to network issues or node crashing; you could automate bringing a new node or an array of nodes online to handle your traffic. With Akka 2.1 cluster membership you can subscribe to node ring changes such as when a node has joined, is up, left, exited, is unreachable, downed, or removed. The full cluster state [akka.cluster.ClusterEvent.CurrentClusterState], is published to a subscriber once on startup to offer an initial snapshot, followed by any incremental changes thereafter.

So what does the external API look like? Of all the intricacies and inner workings of the cluster itself, the public API is extremely simple to use:

My next post will go into depth on the new AdaptiveLoadBalancingRouter, its implementations, the new cluster metrics API, and their combined roles in contributing to overall cluster health and node uptime.

Helena Edelson
Twitter: @helenaedelson

Recent comments

Blog comments powered by Disqus

1 Notes

  1. hakkers posted this