Let it crash

2.2 Spotlight: Cluster Singleton

patriknw — Tue, 18 Jun 2013 09:27:40 +0200

Using a singleton should not be the first design choice, but for some use cases it is convenient and sometimes also mandatory to ensure that you have exactly one actor of a certain type running somewhere in the cluster.

Therefore, this specific use case is made easily accessible by the Cluster Singleton Pattern in the contrib module. It manages singleton actor instance among all cluster nodes or a group of nodes tagged with a specific role. The singleton actor is always running on the oldest member. The cluster failure detector will notice when oldest node becomes unreachable due to things like JVM crash, hard shut down, or network failure. Then a new oldest node will take over and a new singleton actor is created.

2.2 Spotlight: Oldest Cluster Member

patriknw — Fri, 14 Jun 2013 14:09:56 +0200

The cluster knows the order members were added to the cluster. akka.cluster.Member has a method isOlderThan to compare two members by their age. This is useful when you need a stable order of the nodes.

For example, this is how you can keep track of the members, sort them by age, and send each message to the first 3 nodes:

2.2 Spotlight: Cluster Node Roles

pionic — Tue, 11 Jun 2013 02:30:39 +0200

Not all nodes of a cluster need to perform the same function: there might be one sub-set which runs the web front-end, one which runs the data access layer and one for the number-crunching. Deployment of actors—for example by cluster-aware routers—can take node roles into account to achieve this distribution of responsibilities.

The roles of a node is defined in the configuration property named akka.cluster.roles and it is typically defined in the start script as a system property.

2.2 Spotlight: Cluster Joining Improvements

patriknw — Fri, 07 Jun 2013 16:26:57 +0200

The most important change is that an actor system can only join a cluster once. Additional attempts will be ignored. When it has successfully joined it must be restarted to be able to join another cluster or to join the same cluster again. It can use the same host name and port after the restart, but it must have been removed from the cluster before the join request is accepted. Unsuccessful join attempts are automatically retried.

These strict rules means that you can safely restart a process without any confusion between the state of the old and new member with same host name and port.

Startup of seed nodes has also been improved. Seed nodes and other nodes can be started in any order. The node configured as the first element in the seed-nodes configuration list must be started when initially starting a cluster, but thereafter it can be stopped and started again just like any other node. All nodes can have the same configuration and start script.

Your feedback pushed us to an improved solution for how nodes can join the cluster. Thanks for reporting!

Akka 2.2.0-RC1 Released

pionic — Tue, 04 Jun 2013 12:27:48 +0200

Dear hAkkers,

we—the Akka committers—proudly announce the FIRST RELEASE CANDIDATE for Akka 2.2.0 “Coltrane”. Half a year has passed since the release of Akka 2.1.0 “Mingus” and much has happened in our code base. User-visible API changes have been kept to a minimum, though, as most work happened under the hood; more on that later. First let us take a look at what Coltrane brings you:

Cluster support (with a big thank-you for the feedback throughout the development cycle); we have continued to improve on the preview offered with Mingus, e.g. adding node roles or distributed pub-sub or the cluster client.
Fully actor-based remoting implementation with support for multiple transports and high-level failure detection.
An extremely scalable actor-based network IO layer with SSL support and a protocol pipeline infrastructure.
Reliable supervision and deathwatch for remote actors.
An experimental preview of Typed Channels.

This is just a very high-level overview of the biggest pieces, there have also been improvements in other areas like test coverage of our OSGi bundles, or how actor failures are logged, or in the semantics of ActorContext.unwatch(), or a more performant rewrite of Agents, or how dispatchers and mailboxes can be configured outside of your code.

This is what we intend to ship as 2.2.0 final unless issues are found, so please test it thoroughly and report back. Failures are important to hear about, but praise also does not hurt :-)

What changes for existing code?

We have continued on our path to unify the semantics between local and remote actor references. The most obvious difference was that remote references were bound to a name whereas local ones were bound to a specific actor lifecycle. If the local actor stops, the reference stops working, which was not the case for the remote reference in case a new actor was created at the same path. We changed it such that the local rules apply to remote references as well, making the “self” references of actors created subsequently with the same name distinct from each other. As a consequence, ActorRef equality was changed by taking into account the actor’s UID (which you can see in ActorRef.toString now).

The other most visible change concerns the creation of Props. Using anonymous inner classes as factories for your actors means that the required “$outer reference”—the reference to the enclosing scope—will have an influence on the serializability of your Props as well as on the thread-safety. Therefore we have rebased the inner workings of Props on the desired actor’s Class<?> and its constructor arguments. The benefit is that no closures are formed which would invite you e.g. to call methods on the enclosing actor, and also that serialization bindings are used to serialize the constructor arguments. Previously all Props were serialized only with Java serialization. In order to make full use of the potential of this approach we have deprecated some of the existing ways to obtain Props and introduced new ones.

Besides these changes there have been several smaller modifications, please read the migration guide to see if you may need to adapt your application while upgrading.

Where do I get it from?

The artifacts have been published to Maven Central as usual, but this time for two different Scala versions:

using Scala 2.10.1

“com.typesafe.akka” % “akka-actor_2.10” % “2.2.0-RC1”

using Scala 2.11.0-M3

“com.typesafe.akka” % “akka-actor_2.11.0-M3” % “2.2.0-RC1”

Development Statistics (compared to v2.1.0)

1044 files changed, 85693 insertions, 35814 deletions, 23 committers

commits added removed

commits added removed

 127   20030    9910 Patrik Nordwall
  99   20073    8543 Roland
  79   17228    9401 Endre Sándor Varga
  69    7482    6618 Viktor Klang
  49    4820    2677 Björn Antonsson
  20    1610     422 Johannes Rudolph
  18    1586     942 Mathias
  10     135     175 Dario Rexin
   9    1764     440 Rich Dougherty
   5     311     139 RickLatrine
   5    1463     306 Christophe Pache
   3    1212     490 Raman Gupta
   2      83      20 Kaspar Fischer (hbf)
   2      12      12 Ricky Elrod
   2      95      53 Kevin Wright
   2     163      66 Raymond Roestenburg
   2      48      29 Jonas Boner
   1      10       3 Michael Pilquist
   1     548      77 Helena Edelson
   1      38      24 Matthew Neeley
   1       8      10 Peter Vlugter
   1       3       3 Thomas Lockney
   1      36      35 Derek Mahar

Thanks for all the external contributions, 23 committers is quite an outstanding number for a toolkit like Akka.

Assembla milestones:

https://www.assembla.com/spaces/akka/milestones/2690943-2-2-m1

https://www.assembla.com/spaces/akka/milestones/2896213-2-2-m2

https://www.assembla.com/spaces/akka/milestones/2896943-2-2-m3

https://www.assembla.com/spaces/akka/milestones/3089253-2-2-0-rc1

In total we closed 508 tickets on these four milestones.

Akka is released under the Apache V2 license.

Happy Hakking!

Akka 2.1.4 Released

drewhk — Tue, 14 May 2013 15:58:09 +0200

Dear hAkkers,

We—the Akka committers—are pleased to be able to announce the availability of Akka 2.1.4. This is the fourth maintenance release of the 2.1 branch, containing documentation improvements and fixing several issues including:

CallingThreadDispatcher deadlock problems
Stack overflow while loading messages from persistent mailbox
More robust recovery in remoting in case of malformed messages
Proper deployment of typed actors

… and several smaller fixes

IMPORTANT NOTE:

Release 2.1.3 was broken due to a bug in the release scripts and build process, please use this release instead. There were no code changes between 2.1.3 and 2.1.4.

This release is backwards binary compatible with version 2.1.0, 2.1.1, and 2.1.2, which means that the new JARs are a drop-in replacement for the old ones (but not the other way around). Always make sure to use at least the latest version required by any of your project’s dependencies.

Known Issues

ActorSystem.shutdown() does not terminate all threads if akka.remote.netty.use-dispatcher-for-io is set to true

Migrating from Older Releases

When migrating an existing project from Akka 2.0.x please have a look at our migration guide:
http://doc.akka.io/docs/akka/2.1.4/project/migration-guide-2.0.x-2.1.x.html

When migrating from the Akka 1.3.x series please follow first the migration guide towards version 2.0.5:
http://doc.akka.io/docs/akka/2.0.5/project/migration-guide-1.3.x-2.0.x.html

About the Cluster Support

The “akka-cluster” module is published under the name “akka-cluster-experimental” to emphasize that its status is not yet final. This denomination is not due to sub-par standard of the module; the cluster support has been tested thoroughly and it works as documented. The reason for the “experimental” tag is that this rather important module is now presented to the general public for the first time, and although we have received valuable feedback from early adopters we anticipate possible API changes in order to meet all of your requirements. Work is continuing on Akka’s cluster support, and we will formally declare it officially supported and stable with the next major release—Akka 2.2 ‘Coltrane’. Please help us make it the best possible solution by continuing to give feedback on the mailing list and telling us what can be improved.

IMPORTANT NOTICE

The artifacts comprising this release have been published to https://oss.sonatype.org/content/repositories/releases/ and also to Maven Central. In addition, we adopted the SBT standard of encoding the Scala binary version in the artifact name, i.e. the core actor package’s artifactId is “akka-actor_2.10”.

2.1.4 compared to 2.1.2:

12 tickets closed
40 files changed, 437 insertions(+), 222 deletions(-)
… and a total of 5 committers!

Fixed Tickets:

2911 DOC: IOManager error handling
3163 DOC: Misplaced headings in cluster docs
3180 Backport CallingThreadDispatcher fixes
3182 Stack overflow when loading messages using persistent mailbox
3220 backport fix for ActWithStash vs. lifecycle hooks
3241 remoting breaks down with protobuf errors (truncated messages)
3250 Backport: 3182 - Stack overflow when loading messages using persistent mailbox
3276 actorSelection(/…) should start matching at rootGuardian
3282 publish EPUB docs for 2.1.x
3328 backport FSM.nextState memory retention fix
3336 TypedProps ignores deployment options
3338 2.1: update to genjavadoc 0.5

Credits:

commits added removed
      7   131      49 Endre Sándor Varga
      7   110      14 Roland
      2    96      88 Björn Antonsson
      2    30       8 Viktor Klang
      1    15      14 Patrik Nordwall

Website:

http://akka.io

Binary distribution:

http://download.akka.io/downloads/akka-2.1.4.tgz
http://download.akka.io/downloads/akka-2.1.4.zip

Documentation:

http://doc.akka.io/docs/akka/2.1.4/
http://doc.akka.io/japi/akka/2.1.4/
http://doc.akka.io/api/akka/2.1.4

Issue tracker:

http://www.assembla.com/spaces/akka/

Mailing list:

https://groups.google.com/group/akka-user

Akka is released under the Apache V2 license.

Akka 2.1.3 Released - IMPORTANT: this release is broken

drewhk — Mon, 13 May 2013 21:54:00 +0200

IMPORTANT NOTICE

Release 2.1.3 is broken due to a bug in the release script. Please use 2.1.4 instead.

The original announcement for 2.1.3 is available below

Dear hAkkers,

We—the Akka committers—are pleased to be able to announce the availability of Akka 2.1.3. This is the third maintenance release of the 2.1 branch, containing documentation improvements and fixing several issues including:

CallingThreadDispatcher deadlock problems
Stack overflow while loading messages from persistent mailbox
More robust recovery in remoting in case of malformed messages
Proper deployment of typed actors

… and several smaller fixes

This release is backwards binary compatible with version 2.1.0, 2.1.1, and 2.1.2, which means that the new JARs are a drop-in replacement for the old ones (but not the other way around). Always make sure to use at least the latest version required by any of your project’s dependencies.

Known Issues

Due to the restriction imposed by binary compatibility—which is kept for the patch releases within a minor release such as 2.1.x—not all known issues can be fixed. The currently known issues in the 2.1.x series are:

ActorSystem.shutdown() does not terminate all threads if akka.remote.netty.use-dispatcher-for-io is set to true

Migrating from Older Releases

When migrating an existing project from Akka 2.0.x please have a look at our migration guide:
http://doc.akka.io/docs/akka/2.1.3/project/migration-guide-2.0.x-2.1.x.html

When migrating from the Akka 1.3.x series please follow first the migration guide towards version 2.0.5:
http://doc.akka.io/docs/akka/2.0.5/project/migration-guide-1.3.x-2.0.x.html

About the Cluster Support

IMPORTANT NOTICE

The artifacts comprising this release candidate have been published to https://oss.sonatype.org/content/repositories/releases/ and also to Maven Central. In addition, we adopted the SBT standard of encoding the Scala binary version in the artifact name, i.e. the core actor package’s artifactId is “akka-actor_2.10”.

2.1.3 compared to 2.1.2:

12 tickets closed
40 files changed, 437 insertions(+), 222 deletions(-)
454 pages of docs vs 454 in v2.1.2
… and a total of 5 committers!

Fixed Tickets:

2911 DOC: IOManager error handling
3163 DOC: Misplaced headings in cluster docs
3180 Backport CallingThreadDispatcher fixes
3182 Stack overflow when loading messages using persistent mailbox
3220 backport fix for ActWithStash vs. lifecycle hooks
3241 remoting breaks down with protobuf errors (truncated messages)
3250 Backport: 3182 - Stack overflow when loading messages using persistent mailbox
3276 actorSelection(/…) should start matching at rootGuardian
3282 publish EPUB docs for 2.1.x
3328 backport FSM.nextState memory retention fix
3336 TypedProps ignores deployment options
3338 2.1: update to genjavadoc 0.5

Credits:

commits added removed
      7   131      49 Endre Sándor Varga
      7   110      14 Roland
      2    96      88 Björn Antonsson
      2    30       8 Viktor Klang
      1    15      14 Patrik Nordwall

Website:

http://akka.io

Binary distribution:

http://download.akka.io/downloads/akka-2.1.3.tgz
http://download.akka.io/downloads/akka-2.1.3.zip

Documentation:

http://doc.akka.io/docs/akka/2.1.3/
http://doc.akka.io/japi/akka/2.1.3/
http://doc.akka.io/api/akka/2.1.3

Issue tracker:

http://www.assembla.com/spaces/akka/

Mailing list:

https://groups.google.com/group/akka-user

Akka is released under the Apache V2 license.

2.2 Spotlight: Deprecation of Closure-Taking Props

pionic — Sat, 27 Apr 2013 14:19:00 +0200

From 2.2-M3 onwards you will probably notice several new deprecation warnings when trying out the Akka update on an existing project since all factories taking UntypedActorFactory, Creator<Actor> or Scala by-name arguments will produce warnings. We know that some of you will—at first sight—not like this, but please bear with us because we do this for a very good reason: we want to make your code safer!

The background for this change is that we have seen on many occasions code like the following:

… or in Scala:

It is an easy mistake to make and sometimes harmless, until you happen to call a method which is not thread-safe (as shown above) or you want to make this a remote deployment. In the first case you break actor encapsulating by implicitly passing “this” into the closure—deviously hidden by closure syntax—introducing race conditions and releasing the whole host of concurrency demons. In the second case the deployment will fail mysteriously because the enclosing class—often an Actor—is not serializable.

Akka’s mission is to make your life less frustrating, so we chose to replace those hidden failures which are hard to debug by early warnings from your dearest friend, the compiler.

So What Shall I Do?

The foremost use of the deprecated methods was to construct actors which take arguments. We have made this even simpler than before:

This syntax actually improves the LoC balance in the Java case considerably. For Scala it looks like this:

And What About Dependency Injection?

Another use-case for UntypedActorFactory was to let another framework perform the actual actor creation, e.g. OSGi/CDI/Spring/you-name-it. For those cases allow me to refer to our documentation for Java and Scala.

What the Future May Bring

For Scala there are two more use cases to consider: the inline construction of ad-hoc actors and the cake pattern—its close relative. The former is dangerous when performed inside another actor for the reasons outlined above, while the latter is usually safe. Therefore we plan on introducing macro-based alternatives which allow your favorite syntax to be used while not suffering from the pitfalls; stay tuned but please do not literally hold your breath, it may take several weeks until more pressing matters are resolved. Until then we are truly sorry for littering your builds with some false warnings.

Akka 2.2-M3 Released

patriknw — Fri, 19 Apr 2013 21:50:53 +0200

Dear hAkkers,

we—the Akka committers—are pleased to announce the THIRD PRE-RELEASE MILESTONE of Akka 2.2 “Coltrane”. As discussed earlier this is to be the final and feature-complete milestone, so please take a good look and give feedback. We will gladly fix all rough edges you report while we will thoroughly test and benchmark and if necessary improve the current state of the toolkit. We do this so that we can then confidently proclaim the first release candidate in a few weeks.

What’s New?

Reliable delivery of system messages, meaning that remote DeathWatch and remote deployment now work properly even if the network fails (DeathWatch worked already for the cluster case)
Failure and DeathWatch communication is reliable (technically the corresponding messages became a system messages to profit from reliable delivery); failure signaling semantics changed such that the supervisor strategy may be invoked earlier than in previous versions (the Failed message “jumps the queue”)
The generation of the Terminated message has been made more intuitive: you will no longer receive it after having unwatched the actor in question; also DeathPactException now leads to Stop instead of Restart in the defaultStrategy
Mailboxes can now be configured separately from dispatchers, either from Props or from deployment configuration section, for example this means that it is easier to use the stashing mailbox
Cluster nodes cannot rejoin a cluster, UIDs have been introduced to prevent communication with systems which were removed from the cluster
ActorRefs now refer to a specific actor incarnation at a path; if you create a new actor at the same path then old ActorRefs will not send to it (this used to be the case only locally, hence we removed the difference in semantics observed with remoting); this change means that actorFor() needed to be deprecated in favor of actorSelection() which has been brought up to speed wrt. remote look-ups
The IO layer learned SSL and WriteFile for TCP, names have been cleaned up for UDP, spray-io’s pipelines have been incorporated, complete samples including ACK- and NACK-based back-pressure have been added and more documentation updates will be coming
Props have been restructured to not rely on closures internally but only on Class[_] and arrays of (serializable) arguments; this has been done to avoid closing over actor internal state or other non-serializable data; consequently the closure-taking Props(new Actor …) and the UntypedActorFactory have been deprecated (in the Java case this usually even saves a few lines of code on your end); there will be a more detailed blog post on this soon
Lots of little fixes and hardening, especially of the cluster code
Cluster-aware distributed Pub-Sub module and external cluster client

This is a lot and should probably justify the time it took since the last milestone which was released exactly one month ago.

What’s in the Pipeline?

Besides testing and benchmarking there are some things we need to clean up internally before calling it a final, and we also want to comb through especially the JavaDoc since that is just a machine translation of the ScalaDoc and does not everywhere look as complete as it could.

Release Details

When migrating an existing project from Akka 2.1.2 please have a look at our migration guide: http://doc.akka.io/docs/akka/2.2-M3/project/migration-guide-2.1.x-2.2.x.html

v2.2-M2 compared to Akka v2.2-M3:

* 84 tickets closed, see the assembla milesone

* 553 files changed, 32738 insertions(+), 15684 deletions(-)

* 637 pages of docs vs 489 in v2.2-M2

* … and a total of 10 committers!

Special thanks go to Christophe and Raman for their contribution of an OSGi sample application, and to Mathias and Johannes of spray.io for their continued work on the IO layer.

Akka is released under the Apache V2 license.

Credits:


commits added removed
   20    6439    2396 Patrik Nordwall
   13    4035    2077 Viktor Klang
   10    7806    2061 Roland
    8    1781     341 Björn Antonsson
    8    5586    3101 Endre Sándor Varga
    8     265      93 Johannes Rudolph
    4      40      22 Mathias
    3    1212     490 Raman Gupta
    3    1336     240 Christophe Pache
    1     129      23 Rich Dougherty

Take it for a spin! Happy hAkking!

Akka 2.2-M2 Released

patriknw — Mon, 18 Mar 2013 15:17:33 +0100

Dear hAkkers,

We—the Akka committers—are pleased to be able to announce the availability of Akka 2.2-M2 (PRE-RELEASE MILESTONE TWO).

It is a preview on what is going to become Akka 2.2, so please give us feedback!

The most notable features in this release are:

Built and tested against Scala 2.10.1.
Comes with true Javadoc generated by the GenJavaDoc plugin.
Upgrade Netty to 3.6.1 and clean up shutdown of threads that have been borrowed by Netty.
Actor failures are logged by the supervisor, which makes it possible to customize the the logging in the supervisor strategy.
Reduced log output of the new remoting.
Cluster node roles. Cluster singleton for group of nodes tagged with a role. Cluster aware routers restricted by role. Cluster startup once all required per-role node counts are reached.
Cluster member events are published when state change first seen and events for non-leader transitions have been removed.
Optimizations of the cluster gossip protocol.

When migrating an existing project from Akka 2.1.1 please have a look at our migration guide: http://doc.akka.io/docs/akka/2.2-M2/project/migration-guide-2.1.x-2.2.x.html

Akka 2.2-M2 compared to 2.2-M1:

54 tickets closed, see the assembla milestone
251 files changed, 5168 insertions(+), 3098 deletions(-)
489 pages of docs vs 477 in v2.2-M1
… and a total of 9 committers!

Documentation: http://doc.akka.io/docs/akka/2.2-M2/

Akka is released under the Apache V2 license.

Credits:

commits added removed
   19    2119    1348 Patrik Nordwall
   10     528     371 Björn Antonsson
    6      67      41 Viktor Klang
    5     289     296 Roland
    4     106      81 Endre Sándor Varga
    2     527     128 Rich Dougherty
    2      95      53 Kevin Wright
    1      21       7 Johannes Rudolph
    1      31      19 Mathias

Take it for a spin! Happy hAkking!

Akka 2.1.2 Released

rhshnet — Fri, 15 Mar 2013 20:15:49 +0100

Dear hAkkers,

We—the Akka committers—are pleased to be able to announce the availability of Akka 2.1.2. This is the second maintenance release of the 2.1 branch, where the main changes are that it was built and tested against Scala 2.10.1 and that it comes with true Javadoc generated by the GenJavaDoc plugin.

This release is backwards binary compatible with version 2.1.0, and 2.1.1, which means that the new JARs are a drop-in replacement for the old ones (but not the other way around). Always make sure to use at least the latest version required by any of your project’s dependencies.

Known Issues

Due to the restriction imposed by binary compatibility—which is kept for the patch releases within a minor release such as 2.1.x—not all known issues can be fixed. The currently known issues in the 2.1.x series are:

ActorSystem.shutdown() does not terminate all threads if akka.remote.netty.use-dispatcher-for-io is set to true

Migrating from Older Releases

When migrating an existing project from Akka 2.0.x please have a look at our migration guide

When migrating from the Akka 1.3.x series please follow first the migration guide towards version 2.0.5

About the Cluster Support

The “akka-cluster” module is published under the name “akka-cluster-experimental” to emphasize that its status is not yet final. This denomination is not due to sub-par standard of the module; the cluster support has been tested thoroughly and it works as documented. The reason for the “experimental” tag is that this rather important module is now presented to the general public for the first time, and although we have received valuable feedback from early adopters we anticipate possible API changes in order to meet all of your requirements. Work is continuing on Akka’s cluster support, and we will formally declare it officially supported and stable with the next major release—Akka 2.2 ‘Coltrane’. Please help us make it the best possible solution by continuing to give feedback on the mailing list and telling us what can be improved.

IMPORTANT NOTICE

The artifacts comprising this release candidate have been published to https://oss.sonatype.org/content/repositories/releases/ and also to Maven Central. In addition, we adopted the SBT standard of encoding the Scala binary version in the artifact name, i.e. the core actor package’s artifactId is “akka-actor_2.10”.

2.1.2 compared to 2.1.1:

* 5 tickets closed
* 49 files changed, 361 insertions(+), 287 deletions(-)
* … and a total of 3 committers!

Fixed Tickets: see the assembla milestone

Happy hAkking!

Release Status Update for Akka 2.2 “Coltrane”

pionic — Thu, 14 Mar 2013 14:26:49 +0100

Dear hAkkers,

the end of the first quarter of 2013 is drawing close and many of you are looking forward to the release of “Coltrane” a.k.a. Akka 2.2. The main new feature in this release will be cluster support without the“experimental” tag. We have received some encouraging feedback for the cluster preview shipped withAkka 2.1 and have been developing a few things on top of it ourselves, which has led us to replace and/or polish some parts (the new actor-based remoting comes to mind as well as improving the formulation of the core ActorCell to keep it well-oiled and maintainable) and has also highlighted a few warts which need to be removed. The latter include making the internal parent–child communication reliable in all cases—which means for example changing the way how failure is signaled to the parent—and it also means bringing the life-cycle semantics of local and remote ActorRefs in sync (up to 2.1 a remote ActorRef would look up the current inhabitant of a name on the remote host for each message send, even if that actor terminated int he meantime and a new one was created with the same name).

We also discovered some missing features which would make a lot of sense on the Coltrane milestone, for example the ability to slice a cluster into multiple pieces with different roles, or fixing the message serialization protocol to support interoperability between different versions of Akka in the future. The precise details of how far the latter reaches remain to be determined. We could have placed these features on the next milestone—Rollins—but we felt that they are more urgently needed than the cluster name service.

All in all we have expanded the scope of Coltrane and also spent energy on removing technical debt before it accumulates, which leads to a delay in getting the release out. We will cut a new milestone next Monday which contains some of the aforementioned work, but it will take approximately until the first week of April until we can call Coltrane feature complete. At that point we will release another milestone so all the curious and adventurous among you can try it out in anger, which is also what we will then be doing: in order to deliver the best possible Coltrane release we will concentrate for a few weeks on giving it a thorough beating, writing more tests, running benchmark applications and so on. This will flush out bugs and help us identify and fix remaining usability issues, and we hope that you—our dear and helpful community—will lend a hand with this.

The Second Step: Akka Typed Channels

pionic — Tue, 12 Mar 2013 14:58:00 +0100

The previous post described the ground work, getting geared up to employ the power of macros within Akka and doing type calculations. This post focuses on the user-visible features which are enabled by the macro machinery, presenting a new face for actor communication. The untyped “receive” method is replaced by declaration of any number of different input channels (though the single-mailbox principle is kept); the untyped “!” operator is replaced by different ones to avoid confusion.

Let me apologize at this point to our Java users. We try to keep both sides of Akka on par wherever possible but this is a feature which is fundamentally impossible to provide in Java. Please do not look at TypedActors now for compensation, they do not solve the same issues and their restrictions still apply. I am sincerely sorry.

With that out of the way, let us go in medias res:

This little snippet contains most of what typed channels are about. The actor which is being declared is no longer untyped, instead it lists requirements for those who wish to communicate with it: it will accept JDouble messages as input (which is just java.lang.Double for reasons discussed below), and it expects the sender to handle replies of type Double. This is encoded as the pair preceding the :+: operator, whose function is to build lists of these input/output type pairs, and like every good list it is terminated with a TNil. The first type argument to the Channels trait describes the requirements for this actor when talking to its parent; in this case that is not done, hence the empty type list.

The implementation then concludes with the definition of the behavior for the JDouble channel, which is to multiply it by itself and send it back to the sender. Note that the sender reference is passed into the behavior here, since it is typed such that sending anything but a Double will not be accepted.

Having statically checked actor interactions is a worthy goal in itself, as has been brought up many times on the mailing list or when talking with fellow hAkkers at conferences, but having it enables more! Let us look at a slightly more complex example:

This actor follows the same structure, but it accepts a constructor argument describing how to build another actor. That recipe is used to create a child actor which is accessed by the targetRef. Besides initiating the asynchronous creation of the child the createChild method also verifies that the StringToDouble actor is fit for being a good parent, i.e. that it can digest all messages the child may want to send to it. This example is a simple case since the target actor does not send to its parent (see the first TNil in target’s type signature).

The three lines before creating the child supply the contextual information needed to run the behavior below: when a String is received, it is parsed into a Double (java.lang.Double to be exact) and sent to the targetRef using the “ask” operator. This does the roughly the same thing as your good old “ask pattern”, returning a Future[Double] (since we have declared the target channel this way). This future is then mapped into a Future[String] and piped back to the sender.

One revolutionary feature is that in this syntax the message flows consistently from left to right, a feat that is made possible without performance cost by implicit value classes.

The interesting part about this second example is that now we have a higher-order actor: StringToDouble is comparable to a function which takes a function JDouble => Double and turns it into a function String => String. In the case at hand these functions are actually referentially transparent since the actors are stateless, making the connection to functional composition quite evident. In the general case this allows composition on a message protocol level, including aspects like timing of messages or altering delivery guarantees.

Viewing an actor as a function represented by its typed channel allows also another interpretation: the behavior to be invoked is determined not only by the recipient of the message but also the channel (since a channel list can contain multiple input–output pairs), which means that this implements dynamic multiple dispatch.

The approach shown above is merely the most static one (short of hard-wiring the child’s creation): there are other ways of achieving composition, for example by passing channel references between actors in messages. Since everything is statically type-checked, this opens up a whole new way of structuring large actor systems.

But enough of the theoretical talking. If you want to try out this collaboration in practice you could run an application like the following:

This shows the last element of the typed channels API: top-level actors are created by making use of the ChannelExt Akka extension. The rest uses Future-trampolining to read lines from the terminal and feed them to the actor, displaying the results as we go.

If you want to read more about the details of the API, please refer to the Akka documentation. The full code for the samples—with all imports—is available here. And it works with the published artifacts of Akka 2.2-M1, so give it a spin!

Caveat Emptor

The current state of the akka-channels-experimental module is “experimental”, there are some features missing and the API will also still be changed (e.g. adding textual method names and possibly changing existing ones). One of the missing features which is visible in the sample is that it is currently not possible to declare input channels of primitive types like Double—due to an oversight this channel would not ever receive messages.

But that is unfortunately not the end of it: as this document explains Scala’s reflection library is currently not thread-safe (and 2.10.1 will also not be). I carefully worked around the issue by ensuring that not two actors are starting up at the same time (in case you were wondering about the lazy val up there).

These issues will be fixed with time, but it is never too early to look at a new tool and give feedback!

Akka 2.1.1 Released!

pionic — Tue, 26 Feb 2013 13:57:33 +0100

Dear hAkkers,

We—the Akka committers—are pleased to be able to announce the availability of Akka 2.1.1. This is the first maintenance release of the 2.1 branch, which besides documentation updates and the fixes mentioned below also contains a back-port of most of the cluster-related changes in the recently released 2.2-M1.

The issues fixed are:

release thread pool used by Netty’s ExecutionHandler more timely
passive remote connections now work when using secure cookies
do not silently swallow exceptions in ForkJoinPool executor anymore
fix a bug in HashedWheelTimer which would schedule events incorrectly if delay is multiple of one wheel rotation (thanks Matthew Neeley)
fix reporting of error condition when IOManager binds to a port
make sure to set Thread.interrupt() flag when catching InterruptedException during tell()
Netty does not close the connection anymore upon encountering a non-serializable or too large message
PinnedDispatcher does not allow its thread to time out anymore, keeping the actor on the same thread (as opposed to only on its own thread)
actually disallow sending `null` messages (was always forbidden by documentation and only worked on some special ActorRefs)
StdoutLogger restores stdout-loglevel during system shutdown
do not leak the default dispatcher anymore
do not set SO_REUSEADDR for remoting on Windows (which implements this in a buggy way)
properly log exceptions without a message in JulEventLogger (thanks Sam Halliday)

The changes in the cluster module are:

cluster metric-aware adaptive load balancing routers
automatically load cluster extension when used, no need to declare in `akka.extensions` anymore
fire DeathWatch only upon member DOWN, not already at UNREACHABLE
allow configuration of minimum number of members before starting the cluster
cluster singleton pattern
detect failure even when no heartbeat was ever sent

In addition there have been a great number of documentation updates including a whole new section on terminology.

This release is backwards binary compatible with version 2.1.0, which means that the new JARs are a drop-in replacement for the old ones (but not the other way around). Always make sure to use at least the latest version required by any of your project’s dependencies.

Known Issues

ActorSystem.shutdown() does not terminate all threads if akka.remote.netty.use-dispatcher-for-io is set to true

Migrating from Older Releases

When migrating an existing project from Akka 2.0.x please have a look at our migration guide.

When migrating from the Akka 1.3.x series please follow first the migration guide towards version 2.0.5.

About the Cluster Support

IMPORTANT NOTICE

2.1.1compared to 2.1.0:

55 tickets closed
604 files changed, 8100 insertions(+), 2626 deletions(–)
… and a total of 8 committers!

Fixed Tickets: see the assembla milestone

Happy hAkking!

2.2-M1 Spotlight: Cluster Singleton Pattern

patriknw — Thu, 21 Feb 2013 14:10:59 +0100

This can be implemented by subscribing to LeaderChanged events, but there are several corner cases to consider. Therefore, this specific use case is made easily accessible by the Cluster Singleton Pattern in the contrib module.

The singleton actor is always running on the leader member, which is nothing more than the address currently sorted first in the member ring. This can change when adding or removing members. With the names given above the path of singleton actor can be constructed by subscribing to LeaderChanged cluster event and the actor reference is then looked up using actorFor:

2.2-M1 Spotlight: Startup when Cluster Size Reached

patriknw — Wed, 20 Feb 2013 14:08:00 +0100

With a configuration option you can define required number of members before the leader changes status of ‘Joining’ members to ‘Up’.

You can defer startup of actors until the cluster has at least the defined number of members.

Read more in the docs for Scala or in the docs for Java.

The New Cluster Metrics-Aware Adaptive Load-Balancing Routers and The Cluster Metrics API

patriknw — Tue, 19 Feb 2013 12:45:00 +0100

Finally with the release of Akka 2.2-M1, one of the most notable features in Akka Cluster becomes available - metrics-based adaptive cluster-aware routers. While cluster-aware routers have been available since Akka 2.1, the internal cluster metrics API and it’s initial consumer of this valuable data now become usable.

Building on my previous post, Cluster Health, State and Fault Tolerance, now the fun can begin.

Proactive Fault Tolerance Versus Reactive

Reactive Fault Tolerance

Anyone using Akka is familiar with the Supervisor Strategy. For some, it is the actor’s fault tolerance that led them to initially try Akka, and a primary reason for final adoption. But what does the Supervisor Strategy give us? Reactive fault tolerance within the actor hierarchy.

Preemptive Fault Tolerance

The release of Akka 2.1 saw the addition of the cluster’s AccrualFailureDetector, based on ‘The Phi Accrual Failure Detector’ by Hayashibara et al. This was aptly renamed PhiAccrualFailureDetector and refactored to Akka Remote in 2.2.

This FailureDetector strategy employs “a suspicion level on a continuous scale” to express current network conditions and node availability across a cluster. A configurable threshold value, -log10(1 - F(timeSinceLastHeartbeat), and historical heartbeat inter-arrival times all factor into the decision of whether a failure may have occurred. Not to be overlooked is that the failure detection threshold can be tuned to affect suspicion level, the potential for false positives, and time to detect actual crashes, should they occur. This is fault tolerance over distributed members of node rings with preemptive suspicion-level awareness.

Predictive Fault Tolerance

The AdaptiveLoadBalancingRouter and the Cluster Metrics API

Member nodes of the cluster collect system and JVM health metrics which are both gossipped around the node ring and made available to local subscribers. The primary consumers of this data are the load-balancing routers.

Cluster metrics are at present primarily for load-balancing of nodes. The Cluster Metrics API manages metrics sampling at a regular frequency, prepares highly variable data for further analysis by other entities, and gossips the latest cluster metrics data around the node ring. Each node in the cluster also publishes the current snapshot of this data to the event stream for the AdaptiveLoadBalancingRouter to assist in determining the need to redirect outbound traffic to least-loaded nodes.

The AdaptiveLoadBalancingRouter consumes the latest cluster metrics and pipes the data through particular MetricsSelectors to create a pool of weighted routees based on remaining capacity for message routing. Nodes exhibiting lower load have a higher probability of being routed to than nodes under higher load.

The abstract CapacityMetricsSelector calculates weights from capacity, and is the basis for the following metrics selector implementations:

HeapMetricsSelector calculates capacity of remaining JVM heap memory: (max - used) / max
CpuMetricsSelector calculates remaining CPU capacity based on the sum of User + Sys + Nice + Wait: 1 - utilization
SystemLoadAverageMetricsSelector calculates remaining system load capacity. If the average is nearing the number of cpus/cores it is indicative of possible bottlenecks: 1 - (load / processors)
MixMetricsSelector aggregates capacity of heap, cpu and load to calculate combined capacity
A custom akka.cluster.routing.MetricsSelector can also be created

The Cluster Metrics API

The default metrics sampling is delegated to a MetricsCollector using JMX MBeans. However you can optionally employ the Hyperic System Information Gatherer (SIGAR) MetricsCollector for a broader range of metrics with more precise values. The SIGAR collector is only utilized if both the SIGAR lib and its appropriate libsigar-{processor-type}-{operating-system}.dylib companion are detected on the classpath. The SIGAR collector is based on the JMX collector to still provide data points for JVM heap memory used, committed, and max (if available for the specific OS). SIGAR also returns the combined CPU metric, i.e. the sum of User + Sys + Nice + Wait. This metric describes the amount of time the CPU spent executing code during n-interval and how much more it could theoretically. Note that 99% CPU utilization can be optimal or indicative of failure.

To add SIGAR, download and add the SIGAR jar: java -classpath sigar-1.6.5.132.jar. Then add the appropriate native library using -Djava.libarary.path.

Smoothing of the data for each monitored process is delegated to an exponentially weighted moving average (EWMA) function. Such an approach captures short-term movements in volatility for a conditional volatility forecasting model. The decay factor provides a statistical streaming data model that is exponentially biased towards newer entries. Unlike a standard moving average model, EWMA only needs the most recent forecast value to be kept, which allows this to be both lighter weight and more efficient.

How To Leverage The Functionality

Subscribing to Metrics Events

Subscribing to cluster metrics events directly allows easy access to the data in order to implement custom event-based logic.

Custom Metrics Collector

Should you need to roll your own metrics collector versus using the default akka.cluster.JmxMetricsCollector or akka.cluster.SigarMetricsCollector, start by looking at these two. Then simply configure your class FQCN.

Parting Thoughts

For application engineers the AdaptiveLoadBalancingRouter and the Cluster Metrics API cover two primary concerns: predictive, proactive fault tolerance and increased performance.

For cloud engineers, stop and consider the potential for capacity planning that could be derived from this statistical data. A machine learning kernel for cloud. Now consider linking this real-time data to EC2, VMware cloud, and other APIs.

Wouldn’t it be nice to automate spinning up new nodes when bottlenecks arise in your Data Centers when VMs near their capacity? Watching it all unfold on the Typesafe Console.

Helena Edelson
Twitter: @helenaedelson

2.2-M1 Spotlight: Adaptive Load Balancing Based on Cluster Metrics

patriknw — Mon, 18 Feb 2013 13:43:11 +0100

The member nodes of the cluster collects system health metrics, such as heap memory and CPU usage. The metrics is spread to other nodes and consumed by a new type of router that performs load balancing of messages to cluster nodes based on the cluster metrics data.

The router is defined in the same way as other routers:

and in this case it’s configured as follows:

Many thanks to Helena Edelson for her great contributions to the metrics design and implementation. In an upcoming blog post she will describe this topic in more detail.

Read more in the docs for Scala or in the docs for docs for Java.

Akka 2.2-M1 Released

patriknw — Sat, 16 Feb 2013 10:14:00 +0100

Dear hAkkers,

We—the Akka committers—are pleased to be able to announce the availability of Akka 2.2-M1 (PRE-RELEASE MILESTONE ONE).

It is a preview on what is going to become Akka 2.2, so please give us feedback!

The most notable features in this release are:

Cluster support improvements

Metrics-based adaptive cluster-aware routers
DeathWatch fires upon node DOWN (previously on UNREACHABLE)
Configurable minimum number of nodes before cluster starts
ClusterSingleton actor pattern
Hardening of the code by adding stress tests
various fixes, optimizations

Rewritten Remoting layer to be actor-based and thus lock-free

Smaller SPI for implementing other transport drivers
Support for multiple transports within one ActorSystem
No more blocking message sends due to broken connections or during connection establishment
Currently still using Netty, but will switch to Akka IO soon
Completely new IO layer written in collaboration with the spray.io team
Actors all the way down—until you hit NIO
No external dependencies
Extreme scalability
No compromises on the API (i.e. fully exposing the different capabilities of e.g. TCP and UDP)

Typed Channels for Scala

Type-safety for message sends which is on par with type checking of method invocations
Based on Scala 2.10 macros; unfortunately not possible to do in Java
Very early preview, expect API changes within the next weeks

The akka.actor.IO support has been deprecated and is being replaced with the new IO layer in package akka.io. The question of why this has been done as opposed to just wrapping Netty behind actors has been answered recently on the spray mailing list (https://groups.google.com/d/topic/spray-user/7l1RbqxRuic/discussion), in a nutshell the reason for joining forces with the spray.io team was that Netty—while being a great network framework—replicates a lot of what Akka actors solve (thread management, the pipeline stages as light-weight special actors) which makes it cumbersome to keep resource usage of an Akka application as low as it could be. Another aspect is that Netty as a Java library heavily uses mutability, comes with its own Futures, etc. So the decision was made to create a very lean, specialized actor-based layer on top of NIO which from now on will form the foundation on which Akka and Spray base their network support.

When migrating an existing project from Akka 2.1.0 please have a look at our migration guide.

Take it for a spin!

Documentation: http://doc.akka.io/docs/akka/2.2-M1/

Akka is released under the Apache V2 license.

2.2-M1 compared to Akka 2.1.0:

121 tickets closed (2.2-M1 2.1.x)

744 files changed, 41156 insertions(+), 16523 deletions(-)

Credits:

commits added removed
   62    6476    3442 Roland
   58    8925    4476 Patrik Nordwall
   57   10824    6041 Endre Sándor Varga
   43    3040    4311 Viktor Klang
   17     819     440 Björn Antonsson
   11    1324     322 Johannes Rudolph
   10    1109     527 Mathias
    6    1108     289 Rich Dougherty
    5     311     139 RickLatrine
    2      83      20 Kaspar Fischer (hbf)
    2     127      66 Christophe Pache
    2     163      66 Raymond Roestenburg
    1      10       3 Michael Pilquist
    1     127     127 drewhk
    1     548      77 Helena Edelson
    1      38      24 Matthew Neeley
    1       8      10 Peter Vlugter
    1       3       3 Thomas Lockney
    1      36      35 Derek Mahar

Happy hAkking!

Tuning Dispatchers in Akka Applications

pionic — Thu, 17 Jan 2013 11:42:23 +0100

One of the biggest questions I encounter among users of Akka is how to use dispatchers to create failure zones and prevent failure in one part of the application from affecting another. This is sometimes called the Bulkhead Pattern. And once I create the failure zones, how do I size the thread pools so that we get the best performance from the least amount of system resources used? The truth is, every application is different, and you must build your system and measure its performance under load to tune it effectively. But here are some tips and tricks.

Most people building an Akka application start out with a single ActorSystem, using the default dispatcher settings with a minumum number of threads of 8 and a maximum number of threads of 64. As the application grows, they notice that futures time out more frequently since futures in Akka actors often use the actor’s dispatcher as their ExecutionContext implicitly.

Because of the limited resources of one thread pool for all actors and futures in their application, resource starvation is occurring. When that happens, I recommend that they identify actors using futures and consider where you can use a separate dispatcher or ExecutionContext for those futures so that they do not impact actors with their thread usage. We want to limit the impact of the work of those futures on the actors handling messages in their mailbox.

As a temporary workaround, I have noticed some people try to use a PinnedDispatcher for each actor so that starvation is less likely. Actors created with PinnedDispatcher will receive their own dedicated thread that lives up until the keep-alive-time configuration parameter of the ThreadPoolExecutor (default of 60 seconds) is not exceeded. However, this is really not a viable solution for production except for very specific use cases, such as service-oriented actors handling a lot of load. For most other tasks, you WANT to share resources among actors with similar roles and risk profiles so that you aren’t using large amounts of resources dedicated to each actor. In addition, starting and restarting threads takes time, and each has a default size of 512KB. You will use up your memory very quickly in a system that relies primarily on actors created with PinnedDispatcher.

The key to separating actors into failure zones is to identify their risk profile. Is a task particularly dangerous, such as network IO? Is it a task that requires blocking, such as database access? In those cases, you want to isolate those actors and their threads from those doing work that is less dangerous. If something happens to a thread that results in it completely dying and not being available from the pool, isolation is your only protection so that unrelated actors aren’t affected by the diminishment of resources.

You also may want to identify areas of heavy computation through profiling, and break those tasks out using tools such as Routers (no shared mailboxes and thus no work-stealing) and BalancingDispatcher (one mailbox for all “routees”, and therefore work-stealing in nature). For those tasks that you assign to Routers, you might also want them to operate on their own dispatcher so that the intense computation tasks do not starve other actors waiting for a thread to perform their work.

As a side note, I tell people in my Effective Actors presentation never to parallelize with Routers before you have measured the hot spots in your application. Too often, developers (including myself) make assumptions about where we expect performance to be slow and try to mitigate those problems up front. If you suspect you will need to use a Router somewhere, Akka Team Lead Roland Kuhn recommends that you pass around actor references from supervisors to compute-intensive children so that the ‘context.parent’ of their children are free to become a router later on with minimal impact. But do not create Routers before you have proven at runtime under load that you absolutely need to.

Now the question becomes how to size your dispatchers, and this is where the Typesafe Console can be very handy. In systems where you have several or many dispatchers, keep in mind that the number of threads that can be run at any time on a box is a function of how many cores it has available. In the case of Intel boxes where hyperthreading is available, you could think in terms of double the number of cores if you know that your application is less CPU-bound. I recommend sizing your thread pools close to the number of cores on the box where you plan to deploy your system, and then run your system under a reasonable load and profile with the Typesafe Console. You can then externally configure the thread pool sizes and check the impact at runtime.

When using the Typesafe Console, watch the Dispatchers view to see if the latency of message handling is within acceptable tolerances of your non-functional requirements, and if not, try adjusting the number of threads required upward. Remember, you’re setting the minimum number of threads, the maximum number of threads and the “parallelism-factor”. This is the ceiling of the number of cores on your box multiplied by that factor is calculated to determine the thread pool size, bounded by the max and min settings you give.

The Typesafe Console also shows you something else that is very important to watch - the size of each actor’s mailbox. If you see an actor whose mailbox is perpetually increasing in size, you need to retune the threads for its dispatcher or parallelize its task by making it a Router so that it has the resources it needs to keep up with the demands placed on it by the system. The receipt of messages into an actor’s mailbox can be bursty in nature, but you shouldn’t have actors with mailboxes that aren’t handling the traffic coming to them fast enough to keep up with the load.

Once you have an idea of the number of threads you need to handle burstiness in your application (if any), sit down with your team and figure out the minimum and maximum bounds of each thread pool. Don’t be afraid to add a few extra threads to the max to account for possible thread death in production, but don’t go overboard.

Also, pay close attention to your “throughput” setting on your dispatcher. This defines thread distribution “fairness” in your dispatcher - telling the actors how many messages to handle in their mailboxes before relinquishing the thread so that other actors do not starve. However, a context switch in CPU caches is likely each time actors are assigned threads, and warmed caches are one of your biggest friends for high performance. It may behoove you to be less fair so that you can handle quite a few messages consecutively before releasing it.

There are a few edge cases. If you have a case where the number of threads is equal to the number of actors using the dispatcher, set the number extremely high, like 1000. If your actors perform tasks that will take some time to complete and you need fairness to avoid starvation of other actors sharing the pool, set the throughput to 1. For general usage, start with the default value of 5 and tune this value for each dispatcher so that you get reasonable performance characteristics without the risk of making actors wait too long to handle messages in their mailboxes.

For more information, see this page of the documentation on finding bottlenecks in Akka. Hope that helps you with tuning your Akka actors, and happy hakking!