Shutdown Patterns in Akka 2

I’ve seen a question pop up a number of times on the Akka Mailing List that looks something like: "How do you tell Akka to shut down the ActorSystem when everything’s finished?" It turns out that there’s no magical flag for this, no configuration setting, no special callback you can register for, and neither will the illustrious shutdown fairy grace your application with her glorious presence at that perfect moment. She’s just plain mean.

In this post, we’ll discuss why this is the case and provide you with a simple option for shutting down “at the right time”, as well as a not-so-simple-option for doing the exact same thing.

What the Heck Does “Finished” Mean?

Before we talk about how to shut down when the app is finished, we have to first describe what it means to say the app is “finished” at all. The most natural answer to this question appears to be, "When all the Mailboxes are empty." Natural, yes; correct, no. :)

The obvious reason for this lack of correctness is due to the possibility that there could be a person out there who hasn’t sent their message yet. The app isn’t finished because it hasn’t seen all of it’s work.

But there’s something more subtle here. Actor Mailboxes can be empty while they are still chugging away doing “stuff”. Have a look…

The probability of this occuring while you have hundreds of Actors chugging away is pretty low, but that probability increases as the app winds down. Even if the odds were 0.004%, it wouldn’t matter - 0.004% is still a pretty questionable platform on which to design an algorithm.

"Ah, but Akka can know if the Actor is working, so don’t shut down while there are non-empty Mailboxes and Actors are working!” you may say. Sure, but what if the Actor spawns work off to a Future? Or it has a request out to another machine waiting on a response? You wouldn’t want to shut down in these cases. On the flip-side, if there is a simple busy-wait ticker somewhere that keeps the app “busy” even though it’s not, would you want to stay running forever? Probably not.

The bottom line is that an app is “finished” when the app says it’s finished, not when Akka guesses that it’s finished. What we’re going to cover are mechanisms by which you can tell Akka that you’re done, but it’s going to depend on what it is that your app is doing.

The DABOWoTeD Pattern

A common idiom is the Do A Bunch Of Work Then Die pattern. It’s pronounced just like you’d think… It’s this type of program that people seem most concerned with, when it comes to shutting down after completion. After all, apps that are supposed to run 24/7 generally aren’t concerned with this, since they have to handle hardware failures more often than shutdowns.

The idea with DABOWoTeD is that you’ve created a bunch of Actors to do some work, and when they’re “done” that work, it’s time to shut the system down. This is also possible with Futures, and we’ll cover those first.

Futures

Futures are easy because there’s a really simple way to know when everything’s done - use a sequence. Essentially, you’re just doing a big fork-join, and when everything joins up, shut the system down. It looks something like this:

It just doesn’t get much simpler than that, so if you’ve got a fork-join execution, just be awesome, use that pattern and collect your profit.

Actors

Actors are more interesting here because they are generally employed when the application is more complex. It’s not so clear what “finished” means when algorithms with Actors are involved because of that higher level of complexity.

We’re going to make a very reasonable assumption here: at least one Actor knows when things are “finished”. With this simple assumption, we have enough information to create our shutdown hook, which is going to be related to the Terminator that I described a short while ago.

The Reaper

The Reaper is an Actor that is in charge of collecting dead souls. He has been told to watch over a number of Actors, waiting for them die. When he sees the last one give up its ghost he performs some action, and that action will be, in this case, to shut down the ActorSystem. Never fear the reaper.

In the above diagram, we see that the Reaper is just another Actor, and he has been told to watch the green nodes, B, C, D and H. No single one of these key Actors denotes the completion of the application, but when all of them are finished, then we know that the application is done. They signal their completion by dying, and when the Reaper collects all those souls, he’s going to shut down the system.

Reaper Code

The Reaper itself is wonderfully simple. All it has to do is accept a message that tells it to watch an ActorRef, and then make a call to a method when everything it’s watching has kicked the bucket.

A “Production” Reaper

Given the simplicity of the Reaper, we can now create an even simpler Production Reaper, that shuts down the ActorSystem when everything’s 6 feet in the ground.

Testing the Reaper

It’s always nice to have tests. Here we use Akka’s TestKit along with its TestProbe and ImplicitSender to hook up everything we need to test our Reaper.

Application with PoisonPill

The Reaper is very good at being flexible when the goal is to drain a set of (one or more) Mailboxes. Let’s say you have a message processing system whereby you send a bunch of messages (or work) to some Actors, which may send more messages (or work) to other Actors. What you want to do is shut down when a key set of Mailboxes have been drained. For this, we use Akka’s PoisonPill message. The PoisonPill goes into the Actor’s Mailbox and will get processed in due course, which will kill the Actor. The key point is that all of the messages that exist ahead of the PoisonPill will be processed. This is different than just shutting down an Actor using system.stop.

We can see how PoisonPill messages can be propagated through an Actor application in the following:

Once the last PoisonPill message has been processed, the Reaper will kick in and do whatever it is you want done, such as shut down the ActorSystem.

Conclusion

Akka doesn’t know when your’e done doing what it is you’re doing since only you know what it is that you’re doing. Here we’ve covered a couple of mechanisms you can employ for knowing when to shut down your system. There are, of course, many others that you can probably think of, including something as simple as an external message that tells a specific Actor to ShutDown.

With the Reaper pattern we’ve made use of the Actor paradigm to put a level of abstraction into the system, which uses the basic functionality of Actors (namely that of DeathWatch) to add “system level” behaviour to our applications. The Untyped Actor, along with the basic tool of DeathWatch, provide the kind of flexibility that you can turn to in order to add some cool functionality to your apps.

About the Author

Derek Wyatt is a Software Developer and Architect at Primal, helping to create Interest Networks using graphs and other cool stuff. He is also the author of an upcoming book on Akka to be published by Artima Publishing. You should go get a copy when it comes out… really. It’ll have pictures! Nothing dirty, though…

Recent comments

Blog comments powered by Disqus

3 Notes

  1. caffeinated-tesla reblogged this from hakkers
  2. Derek Wyatt submitted this to hakkers