Lately I’ve been thinking a lot about the design of Actor systems and actor hierarchies, specifically as implemented by Akka. In a previous series of blog posts, I used Akka to build a telnet-gateway that went through to an Akka Actor System to support a MUD (Multi-User Dungeon/Dimension). Having learned a thing or two about high performance systems since then, I have changed my perspective on how some of these things should be designed.
User Space Modeling vs. Task Modeling
Let’s assume for a minute that we are building an MMO server that supports commerce. Players can walk into a shop and perform a number of tasks. Within the shop, players can sell goods from their inventory, buy goods from the shop keeper, or use the shop keeper as a means to engage in secure trade between players (assume for the sake of this example that players can’t exchange goods unless they’re standing in a shop – it helps illustrate my point later).
When modeling an Actor, especially in my MUD, it can be very easy to think of Actors with a clear correlation to objects within your virtual world. For example, we might create an Actor for the shop keeper and we might send him messages like:
shopKeep = context.ActorOf .... (find the shopkeeper standing in the same room as the player) shopKeep ! SellStuff(player, 21, ItemPrototypes.JunkSword) shopKeep ! SellStuff(player, 57, ItemPrototypes.Seashell)
In this scenario, every shop keeper in the game is a unique instance of the ShopKeeper actor. Each one can maintain their own state (their inventory or wares). Sounds good, right? My inner MUDder likes this because it maps pretty much to what the virtual world looks like on my game client. So let’s scale this out a bit. Let’s say the game has 5,000 shop keepers total. 5,000 instances of a shop keeper, each consuming roughly 300 bytes (Akka actor overhead is 300 bytes) plus whatever it takes for their state, that’s an overhead of roughly 1.4MB to manage the overhead of my shop keepers. Meh, 1.MB is nothing in relative terms, so that’s no big deal. So far so good, right? Maybe, but my new perspective says no. This particular type of scale-out is only useful if there is an even distribution of players compared to shop keepers. This is hugely important to remember. What if my game currently has 10,000 concurrent users but instead of being spread out evenly across my virtual universe, someone posted a “FREE BEER AT SHOPKEEPER2921’s HOUSE! PAR-TAY!” sign at the Orc’s Crossing. Now, I’ve got 7,500 players crowded into Shopkeeper2921‘s shop. Graphical client problems aside, this means this one actor is now trying to handle messages from 7,500 players who are buying, selling, and trading with each other. In short, that actor is going to get backed up, the game will appear slow to 7,500 of it’s players, and the proverbial shit has hit the fan.
Creating an actor for every shop keeper, actors for the shop keeper’s shop, actors for each person’s weapon, etc is what I am going to call user space modeling. I have no idea if this is real term, but it is now. It means that the architect of the actor system has decided to map user perceived actors to real actors. This is what I was doing with my MUD. What I should be doing for better performance and certainly better (cloud-sized) scalability is task-based modeling.
Let’s take a look at this problem from a slightly different point of view, task-based modeling. Instead of creating actors to back virtual constructs like shop keepers, swords, and shops, we create actors that perform tasks. If we take a look at our game client, we see that we might be sending packet-messages like “sell stuff”, “buy stuff”, “trade offer”, “trade counter”, and “trade confirm”. Instead of sending these messages to whichever shop keeper happens to be standing near the player (user space modeling), we’ll simply send the messages to whichever Commerce Actor happens to be available based on Akka routing.
Let’s say now that we create an actor CommerceActor who accepts messages in case class form Sell, Buy, TradeOffer, TradeCounter, TradeConfirm. Instead of sending those messages to a nearby virtual shopkeeper, let’s send them to the “least busy” commerce actor among the N instances of CommerceActor that we’ve created with the Router mix-in trait.
Now our commerce logic might look something like this (did a little DSL-style stuff with the items just for giggles):
val shopKeeper = system.actorOf(Props[Commerce].withRouter( RoundRobinRouter(nrOfInstances = 10))) shopKeeper ! Buy(shopId, 21 VorpalSwordOfDooom) shopKeeper ! Trade(playerOne, playerTwo, 15 Gold)
Here the code is using a “round robin” router, which means each instance will be given a message in round robin fashion. You can also use a “least full mailbox” router, which essentially load balances it so that the shop keeper currently having the smallest backlog of transactions to process will be given the message. This feels right, and will scale. Even better, the routing configuration for a particular type of actor can be configured in the application.conf file. This means that you can dynamically change the number of instances of certain types of actors. This way, if players are suddenly doing a crapload of commerce and bogging your system down, you can beef up the number of instances of your commerce actor.
In task-based modeling for an actor system, actors are as the Akka documentation recommends as a best practice, encapsulations for behavior and state that perform discrete tasks and either complete the task or delegate sub-tasks to other actors. The supervisor/actor hierarchy is created based on the work that needs to be done, which doesn’t necessarily have a 1:1 mapping with the virtual world in which players inhabit.