Kotan Code 枯淡コード

In search of simple, elegant code

Menu Close

Tag: actor systems

Building an MMO Server in Akka : Using MongoDB as a Backing Store

The story so far: I’ve created an Akka application that is using non-blocking Akka IO over sockets to communicate via Google Protocol Buffers with iOS (or potentially others…) clients. In the last blog post, I created some Actors that allow objects that have a physical presence in space to respond to messages, namely radar pings. This allows objects to “appear” to players. Their client will get radar pong messages, which will allow the client GUI to create icons for the items in the radar pong like planets, star ports, ships (owned players or NPCs), harvestable asteroids, and so on.

That’s all well and good, but in the last blog post, I had to hard code things like the name, description, and location of the objects in the universe. The first thing I needed to do was to refactor the PhysicalObject trait … in the last iteration it had “magic knowledge” of the types of physical objects that might be using that trait. So here’s my new physical object:

package com.kotancode.ulysses.space

import com.kotancode.ulysses.space.Vector3D._
import akka.actor._

case class RadarPing(source: ActorRef, origin: Vector3D, scanDistance: Int)
case class RadarPong(source: ActorRef, origin: Vector3D, objectType: Int, name:String, description: String)

object RadarObjectTypes {
	val Planet = 1
	val Station = 2
	val Port = 3
	val Ship = 4

	val Unknown = 99
}

trait PhysicalObject extends Actor with ActorLogging {
	def replyPong(p:RadarPing): Option[RadarPong]

	def receiveSpatial: Receive = {
		case p:RadarPing => {
			replyPong(p).map { pongReply => p.source ! pongReply }
		}
	}
}

There’s a couple of new things happening here. First, this trait no longer has a member variable called location. Second, in the last iteration, this trait constructed the pong reply itself. This would have been really problematic because I would have ended up with one method with a pile of “switch” cases to determine how it should reply in different circumstances… and that would’ve been smelly. Now, it is up to the actor using this trait to define a method that creates pong replies. Also, note that the abstract method that the concrete actors will implement returns an Option so I can just run a map over that – if the method returns a None then the above code won’t deliver the pong message.

Now that I’ve cleaned up my physical objects (to make them easier to be instantiated dynamically based on data from a backing store like Mongo), I can create a Persistable trait:

package com.kotancode.ulysses.state

import akka.actor._
import com.mongodb.casbah.Imports._
import org.bson.types.ObjectId

case class LoadState(id:String)
case object PersistState

object BackingStoreKeys {
	val Name = "name"
	val Location = "location"
	val Description = "description"
	val ID = "_id"
}

trait Persistable extends Actor {
	var backingObject: MongoDBObject = null

	def receivePersistable: Receive = {
		case LoadState(id) => {
			loadFromMongo(id)
		}
		case PersistState => {
		  // Nothing here yet...
		}
	}

	def persistenceCollectionName:String

	def loadFromMongo(id:String) = {
		val mongoClient = MongoClient()
		val ulysses = mongoClient("ulysses")
		val collection = ulysses(persistenceCollectionName)
		val objectId = new ObjectId(id)
		backingObject = collection.findOne(MongoDBObject(BackingStoreKeys.ID -> objectId)).getOrElse { throw new IllegalArgumentException(s"id ${id} not found in ${collection}")}
		println(s"backing object ${backingObject}")
	}
}

Here I’m using the Casbah library for Scala to talk to MongoDB. What’s worth mentioning here is that the only way a persistable actor should load its state is after it has received a message to do so. This keeps the data-loading activity in the thread-safe message pattern already enforced by Akka, so I don’t have to worry about race conditions where I have to delay some activity until after my reference data has been loaded (this is a common problem I see regularly in message-based applications).

Let’s see what our new, cleaner Planet actor looks like now:

package com.kotancode.ulysses.space

import com.kotancode.ulysses.space.Vector3D._
import com.kotancode.ulysses.state.Persistable
import com.kotancode.ulysses.state.BackingStoreKeys
import com.mongodb.casbah.Imports._

import akka.actor._

object Planet {
	val BackingCollectionName = "planets"
}

class Planet extends Actor with PhysicalObject with Persistable {

	override def persistenceCollectionName = Planet.BackingCollectionName

	override def replyPong(ping:RadarPing): Option[RadarPong] = {
		val currentLocation = Vector3D.fromBackingObject(backingObject.as[BasicDBObject](BackingStoreKeys.Location))
		if ( (currentLocation distanceFrom ping.origin) < ping.scanDistance) {
			Some(RadarPong(self,
				currentLocation,
				RadarObjectTypes.Planet,
				backingObject.as[String](BackingStoreKeys.Name),
				backingObject.as[String](BackingStoreKeys.Description)
			))
		}
		else {
			None
		}
	}

	def receivePlanet: Receive = {
		case _ => log.debug("Received some message for a planet.")
	}

	def receive = {
		receivePersistable orElse receiveSpatial orElse receivePlanet
	}
}

In my Mongo database, the location field contains a nested object with the properties x, y, and z. The Vector3D class has a utility method on it (I will refactor this into a “pimp” or “augment” pattern later) that converts a Casbah BasicDBObject into a 3D vector by extracting these properties.

Finally, remember that Universe class? The one that hard-coded the instantiation of the “Utopia 1” planet? Here’s what it looks like now:

package com.kotancode.ulysses.space
import com.kotancode.ulysses.state.LoadState
import com.kotancode.ulysses.state.Repository

import akka.actor._

class Universe extends Actor with ActorLogging {

	def loadPlanets = {
		for (id <- Repository.allIds(Planet.BackingCollectionName)) {
			context.actorOf(Props(new Planet()), name=id) ! LoadState(id)
		}
	}
	override def preStart() = {
		loadPlanets
	}

	def receive = {
		case p:RadarPing => context.children foreach (_.forward(p))
		case _ => log.debug("Received a universe message")
	}
}

Here I’m just making use of a simple object I created called Repository that issues a Casbah query to Mongo – it’s an empty query on a collection so it returns all the objects and then converts the Object IDs into strings and yields them so they can be used in an enumerator, as shown in the above code.

Now, instead of having a hard coded planet, I can create multiple planets in my database and this code will instantiate an actor for each of those planets, each planet will be backed by its own unique Mongo DB document. Here’s some trace log output that shows a pong reply from each of the 3 planets I created in my MongoDB:

[DEBUG] [03/24/2013 08:31:12.998] [UlyssesAgenda-akka.actor.default-dispatcher-1] [akka://UlyssesAgenda/user/ServerCore/Clients/96e1e42b-5cda-4faa-aa89-019c2bde9794] I received a radar pong RadarPong(Actor[akka://UlyssesAgenda/user/ServerCore/Universe/514de823b485acb28e02bb91],(5.0,5.0,5.0),1,Utopia 1,This is a utopian planet, chock full of win.)

[DEBUG] [03/24/2013 08:31:12.999] [UlyssesAgenda-akka.actor.default-dispatcher-1] [akka://UlyssesAgenda/user/ServerCore/Clients/96e1e42b-5cda-4faa-aa89-019c2bde9794] I received a radar pong RadarPong(Actor[akka://UlyssesAgenda/user/ServerCore/Universe/514e194004538147414b4675],(6.0,6.0,6.0),1,Utopia 2,This is a little less utopian than the other one.)

[DEBUG] [03/24/2013 08:31:12.999] [UlyssesAgenda-akka.actor.default-dispatcher-1] [akka://UlyssesAgenda/user/ServerCore/Clients/96e1e42b-5cda-4faa-aa89-019c2bde9794] I received a radar pong RadarPong(Actor[akka://UlyssesAgenda/user/ServerCore/Universe/514e19b304538147414b4676],(7.0,7.0,7.0),1,Doom,Every game needs a planet doom.)

Now that I’ve got the core framework for persistence up and running, I can do some more useful things like giving players persistence so I can keep their names and locations separate. Once I have that, I can move on to creating the most basic of the actual gameplay functions.

Building an MMO Server in Akka: Objects in Space

I originally wanted to call this post Piiiiiiiggssss innnn Spaaaaace in homage to the Muppet Show, but I figured it would make it harder for people to actually find this content, so, we’re stuck with the boring title for now. In my previous blog post, I talked about enabling multiplayer and rigging up the essential infrastructure to support multiple players within an Akka-backed MMO universe.

So, we’ve got an iPad client (haven’t shown you this yet, mostly because it has no graphics) that connects to our Akka server. This iPad client sends a SignIn protobuf message and then, to keep the test going, sends itself a DirectMessage protobuf message. That’s all well and good if we’re building an MMO chat game, but this is a universe and, sooner or later, we’re going to have to put stuff in our universe.

So let’s take a look at some of the things that we think our stuff in this universe is going to need to do. Firstly, it’s going to need to exist at some point in space, e.g. a coordinate. All of the stuff that exists in space is only discovered by the player by virtue of appearing on their radar scan. I haven’t designed all this out yet, but I know that players who have invested more time and money in their ship’s navigational and scanning equipment will be able to see more stuff, see further out into space, and glean more information from what happens to be out there. Whether that stuff is visible may also have something to do with the thing itself. For example, I might have a cloaking device (though I’d probably have to call it something else, for fear of copyright infringement) which might make it so that cloaked stuff doesn’t appear on radar.

In other words, this is a very quantum style universe – you don’t actually know what’s in it until you inspect it, and the act of inspecting it can alter the results. I might, in the future, decide to code a sneaky NPC that waits to be radar scanned by nearby players and then automatically targets the source of that scan.

So what I’ve decided to do is create a sub-tree of my actor system hierarchy for physical things that exist in space, which includes planets, space ports, ships, star ports, and other stuff that might be harvestable for resources like asteroids and moons. For starters, these things can all be sent a RadarPing message which ultimately originates from a player. These things can then respond to that ping with a RadarPong. The iPad client can then collect radar pongs and use that to dynamically populate a star map of the “nearby universe”. Cool, huh?

So let’s start with the universe:

package com.kotancode.ulysses.space

import akka.actor._

class Universe extends Actor with ActorLogging {
	// create some child objects
	val testPlanet = context.actorOf(Props(new Planet()), name="Utopia1")

	def receive = {
		case p:RadarPing => context.children foreach (_.forward(p))
		case _ => log.debug("Received a universe message")
	}
}

So what we’re doing here is if I send the universeRadarPing message, that message then gets sent to all physical objects within that universe. If the object is out of range of the ping or is cloaked (or something else), then it won’t respond. Otherwise, it will comply and send back a RadarPong message. I know off the bat that I’m going to have many different types of actors that act as physical things in my universe, so this seems like the right time to bust out a trait that encapsulates this behavior:

package com.kotancode.ulysses.space

import com.kotancode.ulysses.space.Vector3D._
import akka.actor._

case class RadarPing(source: ActorRef, origin: Vector3D, scanDistance: Int)
case class RadarPong(source: ActorRef, origin: Vector3D, objectType: Int, name:String, description: String)

object RadarObjectTypes {
	val Planet = 1
	val Station = 2
	val Port = 3
	val Ship = 4

	val Unknown = 99
}

trait PhysicalObject extends Actor with ActorLogging {
	private var location = (6.5, 5.0, 7.2)
	var name = "Game Object"
	var description = "This is a game object"
	var objectType = RadarObjectTypes.Unknown

	def receiveSpatial: Receive = {
		case p:RadarPing => {
			log.debug(s"I received radar ping ${p}, it originated ${p.origin distanceFrom location} units from my location at ${location}.")
			if ( (p.origin distanceFrom location) < p.scanDistance) {
				p.source ! RadarPong(self, location, objectType, name, description)
			}
		}
	}
}

Since my game doesn’t currently have persistence, I’m making up hardcoded values for the planet but, in the future, the planet would ideally reach into some database, identify itself, and then pluck out a map of all of its data, such as its spatial coordinates. The Vector3D class I created back when I was writing a MUD in an earlier version of Akka, and it basically allows me to take a 3-tuple (e.g. (5.0, 1.5, 2.5)) and implicitly turn that into something that supports basic cartesian math. You can see this in the line of code that contains p.origin distanceFrom location, which utilizes the distanceFrom method that I added to my Vector3D class.

Now how about we create a planet that includes the PhysicalObject trait?

package com.kotancode.ulysses.space

import com.kotancode.ulysses.space.Vector3D._
import akka.actor._

class Planet extends Actor with PhysicalObject {

	objectType = RadarObjectTypes.Planet
	name = "Utopia 1"
	description = "This is a utopian planet, chock full of epic win."

	def receivePlanet: Receive = {
		case _ => log.debug("Received some message for a planet.")
	}

	def receive = {
		receiveSpatial orElse receivePlanet
	}
}

This is a pretty simple class and you can see that I’m doing some more hard-coding of values to serve as placeholders until I deal with the persistence deal (remember the YAGNI portion of the Agile Manifesto?). What I really like here is the receiveSpatial orElse receivePlanet partial function chaining. This lets me grab the receive function from the PhysicalObject trait and chain on the receive function from the Planet actor. I don’t have any planet-specific messages but you can imagine I might have messages like Land and TakeOff for when players bring their ships into orbit, etc.

Almost done. We’ve got a planet, which includes the physical object trait (and with it the associated message handling and state … I freaking love traits), and we’ve got a universe that contains the planet. Now we just need to send out radar pings. Rather than make the iPad client manually send radar ping packets, we can use an Akka scheduler on the server side to force the player’s proxy (GameClientProxy actor) to issue the RadarPing to the Universe object periodically.

Here’s the new code for my GameClientProxy class’ preStart() actor method:

lazy val universe = context.actorFor("../../Universe")

override def preStart() = {
	log.debug(s"GameClientProxy pre-start, socket uuid: ${socketHandle.uuid}")
	implicit val timeout = Timeout(1 second)
	import context.dispatcher
	SocketServer.system.scheduler.schedule(
		10 seconds,
		10 seconds,
		universe,
		RadarPing(self,  (5.0, 5.0, 5.0), 10)
	)
}

That’s pretty much it. We know from the code above that we’re going to be sending pongs directly back to the player proxy actor, so we need to respond to that. In the near future, this will involve creating a protobuf object and writing it back out to the client over the socket connection, but for now we’ll just log the fact that we got the pong back:

def receive = {
	case m:DirectMessage => {
		log.debug("I received a direct message.")
		writeRawMessage(m, MessageTypes.DirectMessage)
	}
	case pong:RadarPong => {
		log.debug(s"I received a radar pong ${pong}")
		// writeRawMessage(new raw pong protobuf ...)
	}
}

So that’s it… In just a few quick steps with a few small, lightweight actors, I’ve given my universe the ability to expose an infinitely expandable list of objects to players by virtue of modeling a message-based radar system. I’m nowhere near the point where I can add combat yet, but I’m adding building blocks slowly, which is always fun to do. I think next I’ll work on the “universe map” display for the iPad and possibly figure out a persistence method so I can load planets from my database.

Building an MMO Server in Akka : First Steps Toward Multiplayer

As you can imagine, building a Massively Multiplayer Online Game (MMOG, or MMORPG for “roleplaying” types, also you may have seen MMORTS for online real-time strategy games) is no small task. In fact, it’s pretty daunting. However, the technology we have now, including hardware, software, and virtualization, makes it possible to scale these games to incredible sizes.

To put it in perspective, EverQuest, what I consider the first truly huge, commercially successful MMO (yes, Ultima Online was first, but EQ quickly rose to eclipse UO’s volume), during one of the periods where I was playing, used to have a per-server capacity of around 1700-2000 players. Once our buddies started noticing that the number of players online in a particular server went up close to 2000, many of us started running like hell for the nearest safe spot, city, or useful campground.

Today, while still sharding out its content across a ridiculous number of servers, games like World of Warcraft not only have vastly more content and interaction than EQ did, but each shard can hold many more players. Being able to create a server that can handle that kind of scale and respond to messages from player clients in real-time is a daunting task. The sheer number of minutiae that you need to get absolutely perfect for things to work properly is mind-boggling.

The EverQuest servers, as well as similar servers for games that came shortly thereafter like Anarchy Online, were written in C++. To handle the volume of information from that large a number of simultaneous players, they had to manage their own threads, they had to do absurd low-level optimizations, and they had to make sure that every single line of code on that server was as fast and stable as possible. An entire shard (server process, or, later in EQ’s life, a cluster of server processes) simply could not crash. People paying $10-15/month for the privilege of playing the game (a concept which was brand new and controversial at the time) would not tolerate a server crash.

I am just one person, and giant game companies have entire armies of developers. How can I possibly hope to make an MMO server that players will actually use? Simple. Standing on the shoulders of giants. I don’t need to write all the low-level stuff, I don’t need to write the distributed grid-style asynchronous computing fabric that should underpin my game logic. It’s been done for me, and it’s called Akka.

You may have seen some of my previous blog posts where I talk about some of the low-level details like google protocol buffers and creating a non-blocking socket server using Akka IO. The next step toward being able to build an MMO is the core functionality required for a multiplayer game: asynchronous message routing and transmission.

The data flow for an MMO works something like this: A message comes into your server’s gateway (in my case, a non-blocking socket server). The gateway then does whatever it needs to do with the message and dispatches it accordingly. The message, or a version of it, then floats its way through the “business logic” (your game core) and as a result of it entering and executing your game logic, 0..n messages will be emitted as a result. Some of those messages will come back to the player (“your spell missed you blind moron”), some of those messages will arrive at another player’s client (“moronA failed to hit you with a magic missile”), and some messages will never leave the server environment (“persist this bit of information for me please, mr. database” or “hey NPC, player just missed you with a magic missile, your AI should respond.”). Whether you consider your NPC AI message-based or just part of your game logic is a minor semantic detail.

The point is this: we need to keep track of which players are connected, we need a way to identify those players so that messages can be targeted to them, and we need a way to be able to receive messages, funnel them into our game logic, and emit messages as a result. No matter what kind of game you’re building, no matter what your game logic may look like, this is the essential core nugget of functionality on which all of the rest of your MMO code must be built.

If you remember my dispatch map from this blog post, I can now add a new potential target for dispatch, the GameClientProxy, which is an actor that serves as a gateway between my game logic and the player. If I send the GCP a native scala case-class message, I can expect that as a result, the GCP will produce a protobuf message and put it on the wire, headed for the target player. In some cases I can send the GCP an already built protobuf message, which is the case when a player sends a DirectMessage, a protobuf message to send a “chat” from one player to another.

Here’s a look at my current dispatch map:

val dispatchMap: Map[Int, (Mergeable, String)] = Map(
   1 -> (ZombieSighting.defaultInstance.asInstanceOf[Mergeable], "akka://game/user/ServerCore/DispatchTarget"),
   MessageTypes.SignIn -> (SignIn.defaultInstance.asInstanceOf[Mergeable], "akka://game/user/ServerCore/Clients"),
   MessageTypes.DirectMessage -> (DirectMessage.defaultInstance.asInstanceOf[Mergeable], "akka://game/user/ServerCore/Clients")
  )

I’ve already got the code in place that dispatches these messages and does the protobuf de-serialization, so now I just need a “clients” actor, which is my GCP:

class ClientManager extends Actor with ActorLogging {

	var clientPlayerMap : scala.collection.mutable.Map[String, String] = scala.collection.mutable.Map.empty

	override def preStart() = {
		log.debug("Client manager pre-starting.")
	}

	def playerForUsername(username:String) : ActorRef = {
		context.actorFor(clientPlayerMap(username))
	}

	def removeUUIDFromPlayerMap(uuid:String) : Unit = {
		clientPlayerMap.foreach {  m => if (m._2 == uuid) clientPlayerMap.remove(m._1) }
	}

	def receive = {
		case ClientDisconnected(socketHandle) => {
			removeUUIDFromPlayerMap(socketHandle.uuid.toString())
			// Stop this actor
			context.stop(context.actorFor(socketHandle.uuid.toString()))
			log.debug(s"Player at socket ${socketHandle.uuid} disconnected.")
		}
		case ClientConnected(socketHandle) => {
			val newClient = context.actorOf(Props(new GameClientProxy(socketHandle)), name = socketHandle.uuid.toString())
		}
		case (uuid:String, signIn:SignIn) => {
			clientPlayerMap(signIn.username) = uuid
			log.debug(s"Player ${uuid} signed in ${signIn}")
		}
		case dm:DirectMessage => {
			log.debug(s"Player ${dm.sourcePlayer} sent dm to ${dm.targetPlayer}")
			playerForUsername(dm.targetPlayer) ! dm
		}

	}
}

This class maintains a mapping between socket UUIDs (what I use to identify the game client proxy/GCP actors) and player names. This mapping is the essential bit of state that allows messages to be targeted at players rather than UUIDs. Game clients will be unaware of the UUID of other players (because that could change if they disconnect and reconnect while messages are flowing toward them). Also note that the s”foo ${bar}” syntax is Scala 2.10 string interpolation syntax.

class GameClientProxy(socketHandle: IO.Handle) extends Actor with ActorLogging {

	override def preStart() = {
		log.debug(s"GameClientProxy pre-start, socket uuid: ${socketHandle.uuid}")
	}

	override def postStop() = {
		log.debug("GameClientProxy stopped.")
	}

	def writeRawMessage(msg:com.google.protobuf.GeneratedMessageLite, messageType: Int): Unit = {
		val bb: java.nio.ByteBuffer = ByteBuffer.allocate(4)
		bb.putInt(msg.getSerializedSize)
		bb.flip
		socketHandle.asSocket write ByteString(bb)

		val bb2 = ByteBuffer.allocate(4)
		bb2.putInt(messageType)
		bb2.flip
		socketHandle.asSocket write ByteString(bb2)
		socketHandle.asSocket write ByteString(msg.toByteArray)
	}

	def receive = {
		case m:DirectMessage => {
			log.debug("I received a direct message.")
			writeRawMessage(m, MessageTypes.DirectMessage)
		}
	}
}

This is also my attempt to delay side-effects as long as possiblewhich is a core functional programming mantra. A message being written on the wire to a particular socket is a side effect of the exchange of messages between Actors in my Actor System (the game grid). Note that I didn’t have a “player” class floating around anywhere that I used to represent a single player, instead that’s a GCP. In this blog post, I talked about the importance of modeling an actor system in a way that gets work done which may not be the same as creating an actor for every potential interactive thing visible to a player.

So, with this bit of infrastructure in place, I am now in a position where I can start building small multiplayer interaction. From there, you scale out, you test, and you build more interaction. The key point in a situation like mine where I’m just one person and not an army is that I make small bits work, I test as I go, and more importantly, the act of creating the game should be just as fun as playing it.

So far, with Akka, it’s been a blast.

Dispatching Messages in Scala Without Conditional Statements

So far I have been having a lot of fun tinkering with an Akka IO socket server sitting atop an Actor System that can send and receive protocol buffer messages, ideally when communicating with a mobile client like iOS.

Up to this point, the samples that I have been using all have a single message type and I’ve been dispatching inbound messages to a single fixed Actor. That’s not realistic. I could be getting combat messages, communication messages, navigation messages, commerce/trade messages, and any number of other types of messages. I need to know what class to de-serialize those bytes into (this is why I put the message type as a number on the wire) and I need to know which actor to send that message to once I’ve received it.

The first thing I want is to be able to create a dispatcher map, which maps a message type to a tuple that I’ll be using to store the prototype of the message (the defaultInstance , which can be used to create new instances via mergeFrom) as well as a string that contains the actor path for the actor to which I ultimately want to send the message.

Here’s how I’ve defined my dispatcher map:


type Mergeable = { def mergeFrom(data: Array[Byte]): com.google.protobuf.GeneratedMessageLite }
val dispatchMap = Map(
  1 -> (ZombieSighting.defaultInstance.asInstanceOf[Mergeable], "akka://MyGame/user/ServerCore/DispatchTarget")
)

There’s a couple of interesting things going on here. First, I’m using a structural type to indicate that anything that has a mergeFrom method on it that produces a protobuf message is called a Mergeable and I can then pass stuff around as though Mergeable was an interface, even though the protobuf implementation for Scala that I have doesn’t treat it as such.

Next, I’m mapping an int (which I’ll replace with constants later) to a tuple containing the default instance of the corresponding message and the actor path of the actor to whom I will be dispatching the message.

Next comes the part that I struggled with a litte. I want to be able to dispatch the appropriate message to the appropriate actor but I don’t want to use any statements like if (messageType == 1) { … } , because that’s very un-Scala looking and not as functional as I would like. I know that I have access to the map function in Scala which is pretty powerful, and my first attempt at using it worked, but it wasn’t quite as pretty.

After getting some more advice from the amazing people on Stack Overflow, I converted most of my map and “get” (Option handling) code into a for comprehension. Below is my completed dispatchMessage method, which is part of an Akka IO Iteratee stream:

def dispatchMessage: IO.Iteratee[Unit] =
  repeat {
	for {
		(messageType, byteString) <- readMessage
	} yield {
		for (
			(protoMessage, actorPath) <- dispatchMap.get(messageType);
			theMessage = protoMessage.mergeFrom(byteString.toArray);
			targetActor = SocketServer.system.actorFor(actorPath) )
		yield
			targetActor ! theMessage
  }
}

And now when I get to line 11, I can simply send theMessage to targetActor and the dispatching has taken place and now my facade on top of my business logic is really, really thin and the only thing I need to do is maintain the dispatch map object whenever I add new message support to my application server.

Managing your Actor Systems

If you’ve have any experience with Akka, then you know that the Actor System represents a hierarchy of managed actors. This hierarchy is a supervisory hierarchy and allows for actor supervision much the way Erlang has supported process supervision for … I dunno, since the dinosaur age for sure.

Recently I’ve been writing some code for a non-blocking socket server that has all of its business logic encapsulated within an Akka Actor System. An actor system is a very heavy weight thing and most people recommend that you only keep one of these things around per process. You might feel tempted to create multiple systems to segment different purposes within your application but all of the information I’ve found discourages this practice. I haven’t had enough real-world experience to vouch for this but, people who are way smarter than I am say to only use one actor system per process, so that’s what I’ll do.

I had some trouble when it came time to reference my actor system. I had a companion object of an actor and, unfortunately, an actor’s companion object can’t see the context object that regular actor classes can see. In other words, my method couldn’t refer to the system via context.system, which is the recommended way for actors to refer to the system to which they belong.

So, I thought I would be clever and create a new instance of ActorSystem but re-use the same name that I used the last time. I thought maybe it would seek out and find the existing actor system and just create a new lightweight reference to it. This is wrong. Here’s the proof, create two vals:

val system = ActorSystem("foo")
val system2 = ActorSystem("foo")

Now, create an actor within system via system.actorOf and maintain a reference to it. If the two actor systems are identical, then you should be able to obtain a reference to that actor with system.actorFor using system2 and it’ll be the same actor. This is not the case. Even though both systems have the same name, they are different instances, managing different actor hierarchies, with different state and potentially divergent configurations.

My solution, which may not be the best, is to do the following, in order of precedence:

  1. Refer to the actor system within an Actor class via context.system. This won’t work within actor companion objects, however.
  2. Refer to the actor system by getting it from a global object, e.g. ApplicationServer.system. This should only be done when #1 won’t work.

If someone else has a more clever way of managing and referring to the actor system, I’d love to hear it. So far, this is the only way I’ve been able to keep my code anywhere close to clean.

Designing Akka Actor Hierarchies for Online Games – Commerce

Lately I’ve been thinking a lot about the design of Actor systems and actor hierarchies, specifically as implemented by Akka. In a previous series of blog posts, I used Akka to build a telnet-gateway that went through to an Akka Actor System to support a MUD (Multi-User Dungeon/Dimension). Having learned a thing or two about high performance systems since then, I have changed my perspective on how some of these things should be designed.

User Space Modeling vs. Task Modeling

Let’s assume for a minute that we are building an MMO server that supports commerce. Players can walk into a shop and perform a number of tasks. Within the shop, players can sell goods from their inventory, buy goods from the shop keeper, or use the shop keeper as a means to engage in secure trade between players (assume for the sake of this example that players can’t exchange goods unless they’re standing in a shop – it helps illustrate my point later).

When modeling an Actor, especially in my MUD, it can be very easy to think of Actors with a clear correlation to objects within your virtual world. For example, we might create an Actor for the shop keeper and we might send him messages like:

shopKeep = context.ActorOf .... (find the shopkeeper standing in the same room as the player)
shopKeep ! SellStuff(player, 21, ItemPrototypes.JunkSword)
shopKeep ! SellStuff(player, 57, ItemPrototypes.Seashell)

In this scenario, every shop keeper in the game is a unique instance of the ShopKeeper actor. Each one can maintain their own state (their inventory or wares). Sounds good, right? My inner MUDder likes this because it maps pretty much to what the virtual world looks like on my game client. So let’s scale this out a bit. Let’s say the game has 5,000 shop keepers total. 5,000 instances of a shop keeper, each consuming roughly 300 bytes (Akka actor overhead is 300 bytes) plus whatever it takes for their state, that’s an overhead of roughly 1.4MB to manage the overhead of my shop keepers. Meh, 1.MB is nothing in relative terms, so that’s no big deal. So far so good, right? Maybe, but my new perspective says no. This particular type of scale-out is only useful if there is an even distribution of players compared to shop keepers. This is hugely important to remember. What if my game currently has 10,000 concurrent users but instead of being spread out evenly across my virtual universe, someone posted a “FREE BEER AT SHOPKEEPER2921’s HOUSE! PAR-TAY!” sign at the Orc’s Crossing. Now, I’ve got 7,500 players crowded into Shopkeeper2921‘s shop. Graphical client problems aside, this means this one actor is now trying to handle messages from 7,500 players who are buying, selling, and trading with each other. In short, that actor is going to get backed up, the game will appear slow to 7,500 of it’s players, and the proverbial shit has hit the fan.

Creating an actor for every shop keeper, actors for the shop keeper’s shop, actors for each person’s weapon, etc is what I am going to call user space modeling. I have no idea if this is real term, but it is now. It means that the architect of the actor system has decided to map user perceived actors to real actors. This is what I was doing with my MUD. What I should be doing for better performance and certainly better (cloud-sized) scalability is task-based modeling.

Let’s take a look at this problem from a slightly different point of view, task-based modeling. Instead of creating actors to back virtual constructs like shop keepers, swords, and shops, we create actors that perform tasks. If we take a look at our game client, we see that we might be sending packet-messages like “sell stuff”, “buy stuff”, “trade offer”, “trade counter”, and “trade confirm”. Instead of sending these messages to whichever shop keeper happens to be standing near the player (user space modeling), we’ll simply send the messages to whichever Commerce Actor happens to be available based on Akka routing.

Let’s say now that we create an actor CommerceActor who accepts messages in case class form SellBuyTradeOfferTradeCounterTradeConfirm. Instead of sending those messages to a nearby virtual shopkeeper, let’s send them to the “least busy” commerce actor among the N instances of CommerceActor that we’ve created with the Router mix-in trait.

Now our commerce logic might look something like this (did a little DSL-style stuff with the items just for giggles):


val shopKeeper = system.actorOf(Props[Commerce].withRouter(
    RoundRobinRouter(nrOfInstances = 10)))
shopKeeper ! Buy(shopId, 21 VorpalSwordOfDooom)
shopKeeper ! Trade(playerOne, playerTwo, 15 Gold)

Here the code is using a “round robin” router, which means each instance will be given a message in round robin fashion. You can also use a “least full mailbox” router, which essentially load balances it so that the shop keeper currently having the smallest backlog of transactions to process will be given the message. This feels right, and will scale. Even better, the routing configuration for a particular type of actor can be configured in the application.conf file. This means that you can dynamically change the number of instances of certain types of actors. This way, if players are suddenly doing a crapload of commerce and bogging your system down, you can beef up the number of instances of your commerce actor.

In task-based modeling for an actor system, actors are as the Akka documentation recommends as a best practice, encapsulations for behavior and state that perform discrete tasks and either complete the task or delegate sub-tasks to other actors. The supervisor/actor hierarchy is created based on the work that needs to be done, which doesn’t necessarily have a 1:1 mapping with the virtual world in which players inhabit.