Kotan Code　枯淡コード

In my previous blog post, I introduced the two remaining readers of this blog to a concept and an implementation called the Disruptor, a high-performance ringbuffer-based event pipelining system that can be used to get huge performance boosts to event processing provided you don’t write foolish code.

It takes a while for most people to get used to the idea of a Disruptor, especially when just about every sample I’ve seen for the Disruptor has been related to the financial domain. Those who know me will be well aware of the fact that as I approach a new technology, I typically ask, “Yeah, that’s nice, but can you game with it?”

Turns out, the Disruptor can be used to game. Well, it can be used to write a game server for an MMORPG BTW, for those who don’t know, MMORPG is Massively Multiplayer Online Roleplaying Game, and examplars of its ilk are World of Warcraft, Star Wars the Old Republic, Guild Wars, etc.

First, let’s take a look at the problem that MMORPG server writers need to solve. Each server is typically running a bunch of individual processes. Each of these processes is responsible for handling a portion of the game’s traffic. For example, let’s say you’re in a castle on server kittenmaul. In theory, there might be a process dedicated to handling all of the messages that come in related to kittenmaul‘s Castle zone. These messages are usually a mix of TCP (guaranteed to arrive) and UDP (fire and forget). TCP sends the slow but important traffic whereas UDP sends the fast but non-mission-critical data like player positions. This flood of traffic assaults the server with messages like “player X fired a magic missile at player Y” and “player X is now at location -1,23,21″ and so on.  The server then needs to grab the important bits out of the message, make changes to in-memory state (probably by locking that portion of memory), and then free up processing threads or risk the dreaded zone lag where message processing server-side gets so slow that missiles travel through walls, trolls get stuck in trees, and players die hideous, blood-soaked deaths.

The pipeline to process these messages is probably a multi-threaded tangle of spaghetti that only a single Godlike hacker can decipher and he likes it that way. Buried deep (and I do mean DEEP) within the tangled morass of threading code is intelligence that reveals which messages can be processed in parallel and which ones need to be processed sequentially… if you’ve been vigilant. Chances are, that information is impenetrable and everything is wrapped in locks and critical sections and it’s impossible to tell what the real processing pipeline looks like without external documentation (and we all know how reliable that is 8 months after release).

Wouldn’t it be awesome if there was a way to represent the processing pipeline of messages, create simple, easy-to-maintain, small event handlers and have that be how the castle zone server maintains its pile of in-memory state? The Disruptor might actually help us out here!

For the sake of simplicity, let’s tackle just one small pipeline: spell firing. Without getting into too much detail on the division of roles between the MMO client and the server, let’s assume that the client sends the server messages that look like “player X fired spell Y at player Z” (having first passed some sort of sanity check to ensure that such an action is possible, which will be re-checked on the server as an anti-hacking measure).

Our kittenmaul server is actually running with a hot backup instance in case of failure. Each time we get a message on the live server, we replicate that message to the hot backup. We don’t want this replication to slow us down, so we need to do that “in the background” as it were. Another thing we can do in the background is journal the message to disk. We do this so that if the server goes down we can replay those messages from disk back into the server to restore the state to exactly where we left off (when was the last time you saw an MMORPG with that feature??). Once the logging and replication of the message are done, we are free to act on it by resolving the amount of damage done to the target player. Once we have the amount of damage done, we can then ship off an outbound message to the attacker and attackee’s clients containing the amount of damage done.

Those of you who have written traditional multi-threading code are probably breaking out in a cold sweat right about now. But, what if we could represent this in terms of an input disruptor and an output disruptor that create a super-fast processing pipeline?

Check out the following code:

ExecutorService executor = Executors.newFixedThreadPool(NUM_EVENT_PROCESSORS);

Disruptor<SpellFiredEvent> spellsDisruptor =
   new Disruptor<SpellFiredEvent>(SpellFiredEvent.EVENT_FACTORY, executor,
               new SingleThreadedClaimStrategy(RING_SIZE),
               new SleepingWaitStrategy());
spellsDisruptor.handleEventsWith(spellFiredLogger, spellFiredReplicator);
spellsDisruptor.after(spellFiredLogger, spellFiredReplicator)
               .handleEventsWith(spellFiredDamageResolver);

spellsRingBuffer = spellsDisruptor.start();

Disruptor<DamageDoneEvent> damageDoneEventDisruptor =
           new Disruptor<DamageDoneEvent>(DamageDoneEvent.EVENT_FACTORY, executor,
           new SingleThreadedClaimStrategy(RING_SIZE),
           new SleepingWaitStrategy());
damageDoneEventDisruptor.handleEventsWith(damageDoneHandler);

damageOutputRingBuffer = damageDoneEventDisruptor.start();

The expressive Disruptor DSL lets us fluently describe the activities that can take place in parallel and the ones which rely on the processing of previous events. In the preceding code, it is drop-dead obvious that both the logging and the replicating handlers must complete before we can resolve damage but logging and replication can occur at the same time.

And now in the damage resolution handler, we can dump the outbound damage done events to the output disruptor, which, in a real-world sample, would have down-stream handlers responsible for plucking messages off, converting them into UDP or TCP, and delivering them to clients:

private final static EventHandler<SpellFiredEvent> spellFiredDamageResolver =
        new EventHandler<SpellFiredEvent>() {
            public void onEvent(SpellFiredEvent spellFiredEvent, long l, boolean b)
                                  throws Exception {
                System.out.printf("Resolving damage to %s via spell %d%n",
                   spellFiredEvent.getTargetId(), spellFiredEvent.getSpellId());
                handleCount++;
                long sequence = damageOutputRingBuffer.next();
                DamageDoneEvent event = damageOutputRingBuffer.get(sequence);
                event.setTargetId(spellFiredEvent.getTargetId());
                event.setDamage(5000); // OUCH!
                damageOutputRingBuffer.publish(sequence);
            }
        };

So, the moral of the story here kids is that Disruptors aren’t just for financial types. A second moral of the story is go grab yourself the Disruptor code and write your own MMORPG server