Kotan Code 枯淡コード

In search of simple, elegant code

Menu Close

Tag: go

Complex JSON Handling in Go

In a recent blog post, I showed a fairly naive and contrived microservice example – the equivalent of a RESTful “hello world”. While this example may be helpful in getting some people started, it doesn’t really represent the real world. Yesterday, while spilling my Go language newbsauce all over my colleagues, I ran into an interesting problem that happens more often than you might think.

In most samples involving de-serializing a JSON response from some server, you usually see a fixed schema. In other words, you know ahead of time the expected shape of the reply, even if that means you expect some of the fields to be missing or nullable. A common pattern, however, is for an API to return some kind of generic response object as a wrapper around an optional payload.

Let’s assume you’ve got an API that exposes multiple URLs that allow you to provision different kinds of resources. Every time you POST to a resource collection, you get back some generic JSON wrapper that includes two fields: status  and data. The status field indicates whether your operation succeeded, and, if it did succeed, the data field will contain information about the thing you provisioned. Since the API is responsible for different kinds of resources, the schema of the data field may vary.

While this approach makes the job of the server/API developer easier, and provides a very consistent way of getting at the “outer” information, this generally requires clients to make 2 passes at de-serialization or un-marshaling.

First, let’s take a look at what a wrapper pattern might look like as JSON-annotated Go structs:

// GenericServerResponse - wrapper for all responses from apocalypse server.
type GenericServerResponse struct {
      //Status returns a string indicating success or failure
      Status string `json:"status"`
      //Data holds the payload of the response
      Data interface{} `json:"data,omitempty"`      
}

// ZombieApocalypseOutpost - one possibility for contents of the Data field
// in a generic server response.
type ZombieApocalypseOutpost struct {
    Name        string    `json:"name"`
    Latitude    float32   `json:"latitude"`
    Longitude   float32   `json:"longitude"`
    Status      string    `json:"status"`
}

While this example shows one possibility for the contents of the Data field, this pattern usually makes sense when there are multiple possibilities. For example, we could have one resource that provisions zombie apocalypse outposts, and another resource that provisions field artillery, where the struct inside the Data field would be something like a FieldArtillery struct, but the outer wrapper stays the same regardless.

First, let’s take a look at how the server might create a payload like this in a method rigged up to an HTTP handler function (I show how to set up HTTP handlers in my previous blog post):

func outpostProvisioner(w http.ResponseWriter, r *http.Request) {
    // check r.Method for POST
    outpost := ZombieApocalypseOutpost{Name:"Outpost Alpha",Status:"Safe",
      Latitude:37.323011, Longitude:-122.032252}
    response := GenericServerResponse{Status:"success", Data: outpost}
    b2, err := json.Marshal(response)
    if err != nil {
      panic(err)
    }
    w.Write(b2)
}

Note that on the server side we can usually get away with a single marshaling pass because we’re the ones deciding the contents of the Data field, so we know the structure we want. The client, on the other hand, likely needs to perform some conditional logic to figure out what schema applies to the wrapped payload.

You can’t use runtime typecasting tricks in Go because it doesn’t do runtime typecasting the way other languages like Java or C# might do. When we define something as interface{} like we did with the Data field, that looks like calling it Object on the surface, but as I said – we can’t typecast from interface{} to a struct.

What I can do is take advantage of the fact that we already have marshal/unmarshal capabilities in Go. What follows may not be the best, most idiomatic way to do it, but it seems to work out nicely.

  URL := "http://localhost:8080/outposts/"

  r, _ := http.Get(URL)
  response, _ := ioutil.ReadAll(r.Body)
  r.Body.Close()

  var responseMessage GenericServerResponse
  err := json.Unmarshal(response, &responseMessage)
  if err != nil {
    panic(err)
  }
  var outpost ZombieApocalypseOutpost
  b, err := json.Marshal(responseMessage.Data)
  err = json.Unmarshal(b, &outpost)

  fmt.Println(responseMessage)
  fmt.Println(outpost)

First, we grab the raw payload from the server (which is just hosting the method I listed above). Once we’ve got the raw payload, we can unmarshal the outer or wrapper message (the GenericServerResponse JSON-annotated struct). At this point, the Data field is of type interface{} which really means we know nothing about it. Internally one can assume that the JSON unmarshalling code probably produced some kind of map, but I don’t want to deal with the low-level stuff. So, I marshal the Data field which gives me a byte array. Conveniently, a byte array is exactly what the unmarshal method expects. So I produce a byte array and then decode that byte array into the ZombieApocalypseOutpost struct.

This may seem like a little bit of added complexity on the client side, but there is actually a valuable tradeoff here. We gain the advantage of dealing with virtually all of an APIs resources using the same wrapper message, which greatly simplifies our client code. Only when we actually need the data inside the wrapper do we need to crack it open and perform the second pass, which can also be hidden behind a simple library.

I used to cringe every time I saw this pattern because I’m not a fan of black box data. If the inside of the Data field can vary in schema, then you are at the mercy of the API provider to insulate you from changes to the inner data. However, you’re no more a potential victim of breaking changes using this pattern than you would be without the wrapper.

So, in short, if you trust the API provider using this wrapper pattern, it can actually be a very convenient way to standardize on a microservice communication protocol.

Creating a Microservice in Go

A while back (like, quite a while) I started playing with Go and then unfortunate circumstances kept me from continuing my exploration. My recent exposure to super brilliant people has rekindled some of my desire to explore new languages. Since I’ve been knee-deep in microservices recently, I decided to try and build a microservice in Go.

There are a number of libraries available to Go developers that help with this kind of thing (including Martini). But, I want to see what it looks like to build the most minimal microservice possible. Turns out, that’s pretty damn minimal:

package main

import (
 "encoding/json"
 "net/http"
)

func main() {
 http.HandleFunc("/zombie/", zombie)
 http.ListenAndServe(":8080", nil)
}

type ZombieThing struct {
 Text string
 Name string
 Age int
}

func zombie (w http.ResponseWriter, r *http.Request) {
  zomb := ZombieThing {"Watch out for this guy!", "Bob Zombie", 12}
  b, err := json.Marshal(zomb)
  if err != nil {
    panic(err)
  }
  w.Write(b)
}

So now I just type go run service.go and the service is running. Now I can just curl the service:

$ curl http://localhost:8080/zombie/
 {"Text":"Watch out for this guy!","Name":"Bob Zombie","Age":12}

If you’re wondering why I used upper-case member names for the JSON object being returned, it’s because I’m lazy. Go considers a variable name with lower case to be private or non-exported, so if I were to make the ZombieThing struct have lowercase names, nothing would be exported and the service would return {}.

So, basically we have yet another example in a long list of examples proving that microservices are a language-agnostic architecture and you can build them in pretty much whatever language and framework that suits your needs.

Concurrency in Go with Channels and Goroutines

So far I’ve been walking myself through learning some of the basic features of the Go language and I’ve been impressed with what I’ve seen. I still have this nagging feeling that Go is a solution looking for a problem and I haven’t decided if that’s because I haven’t yet had that “a-ha!” moment where I figure out what problem Go solves miraculously or if it’s just because I’ve only been poking around with it for a few days.

Concurrency is one of the more heavily lauded features of the language, but the community is quick to remind you that concurrency is not parallelism. I won’t go into detail on that subject because it’s a really meaty one and I just want to provide a quick introduction to concurrency here.

The way to think about goroutines is that they are very small, light-weight chunks of processing that you compose in order to chew up a larger task. There may be a correlation here between goroutines and actors and in terms of design, I’ve noticed a near 1:1 correlation between what I would normally code into an actor with what I find being coded into goroutines in samples. The way goroutines communicate with each other and the rest of your code is through channels which can send and receive messages.

One distinction that I think is unique to Go is that sending and receiving on channels is a synchronizing operation. In other words, if you use the arrow notation (<-) to receive a value from a channel, you will block until you receive that value. Conversely, if you send a value on a channel, the code that sent the value will be blocked until some other goroutine plucks that value off the channel .You can get around this with buffered channels, but it seems to be a fundamental thing that goroutines synchronize via communication rather than shared memory.

In my little sample, I decided to take the synchronization “hello world” of a producer and a consumer and implement it in Go. In the following code, I’ve got a type called StockTick and I’ve created a channel that carries stock ticks (channels are strongly typed). I invoke a producer and a consumer and thus have two separate blocks of code communicating with each other:

package main

import (
	"fmt"
	"math/rand"
)

type StockTick struct {
	Ask		float32
	Bid		float32
	Symbol	string
}

func genTick() (tick StockTick) {
	tick.Ask = 45 * rand.Float32()
	tick.Bid = 42.50 * rand.Float32()
	tick.Symbol = "IBM"
	return tick
}

func produceTicks(stocks chan StockTick) {
	for x:=1; x<100; x++ {
		stocks <- genTick()
	}
	close(stocks)
}

func consumeTicks(stocks chan StockTick) {
	for stock := range stocks {
		fmt.Printf("Consumed a tick %v\n", stock)
	}
}

func main() {
	fmt.Printf("Stock Channels FTW\n\n")
	stocks := make(chan StockTick)
	go produceTicks(stocks)
	consumeTicks(stocks)
}

There are a couple of things that I found interesting in this code that caught me by surprise. The first is that I had to close the stocks channel when I was done producing. This is because the range operator creates a deadlock if the code reaches a point where it knows that there can be no more values to consume (the producer goroutine is done). You can get around this in real-world apps where you plan on consuming all day with an infinite loop, but in this case I knew I was done so calling close(stocks) is actually what allows the range operator to terminate in a friendly way without a deadlock.

Next, notice that I used the go keyword to fire off the produceTicks method as an asynchronous goroutine but I didn’t do that for consumeTicks. This is because if I let them both be asynchronous in the background, the main() function would terminate, which kills the go application. In a go app, as soon as main() exits, the entire app quits, even if you have active goroutines.

There are a number of higher-level patterns that you can implement with channels such as sync’ing on acknowledgements sent back on the channel from which a value is received and using a “fan-in” approach where you’re pulling from multiple channels to produce values on a single channel so that you aren’t waiting in lock-step for values to arrive in sequence across channels. As this is still just a series of blog posts covering my introduction to the language, I won’t cover those yet.

As I said, I am still unsure whether I have any problems that Go would be ideal to solve, but I am going to keep plugging away until I decide one way or the other.

Implicit Interfaces in Go

In my continuing obsession with learning a new language, I’ve gotten to the point in experimenting with Go that I want to know what all this implicit interface stuff is about. In my previous blog post, I talked about how Go doesn’t actually have classes, but rather it has explicit receiver functions that look like they are methods being invoked on an object.

The rumor on the street is that Go has a facility similar to duck typing in that when you declare an interface, anything can be said to implement that interface as long as that thing has all of the method definitions declared in the interface. This isn’t entirely unique – dynamic languages are full of stuff like this and Scala’s got the ability to define type aliases that do the same thing. The curious thing about Go’s implicit interface satisfaction is that it is compile-time checked. That’s right – your code won’t compile if you have code that attempts to convert an object into an interface that it can’t satisfy.

Let’s take a real-world example: wielding a chair (what, that’s not real-world to you?). When building MUDs and even some more modern game types, you often run into this type of thing. If a developer didn’t originally intend for an object to be used as a weapon, it can’t be treated like one. But, what if you want to make it so that the player can wield a chair as easily as a sword? Sure, they will have different behavioral characteristics, but you should still be able to wield either.

In a classic programming language problem, this is easy to solve: Create an interface called Wieldable (or IWieldable if you’re a .NET person) and then two classes that both implement that interface: Chair and Sword. In Go, you still create a Wieldable interface that defines a Wield() method, but the difference is you never create a class nor do you explicitly indicate what interface you’re implementing. If there’s an explicit receiver in scope called Wield() for that object, then your code can treat it as a wieldable.

Let’s take a look at some code:

// interfaces
package main

import (
	"fmt"
)

type Wieldable interface {
	Wield()
}

type Chair struct {
	Legs	int
	Wielded bool
	Name	string
}

type Sword struct {
	DamageRating	int
	Wielded			bool
	Name			string
}

type Test struct {
	Wieldy		Wieldable
}

func (s *Sword) Wield() {
	fmt.Printf("Wielding Sword %s\n", s.Name)
	s.Wielded = true
}

func (c *Chair) Wield() {
	fmt.Printf("Wielding Chair %s\n", c.Name)
	c.Wielded = true
}

func main() {
	fmt.Println("Hello World!")

	s := Sword{
		DamageRating: 12,
		Wielded: false,
		Name: "Sword of Doom",
	}

	c := Chair{
		Legs: 4,
		Wielded: false,
		Name: "Death Seat",
	}

	t := Test {
		Wieldy: &s,
	}

	t2 := Test {
		Wieldy: &c,
	}

	t.Wieldy.Wield()
	t2.Wieldy.Wield()
}

In this code I am creating a Sword struct and a Chair struct and then I created an interface called Wieldable and another struct called Wieldy which holds a Wieldable object. To prove that I can convert between Chair/Sword and Wieldable, I create two test objects – each holding a Wieldable converted from either a sword and I then call the Wield() method on each.

So here’s something unexpected – this is a one-way conversion. In a runtime managed language when you typecast a reference to an object from one type to another, the actual underlying object remains unchanged and you get a runtime-limited view onto that object. For example, if you’ve created some serializable object called Zombie, you can cast that to a Serializable and then still get back the original zombie. If you attempt to get a Sword object out of a Test.Wieldy field, you will get an error message from the compiler.

That said, you can still prove that the wield methods did actually change the value of the things you’re wielding:

fmt.Printf("Wield status: %s, %s", s.Wielded, c.Wielded)

The big take-away that I learned from this experiment is that while implicit interfaces are pretty damn awesome, you should not be using them so you can do a pile of random typecasting or to somehow treat your structs like the target for mixins (traits in Scala). They should be used to assert that a particular set of receiver methods exist for a particular data type. And since there are no classes and there is no inheritance, that data type is strictly enforced.

Further, I might have been better off just using embedding to contain a wieldable inside either my chair or sword objects, which would make them both wieldable. Also, interfaces cannot contain fields. So you might be tempted to try and model some statement like “anything wieldable should have a wielded boolean property”. You cannot model this with interfaces.

So I’m already starting to run into scenarios where it feels awkward or downright limiting to model things like business logic (or game logic) in Go. I will keep trying, however, to see if there’s a more Go-idiomatic way of doing things that I’m not seeing because I’m still knee-deep in the class-and-inheritance world.

Object-Oriented Design in Go

One of the first things I tend to look for in exploring a new language is OOD. What does a class look like? How does inheritance work? And, because I’m such a huge fan of Scala, I now ask questions like “Does it support mixins (traits)?”. The trick with Go is that there are no classes. That’s right, it’s a language that can be called object-oriented but you’ll never write a single class.

Let’s take a look at a little bit of sample code, where I’ve created a struct (remember those? Ah, the good old days…) called Mob and I’ve also created one called Coordinate:

type Coordinate struct {
	X	int
	Y	int
}

type Mob struct {
	Hitpoints	int
	Name		string
	AggroRadius	int
	Coordinate
}

Something worth noting here is that it might look like Mob inherits from Coordinate, but that’s not what’s happening. Instead, the Mob struct is embedding Coordinate struct. To really be object-oriented, I need to be able to write methods on my “objects”. Go supports this by allowing you to define functions that have something called an explicit receiver. In other words, when you invoke a method on a struct, that instance of that struct becomes the explicit receiver for that function.

Let’s create a method that moves our mob around a game board. In the process, you’ll see another one of Go’s differences from traditional C in that it can return multiple values. What happens if we invoke a method called MoveTo in a traditional OOP language and it fails? How do we know where the mob stopped en route to its original location? There are other classes we traditionally create like MoveResult or some junk like that, but that’s all ceremony slapped on top of the real problem. Here’s how to do it in Go:

func (mob *Mob) MoveTo(x, y int) (success bool, newLoc Coordinate) {
	success = true
	newLoc = Coordinate{x,y}

	mob.X = x
	mob.Coordinate.Y = y // can choose to be explicit about embed or not

	return success, newLoc
}

In the preceding code, the mob *Mob is declaring an explicit receiver of type Mob and it will be named mob (think of it as being able to provide a name for the traditional this in other OOP languages). X and Y are integer parameters to the method, and success and newLoc are the return values from the method.

Now let’s take a look at how to create a Mob and use it:

func main() {
	var brutus = Mob{
		Hitpoints: 500,
		Name: "Brutus",
		AggroRadius: 20,
		Coordinate: Coordinate{X: 100, Y: 200},
	}

	madeIt, newLoc := brutus.MoveTo(60, 50)
	fmt.Printf("Did I make it? %s\n", madeIt)
	fmt.Printf("Brutus is now at %d,%d\n", newLoc.X, newLoc.Y)

	// Can also shortcut through the embed:
	fmt.Printf("Brutus's X location - %d (or %d)\n", brutus.X, brutus.Coordinate.X)
}

I strongly encourage you, if you’re just learning Go, to copy this stuff in (remember each file needs a package) and run it with something like go run mob.go. Also remember that there’s absolutely no restriction on filename and it doesn’t need to correlate to the package it contains or the structs used. One subtle take-away from this is that you can add explicit receiver functions (methods) to any struct from anywhere in your code.

Here’s something subtle but important: Did you notice that it looked like I used a pointer in the MoveTo method? If you’ve got the code running, go ahead and delete the asterisk and run it again. Note that you don’t get any memory violations or null pointer exceptions or any of the usual crap you would expect from downgrading a pointer to a stack reference. BUT, what does happen is that the modification of the mob’s current location is ignored. Well, it’s not really ignored, but the modification is being made to a stack copy of the mob which is different than the mob reference sitting inside the main() function. To ensure that changes made inside the method actually happen to the shared instance of the mob between main() and the method, you need to use a pointer. This is because the method isn’t some vtable dispatched reflection-discovered shenanigan, it’s a legitimate C-style function that just happens to have been passed a value called mob. This is one of those dichotomies I mentioned in my previous blog post, about how it seems that Go is straddling the line between low-level, C-like functionality and modern awesomesauce like goroutines and asynchronous processing via communication over channels.

In my next post, I’ll show you some stuff about type inference and another cool feature: implicit interfaces.

First Impressions of the Go Language

I recently got a chance to hang out with some very brilliant people and one of the guys mentioned how fond he was of Go, a programming language most commonly associated with Google. To be honest, the fact that it was “yet another thing that came out of Google” is exactly why I hadn’t played with it up until this point. I’d seen a few blog posts but skimmed them. This time, however, I decided to give Go a more fair shake and so I downloaded it and started hacking away. What happened next shocked even me.

I’m not a coding newb, but even I know that there’s a learning curve to each language and recently the learning curves have been getting steeper – Scala and Clojure come to mind, as well as some of the UI frameworks that aren’t languages, but they have language-sized learning curves (AngularJS is the 800 pound gorilla of learning curves). That’s why when I had ‘Hello World’ compiled and running in about 12 minutes after my first click on golang.org, I was shocked.

If you are unfamiliar with Go, it takes a little getting used to because at first glance it seems like a hodgepodge of some features you would never expect to see in the same language. Here’s a quick list that I found over on this blog post entitled Why Go?:

  • Asynchronous – We’re all about reactive programming these days and all the hip kids are building non-blocking, asynchronous code. In fact, if I hear one more Node.js zealot shout “it’s non-blocking” one more time, I will stab him in the face with a web socket. Go has these things called Goroutines that are lightweight async execution blocks and they have channels that make my inner Erlang and Akka fanboy giggle. Bottom line is you write synchronous-looking code but it’s non-blocking and there are no callbacks. Callback hell needs to be avoided at all costs.
  • Concurrency – Paired with the asynchronous support features and support for multi-core with channels is very appealing, especially considering some of the other aspects of the language (check out the next bullet, you’re going to crap yourself)
  • Static Binaries – Oh that’s right. There’s no JVM. There’s no VM of any kind. You don’t have to wait for a runtime to go dig through classpath garbage and dynamically load a hojillion JAR files. These are compilednative binaries. Remember way back in the good old days when you could just pass around and executable and that worked?? Yeah, I had almost forgotten those times as well.
  • Language Features – In addition to looking like C, having asynchronous, actor-like support for concurrency and parallel programming, it also has a lot of the goodies that people think you can only get from some of the newer JVM or functional languages  like type inference and my personal favorite, implicitly satisfied interfaces … wait wait.. it gets better… the implicitly satisfied interfaces are checked and satisfied at compile-time… I almost shed a tear when I saw that.
  • Privacy indicated via case, not keywords – Capital lettered members are exported, lowercase ones are not.
  • Imperative, object-oriented language – You’d think this is contradictory, but with Go it just seems to work. I was very skeptical until I started playing with it. Go doesn’t have classes, yet it supports message passing via methods, polymorphism, and namespacing. Crazy, right?
  • Multiple return values – The language looks like C, but you can invoke methods that return multiple values and it looks just like a Scala unapply syntax. Me likey.

So, I sat down and I downloaded Go and got it working and followed the advice of setting up the $GOPATH environment variable that pointed to a single root under which all of my Go code would reside. It felt a little awkward, but there seemed to be some method to the madness, specifically relating to the fact that Go can fetch its dependencies directly from Github and that it can resolve transitive dependencies without the need for a make file, maven, or sbt. Holy crap – compiling to a native binary without maven or sbt? That’s cause for celebration.

So, let’s get to the hello world? (You can get the instructions to download Go and set up “Hello World” from here at golang.org.

package main

import "fmt"

func main() {
  fmt.Printf("Hello world.\n")
}

When you issue a go install command at the command line, this compiles your Go code. It doesn’t just create a JAR file or a DLL file that will be interpreted (JIT or otherwise) by a runtime like the .NET CLR or the Java Virtual Machine, it creates a static standalone self-executing binary. Ah but you say Java can create executable JARs too, right? This isn’t the same, this is a native executable whereas an executable JAR is just a regular JAR with the JVM launching bootstrapped in.

When I first sat down and started reading the docs on Go, I had mixed feelings and still do. On the one hand, it feels like a step back in that some of it looks like old school C (don’t get me wrong, I love me some old school C) whereas other parts are amazingly terse yet ridiculously powerful (channels, goroutines, slices, transitive no-makefile dependency resolution). I don’t know yet what my final thoughts are, but I know that right now it is appealing to my love of the C language (I think Go is one of the most C-pure looking syntaxes I’ve seen in a long time) with my love of Erlang, the actor pattern, and asynchronous back-end programming.

This is just hello world. If you’ve read this blog before then you know the real test I have to put Go through in order to vet it as a language and I’ll be posting a series of blog posts on that soon. The real question I ask of any new language is this: Can I create a MUD with this language?

I’m about to find out – stay tuned!