Go is an awesome language. It’s simple, powerful, has great tooling and many of us really enjoy using it every day. However, as it usually happens with strongly typed languages, we write a good deal of boilerplate to connect things around.

In this post we’ll cover mostly three points:

1. Why can we build tools with Go that will help reduce boilerplate using code generation.
2. What are the building blocks for code generation in Go.
3. Where can we find examples of code generation tools to learn more.

Why use code generation to reduce boilerplate?

Sometimes we try to reduce boilerplate by using reflection and filling our projects with methods accepting interface{}. However, whenever a method takes an interface{}, we are throwing our type safety out of the window. When using type assertions and reflection, the compiler is unable to check we are passing the right types and we are more exposed to runtime panics.

Some of the boilerplate code we’ve got can be mostly inferred from the code we already have in our project. For that, we can write tools that will read our project’s source code and generate the relevant code.

The building blocks to code generation

Reading code

The standard library has a wonderful set of packages ready to do most of the heavy lifting when it comes to reading and parsing code.

• go/build: gathers information about a go package. Given a package name, it will return information such as what’s the directory containing the source code for the package, what are the code and test files in the directory, what other packages it depends on, etc.
• go/scanner and go/parser: read source code and parse it to generate an Abstract Syntax Tree (AST).
• go/ast: declares the types used to represent the AST and includes some methods to help walking and modifying the tree.
• go/types: declares the data types and implements the algorithms used for type-checking Go packages. While go/ast contains the raw tree, this package does all the work to process the AST so you can get information about types directly.

Generating code

When generating code, most projects just rely on the good old text/template to generate the code.

I recommend starting generated files with a comment indicating that the code is automatically generated, which tool generated it and mentioning that it should not be edited by hand.

1
2
3
4

/*
* CODE GENERATED AUTOMATICALLY WITH github.com/ernesto-jimenez/gogen/unmarshalmap
* THIS FILE SHOULD NOT BE EDITED BY HAND
*/

We can also use the go/format package to format our code before writing it. This package contains the logic used by go fmt.

go generate

Once we start writing tools that generate source code for our programs, two questions appear quickly: At what point in our development process do we generate the code? How do we keep the generated code up to date?

Since 1.4, the go tool comes with the generate command. It allows us to run the tools we use for code generation with the go tool itself. We can specify which commands need to be run using special comments within our source code and go generate will do the work for us.

We just need to add a comment with the following format:

1

//go:generate shell command

Once you have that, go generate will automatically call command whenever it’s run.

There are two points that are important to remember:

• go generate is meant to be run by the developer authoring the program or package. It’s never called automatically by go get
• You need to have all the tools invoked by go generate already installed and setup in your system. Make sure you document which tools you are going to use and where those tools can be downloaded.

Also, if your code generation tool is within the same repository, I would recommend calling go run from go:generate. That way, you can run generate without building and installing the tool manually each time you change the tool.

How do you start building your own tools?

The stdlib packages to parse and generate code are great, but their documentation is huge and making sense of how to use the packages just from the docs can be quite daunting.

The best thing I did when I got into code generation was to learn about some of the existing tools. It serves three purposes:

1. You’ll get some inspiration about the kind of tools you can build.
2. You’ll have the chance to learn from the tools’ source code.
3. You can find some of these tools really useful by themselves.

Projects to learn from

Generating stubs to implement an interface

Have you ever found yourself copying and pasting the list of methods defined in an interface you’ve got to implement?

You can use impl to generate the stubs automatically. It will use the packages in the stdlib to look for the interface and output methods you must implement.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ impl 'f *File' io.ReadWriteCloser
func (f *File) Read(p []byte) (n int, err error) {
    panic("not implemented")
}

func (f *File) Write(p []byte) (n int, err error) {
    panic("not implemented")
}

func (f *File) Close() error {
    panic("not implemented")
}

Generating mocks automatically with mockery

testify has a nice mock package to easily mock your dependencies when you are doing unit testing. Because interfaces are satisfied implicitly, we can specify our dependencies using interfaces and use a mock during unit testing rather than the external dependency.

Here’s a very simplified example about how to mock a theoretical downcaser interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package main

import (
  "testing"

  "github.com/stretchr/testify/mock"
)

type downcaser interface {
  Downcase(string) (string, error)
}

func TestMock(t *testing.T) {
  m := &mockDowncaser{}
  m.On("Downcase", "FOO").Return("foo", nil)
  m.Downcase("FOO")
  m.AssertNumberOfCalls(t, "Downcase", 1)
}

The implementation of the mock is pretty straightforward:

1
2
3
4
5
6
7
8

type mockDowncaser struct {
  mock.Mock
}

func (m *mockDowncaser) Downcase(a0 string) (string, error) {
  ret := m.Called(a0)
  return ret.Get(0).(string), ret.Error(1)
}

Actually, as we can see from the implementation, it’s so straightforward that the interface definition itself has all the information we need to generate a mock automatically.

That’s what mockery does:

1
2

$ mockery -inpkg -testonly -name=downcaser
Generating mock for: downcaser

I always use it with go generate to automatically create the mocks for my interfaces. We just have to add one line of code to our previous example to have a mock up and running.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package main

import (
  "testing"
)

type downcaser interface {
  Downcase(string) (string, error)
}

//go:generate mockery -inpkg -testonly -name=downcaser

func TestMock(t *testing.T) {
  m := &mockDowncaser{}
  m.On("Downcase", "FOO").Return("foo", nil)
  m.Downcase("FOO")
  m.AssertNumberOfCalls(t, "Downcase", 1)
}

Here you can see how everything gets set up once we run go generate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ go test
# github.com/ernesto-jimenez/test
./main_test.go:14: undefined: mockDowncaser
FAIL    github.com/ernesto-jimenez/test [build failed]

$ go generate
Generating mock for: downcaser

$ go test
PASS
ok      github.com/ernesto-jimenez/test 0.011s

Whenever we make a change to an interface we just need to run go generate and the corresponding mock will be updated.

mockery is the main reason I started contributing to testify/mock and became a maintainer for testify. However, because it was developed before go/types was part of the standard library in 1.5, it’s implemented using the lower level go/ast, which makes the code harder to follow and also introduces some bugs like failing to generate mocks from interfaces using composition.

gogen experiments

I’ve open sourced the code generation tools I’ve been building to learn more about code generation in my gogen package.

It includes three tools right now:

• goautomock: is similar to mockery but implemented using go/types rather than go/ast, so it works with composed interfaces too. It’s also easier to mock interfaces from the standard library.
• gounmarshalmap: takes a struct and generates a UnmarshalMap(map[string]interface{}) function for the struct that decodes a map into the struct. It’s built to work as an alternative to mapstructure using code generation rather than reflection.
• gospecific: is a tiny experiment to generate specific packages from generic ones that rely on interface{}. It reads the generic’s package source code and generates a new package using a specific type where the generic package used interface{}.

Wrappping up

Code generation is great, it can spare us from writing tons of repetitive code while keeping our programs type safe. We use it extensively when working on Slackline and we’ll probably use it soon in testify too.

However, remember to ask yourself: is writing this tool worth the time?

xkcd wants to help us by answering that question: