How I Embedded Resources in Go
During my internship period at WSO2, Inc I worked on a project to develop a CI/CD pipeline for WSO2 API Manager. The tooling was mostly done in Golang.
When we were developing the tooling we wanted to have a project initialization phase through the CLI tool. That indeed involves a lot of code generation.
In the beginning, everything was fine. We had few files, and we kept them as slices of bytes and accessed them, which was totally fine.
Nightmare of backticks `````
The project was scaling up and content we wanted to store was growing too. It was very hard to manage these contents as they were basically in Go packages and had to manually edit them each and every time.
One day I wanted to use Markdown for our README file which generated by the tool in the initialization phase of a project, we were generating a simple Text file until then.
Markdown uses backticks for code blocks and Go uses backticks for Raw Strings, I could not find a better way to escape them. It was a total disaster!
We can’t ship these files separately as this was focused to be a small CLI tool, so I kept looking methods to embed resources into Go.
Existing Solutions
Yes, there are existing solutions for the exact problem like Go.rice, but for my task, I thought they are too complex and bloated. All I wanted to do is convert my resources into Go files and access them.
Go generate for rescue
Go is awesome, it comes batteries included. The language is capable of code generation by default. So why not use it. If you are not familiar with what go generate is, read the official blog. Go generate uses a special magic string to identify files that need to be generated
//go:generate <command> <args>
When you execute go generate filename.go
the compiler will check for the magic string and execute the command provided. go generate ./...
will run the command through an entire project looking for files with magic string and executing the command you provided.
Go generate runs the given command relative to the directory containing the file with magic string.
Since we need to make our code generation universal, why shouldn’t we use go itself for code generation?
The Setup
My main requirement was to store files as bytes and access them in my Go files. To fulfil the same requirement I created a package called box
. It was acting as a proxy between consumers and data source. We store our all files inside a special directory called resources inside the box package. It can contain directories as well to organize content.
This makes editing content much more easier and much more clean as they are just files in the system.
This is a very simple Go file, all it does is wrap our map inside a nice ResourceBox
exposing all methods at package level so you can invoke them using box.Get('filename')
anywhere — super cool.
The resource
object plays a key role in the package acting as the singleton to hold all the resources for our box.
Now here is the magic, it has a special comment with //go:generate go run gen.go
.
When you are invoking go generate ./…
this file will be detected and it will run another go program.
Generator
Now the fun part, we want to read our resource directory and store the content of files in a ResourceBox
. For that we are running another Go file called gen.go
which is in the same package box
.
As I described above, the go generate
is smart enough to run the given program relative to the directory it found the magic string, so go run gen.go
would run inside the box directory. It means we can easily read all the files in the resources directory without a problem.
We are going to create a file called blob.go
with all the content we need inside the same package.
Since gen.go
is a program, we need to tell Go that this is not a part of our build. Otherwise, it will complain about declaring multiple packages in the same directory.
So just add
//+build ignore
at the top of the file to inform the compiler to ignore this on the build. So it won’t complain.
Now we need to walk through all the files in the resources directory and convert them to go files.
Go provides a nice way to handle this using filepath.Walk
method. We are simply iterating all the files inside the resources directory (including all the subdirectories as well) and storing them in a map called resources
. We are going to access files using the relative path to the resources directory.
For example to access a file in resources/init/sample.yaml we can use box.Get('/init/sample.yaml')
in a very intuitive way. But what if we run the generation in a Windows machine?
To overcome this we are going to use a feature called ToSlash
in the filepath
package, which will transform the platform native path separator to a slash.
filepath.ToSlash(strings.TrimPrefix(path, "resources"))
Next, we need to create blob.go
with all the data we have. We just need to add all the data to the resources object in the box package.
We just simply need to call resources.Add('/file/path', data)
. What we can do is putting this in the init()
method on the blob.go
which will be initialized for the whole package at once.
package box
func init() {
resources.Add("/init/README.md", []byte{50, 70, 80, })
resources.Add("/init/api_params.tmpl", []byte{100, 110, 125,})
...
}
To do this we simply use Go Templates as following
It simply iterates and creates the actual file containing the name and data for the file, later we will write this to the disk allowing access within the code.
You can see there is a function map passed to the template engine. I used this to create the byte representation of the slice as a string and use it within the template. It’s a very simple function as follows. It takes a slice of byte and returns a comma-separated value string containing all the bytes as numbers.
If you had any back-tick now they all have been converted to some numbers, so no worries
Now we write the file to the disk and we are ready to Go
But wait, my linters are failing
This is because the code is not properly formatted, you can do it yourself after generation.
Nope, we are not gonna do that, Go itself provides the formatter for formatting any go code, we can just invoke it before saving the file to the disk
data, err := format.Source(builder.Bytes())
if err != nil {
log.Fatal("Error formatting generated code", err)
}
Simple as that.
Run it
Now just run go generate ./…
within your application and you can find a file called box/blob.go
, it will contain all the data for resources in your box/resources
directory.
Now all you have to do is invoking
box.Get('/sample/sample_config.yaml')
To get the data from your file. Now it is embedded in your Go binary.
Wrap Up
There were many existing solutions out there for solving this problem of embedding with advanced features. But I wanted to implement a simple solution for solving the problem we had.
Go is a powerful language which has many features that every programmer wants in their tool belt. It has templates, code generation and many more within the standard library.
This solution separates content from the code and makes them intuitive and more easy to manage.
For example, if a content writer wanted to correct some issues in your README file they can simply edit that file and build the program without thinking of the implementation. This is a win/win for both parties.
Code for the entire project: