How to write a pluggable third-party component in Go

Jin Feng
Level Up Coding
Published in
7 min readAug 4, 2020

--

In “A Self-Evolved Microservice Framework in Go”, I described a framework that can evolve itself. It has two pieces, one is the lightweight framework, the other is the pluggable third-party components. You can call this architecture “small framework, big library”. In this article, I will talk about how to create pluggable third-party components. The followings are what you need to consider.

  • External interfaces of third-party libraries
  • The internal structure of the third-party library
  • How to handle configuration parameters
  • How to expand third-party libraries

We use logs as an example to show how to create third-party libraries, I named it “glogger”. The Go language has many third-party log libraries, each of which has advantages and disadvantages. In “Go Microservice with Clean Architecture: Application Logging” I talked about that “ZAP” is so far the best logging library, but it is not perfect, I am still waiting for a better one. If there is a better one in the future, I hope that switching to the new one only needs a small code change, which is already supported by the framework I mentioned above. It is because all logging calls are made through a common interface (rather than the proprietary interface of a third-party library), so that only the creation of a logging library is library-specific (this is the part of the code that needs to be modified), while the calling of logging functions do not need to be changed.

External interfaces of third-party libraries

The calling interface

The above is the interface of the log library. Its most important principle is the universality and cannot be bound with any specific third-party log library. This interface is very important, and its stability is the key to determining whether it can be widely used. As a programmer, one dream is to be able to program like a LEGO building. The idea has been around for decades, but little progress has been made. The main reason is that there is no unified service interface. This interface needs to be cross-language, and all services must have a standard interface. In this way, the application may be pluggable. This ideal has been realized in a few areas, such as “JDBC” in Java, but its limitations are still obvious. For example, it is only suitable for SQL databases, but not for the NoSQL database; and it is only suitable for Java and not applicable in other languages.

For logging, there is an “SLF4J” in Java, which was created to realize the pluggability of logging libraries in Java. But there is no similar thing in Go, so I wrote a generic logging interface myself. It is a simple one, good enough for self-use, but not a standard one, which is difficult to create.

The interface to create an instance

In addition to the calling interface, when you create an instance of the log library, you also need to call an interface. Below is the code. You only need to call the “Build()” function and pass in the required configuration parameters.

The following code is not in the logging library, but in the “Payment Service”, which uses the logging library.

The following is the code of the “Build()” function in the logging library:

One of the dilemmas in the design is whether to put the instance-creation part in the interface. It is obvious that the logging interface must be standardized and defined as a general interface. But what about the function to create an instance? On the one hand, it should be put into the standard interface, so the whole process is complete. But doing so expands the scope of the interface, and instance-creation (including configuration parameters) itself is generally not standardized, and incorporating it into the interface increases the instability of the interface. I finally decided to include it for now and see if it causes any problem.

Configuration parameter definition

Once you include the instance-creation in the interface, it is logical to also include the definition of configuration parameters.

The following is the definition of the configuration parameters in the “glogger” library

The internal structure of the third_party library

Have you wondered why so many Go third-party libraries put all files in the root directory? It appears disorganized and difficult to manage. Why can’t I create subdirectories? When I started writing libraries myself, the reason became obvious.

Before explaining, let me talk about what is the ideal directory structure of third-party libraries first. Its structure is as shown below (still use the log as an example):

glogger.jpg

“logger.go” is the most important file, which includes all external interfaces. The file is placed in the root directory so that other programs only need to import the root directory when calling it. Second, it needs to support multiple log implementation libraries, so each library will have a separate subdirectory. For example, “Logrus” and “Zap” are two implementations that support the common logging interface, and each has its subdirectory to put the wrapper code.

The most difficult part is the problem of circular dependency. Since the external interface is defined in the root directory, and other code in the library depends on the interface, so the dependence direction is from the inside out (from subdirectories to root). The code in the “factory” directory is used to create instances.

There is a “Build()” function in the “factory.go” which will also be called by applications to create the instance of the implementation library. The ideal will be putting the “Build()” function in the “logger.go”, so that an external application only needs to import the root package of the library. But function “Build()” needs to call the factory function of the log library, which will make it depends on function in subdirectories. So the dependency directory is from outside to the inside (from the root directory to subdirectories), and a circular dependency is formed. I tried different ways to solve the problem, but none of them is ideal, the only way to work is putting all files in the root directory. Now I finally discovered the mysterious. Its biggest advantage is that an application only needs to import the root package of the library.

But the problem is that there is no structure inside the library. It is acceptable when there are only a few files. Once the number of files increases, it quickly becomes unmanageable. So, I dropped this option and created two subdirectories, “factory” and “config”. The disadvantage is that now you need three “import” statements to use the interface, and it also exposes the internal structure of the third-party library.

“config.go” and “factory.go” are best placed in the same directory, but this will also cause circular dependencies, so I have to separate them.

How to deal with configuration parameters:

How to coordinate the configuration parameters of the log library and the configuration parameters of the calling application is another difficulty. On the one hand, the configuration parameter definition and processing logic of the third-party library should be in the library, to ensure that the logic of the log library is complete and centralized in one place. On the other hand, all parameters of an application should be stored in one place. It may be stored in a file or code. The framework supports storing configuration parameters in a file, which seems to be a better way. Now, we are in a dilemma.

The solution is to divide the handling of configuration parameters into two parts, one part is the definition and logic of the configuration parameters, and this part is done by the third-party library. The other part is parameter storage, which is placed in the application, which ensures the centralized management of application parameters. When calling the library, an application can pass the parameters to the third-party library, which will decide how to deal with the parameters.

The followings are the code to initialize the “glogger” library in “payment service”. It is a part of the initialization of the entire application container, which is in “app.go”.

The following code initializes the application container. It first reads the configuration parameters and then initializes the container step by step.

Below is the code to read configuration parameters (all parameters of the application, including log configuration parameters) from a file. The code is in “appConfig.go”.

The following code initializes the log library, it passes the parameters reading from a file in the above code to the log library and calls the “Build()” function to get the specific implementation of the log interface.

How to add new implementations

The current interface includes two implementation libraries that support common logging interfaces, “zap” and “Logrus”. If you need to add a new implementation library, such as “glog”, do the following steps.

First, you need to modify “logFactory.go” to add a new implementation library option. Here is the current code:

Below is the code after modification:

Second, you need to create a “glog” subdirectory in the root directory, which contains two files. “GlogFactory.go” and “glog.go”. “glogFactory.go” is the factory file, which is the same as “logrusFactory.go”. “Glog.go” is mainly for parameter configuration and initialization of the log library.

The following is the “logrus” file “logrus.go”. You can use it as an example to write “Glog.go”. The “RegisterLogrusLog()” function is the general configuration of “logrus”, and the “customizeLogrusLogFromConfig()” is the customized configuration based on the parameters passed in by the application.

Conclusion:

This article described what needs to be done to create a pluggable third-party component in Go. It uses log service as an example. The main job is to create a common log interface and add wrapper code to call logging implementation libraries that support the common interface. I have created other general interfaces, such as “gtransaction” for transaction management and “gmessaging” for messaging. The structure of each component is very similar. It mainly contains two parts of code, one is the general interface, and the other is the wrapper code that supports each implementation library. You can add new implementation libraries in the future.

Source Code:

The complete code is in glogger

Index:

1 “A Self-Evolved Microservice Framework in Go”

2 Go Microservice with Clean Architecture: Application Logging

3 “Payment Service”

4 “zap”

5 “Logrus”

6 “glog”

7 “gtransaction”

8 “gmessaging”

--

--

Writing applications in Java and Go. Recent interests including application design and Microservice.