Since golang does not have a uniform coding pattern like java, we, like the rest of the team, used some of the theory presented in the article Go Package Oriented Design and Architecture Layering and then combined it with our previous project experience to define the packge.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

├── cmd/
│   └── main.go //启动函数
├── etc
│   └── dev_conf.yaml              // 配置文件 
├── global
│   └── global.go //全局变量引用，如数据库、kafka等
├── internal/
│       └── service/
│           └── xxx_service.go //业务逻辑处理类
│           └── xxx_service_test.go 
│       └── model/
│           └── xxx_info.go//结构体
│       └── api/
│           └── xxx_api.go//路由对应的接口实现
│       └── router/
│           └── router.go//路由
│       └── pkg/
│           └── datetool//时间工具类
│           └── jsontool//json 工具类

In fact, this division above is only a simple division of functions into packages, there are still many problems in the process of project practice. For example.

For the function implementation do I pass it through the parameters of a function or through the variables of a structure?

Is it safe to use a reference to a database global variable? Is there excessive coupling?

Almost all of the code implementation is implementation dependent rather than interface dependent, so does switching from MySQL to MongDB require all the implementation changes?

So now as we work with more and more code, the code feels more and more chaotic with all the init, function, struct and global variables.

Each module is not independent, it seems to be divided into modules according to logic, but there is no clear relationship between the upper and lower levels, each module may have configuration reading, external service calls, protocol conversion, etc..

Over time, the calls between the different package functions of the service slowly evolve into a mesh structure, and the flow of data and logic becomes more and more complex to sort out, making it difficult to figure out the flow of data without looking at the code calls.

sobyte

But as it says in Refactoring: first make the code work - if it doesn’t work, it can’t produce value; then try to make it better - by refactoring the code so that we ourselves and others understand it better and can keep modifying it as needed.

So I think it’s time for some self-change.

The Clean Architecture

The Clean Architecture sets out a number of requirements for our projects.

  • Independence from the framework. The architecture is not dependent on the existence of certain feature-rich software libraries. This allows you to use these frameworks as tools, rather than cramming your system into their limited constraints.
  • Testable. Business rules can be tested without a UI, database, web server or any other external elements.
  • Independent of the user interface. the UI can be easily changed without having to alter the rest of the system. For example, a Web UI can be replaced with a console UI without changing the business rules.
  • Database independent. You can swap out Oracle or SQL Server for Mongo, BigTable, CouchDB or something else. Your business rules are not bound by the database.
  • Independence from any external bodies. In fact, your business rules don’t know anything about the outside world at all.

sobyte

The concentric circles in the diagram above represent a variety of different areas of software. Generally speaking, the deeper you go the higher the level of software you have. The outer circles are the tactical implementation mechanisms and the inner circles are the strategic core strategy.

For our projects, code dependencies should be outward to inward, one-way single-level dependencies that contain code names, or functions of classes, variables or any other named software entities.

For a clean architecture there are four layers.

  • Entities: Entities
  • Usecase: expresses the application business rules, corresponding to the application layer, which encapsulates and implements all the use cases of the system.
  • Interface Adapters: the software in this layer is basically adapters that are used to convert the data in use cases and entities into data for use in external systems such as databases or the Web.
  • Framework & Driver: the outermost circle is usually made up of frameworks and tools such as Database, Web Framework, etc.

For my project, then, it is also divided into four layers.

  • models
  • repo
  • service
  • api

sobyte

models

Encapsulates various entity class objects, those that interact with the database, those that interact with the UI, etc. Any entity class should be placed here. For example.

1
2
3
4
5
6
7
8
9

import "time"

type Article struct {
    ID        int64     `json:"id"`
    Title     string    `json:"title"`
    Content   string    `json:"content"`
    UpdatedAt time.Time `json:"updated_at"`
    CreatedAt time.Time `json:"created_at"`
}

repo

This is where the database operation classes are stored, the database CRUD is all here. It is important to note that this does not contain any business logic code, as many people like to put business logic here as well.

If you are using ORM, then this is where you would put the code for ORM operations; if you are using microservices, then this is where you would put the code for other service requests.

service

This is the business logic layer, where all the business process code should be placed. This layer will determine what code is requested from the repo layer, whether to manipulate the database or to call other services; all business data calculations should also be placed here; the input accepted here should be that passed in by the controller.

api

This is the code that receives external requests, e.g. gin’s corresponding handler, gRPC, other REST API framework access layers, etc.

Interface-oriented programming

Except for the models layer, layers should interact with each other via interfaces, not implementations. If you want to call the repo layer with a service, then you should call the repo’s interface. Then when modifying the underlying implementation our base class in the upper layer does not need to be changed, we just need to replace the underlying implementation.

For example, if we want to look up all the articles, we can provide this interface in the repo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

package repo

import (
    "context"
    "my-clean-rchitecture/models"
    "time"
)

// IArticleRepo represent the article's repository contract
type IArticleRepo interface {
    Fetch(ctx context.Context, createdDate time.Time, num int) (res []models.Article, err error)
}

The implementation class of this interface can be changed according to the requirements, for example when we want mysql to be used as a stored query, then we just need to provide a base class like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

type mysqlArticleRepository struct {
    DB *gorm.DB
}

// NewMysqlArticleRepository will create an object that represent the article.Repository interface
func NewMysqlArticleRepository(DB *gorm.DB) IArticleRepo {
    return &mysqlArticleRepository{DB}
}

func (m *mysqlArticleRepository) Fetch(ctx context.Context, createdDate time.Time,
    num int) (res []models.Article, err error) {

    err = m.DB.WithContext(ctx).Model(&models.Article{}).
        Select("id,title,content, updated_at, created_at").
        Where("created_at > ?", createdDate).Limit(num).Find(&res).Error
    return
}

If we want to switch to MongoDB to implement our storage, we can simply define a structure to implement the IArticleRepo interface.

Then we can inject the corresponding repo implementation into the service level implementation as we need it, without changing the service level implementation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

type articleService struct {
    articleRepo repo.IArticleRepo
}

// NewArticleService will create new an articleUsecase object representation of domain.ArticleUsecase interface
func NewArticleService(a repo.IArticleRepo) IArticleService {
    return &articleService{
        articleRepo: a,
    }
}

// Fetch
func (a *articleService) Fetch(ctx context.Context, createdDate time.Time, num int) (res []models.Article, err error) {
    if num == 0 {
        num = 10
    }
    res, err = a.articleRepo.Fetch(ctx, createdDate, num)
    if err != nil {
        return nil, err
    }
    return
}

Dependency Injection DI

Dependency injection, or DI for short, used to be common in java projects, but many people in go say it’s not needed, but I think it’s still necessary in large software development, otherwise it can only be passed through global variables or method parameters.

As for what DI is, it is simply a dependent module that is injected into (i.e. passed as an argument to) a module when it is created. For a more in-depth understanding of what DI is, here are some more recommendations Dependency injection and Inversion of Control Containers and the Dependency Injection pattern.

There are two main inconveniences if you don’t use DI. One is that modifications to the underlying classes require modifications to the upper classes, and in large software development processes there are many base classes, so a single link can easily require dozens of files to be modified.

Because of the use of dependency injection, it is inevitable that a large number of new will be written during the initialisation process, for example, in our project we need this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

package main

import (
    "my-clean-rchitecture/api"
    "my-clean-rchitecture/api/handlers"
    "my-clean-rchitecture/app"
    "my-clean-rchitecture/repo"
    "my-clean-rchitecture/service"
)

func main() { 
    // 初始化db
    db := app.InitDB() 
    //初始化 repo
    repository := repo.NewMysqlArticleRepository(db)
    //初始化service
    articleService := service.NewArticleService(repository)
    //初始化api
    handler := handlers.NewArticleHandler(articleService)
    //初始化router
    router := api.NewRouter(handler)
    //初始化gin
    engine := app.NewGinEngine()
    //初始化server
    server := app.NewServer(engine, router)
    //启动
    server.Start()
}

So for such a piece of code, is there any way we don’t have to write it ourselves? Here we can use the power of frameworks to generate our injection code.

In go DI tools are not as convenient as in java, and the main technical frameworks are: wire, dig, fx, etc. Since wire uses code generation for injection, the performance is higher and it is a DI framework introduced by google, so we use wire for injection here.

The requirements for wire are simple, create a new wire.go file (the name of the file is optional) and create our initialisation functions. For example, if we want to create and initialise a server object, we can do so.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

//+build wireinject

package main

import (
    "github.com/google/wire"
    "my-clean-rchitecture/api"
    "my-clean-rchitecture/api/handlers"
    "my-clean-rchitecture/app"
    "my-clean-rchitecture/repo"
    "my-clean-rchitecture/service"
)

func InitServer() *app.Server {
    wire.Build(
        app.InitDB,
        repo.NewMysqlArticleRepository,
        service.NewArticleService,
        handlers.NewArticleHandler,
        api.NewRouter,
        app.NewServer,
        app.NewGinEngine)
    return &app.Server{}
}

Note that the annotation in the first line: +build wireinject, indicates that this is an injector.

In the function, we call wire.Build() to pass in the constructor for the type on which the Server is created. Executing the wire command after writing the wire.go file will automatically generate a wire_gen.go file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Code generated by Wire. DO NOT EDIT.

//go:generate go run github.com/google/wire/cmd/wire
//+build !wireinject

package main

import (
    "my-clean-rchitecture/api"
    "my-clean-rchitecture/api/handlers"
    "my-clean-rchitecture/app"
    "my-clean-rchitecture/repo"
    "my-clean-rchitecture/service"
)

// Injectors from wire.go:

func InitServer() *app.Server {
    engine := app.NewGinEngine()
    db := app.InitDB()
    iArticleRepo := repo.NewMysqlArticleRepository(db)
    iArticleService := service.NewArticleService(iArticleRepo)
    articleHandler := handlers.NewArticleHandler(iArticleService)
    router := api.NewRouter(articleHandler)
    server := app.NewServer(engine, router)
    return server
}

You can see that wire automatically generates the InitServer method for us, which initialises all the base classes in turn. After that, we can just call this InitServer in our main function.

1
2
3
4

func main() {
    server := InitServer()
    server.Start()
}

Testing

Having defined what each layer should do above, we should be able to test each layer individually, even if another layer does not exist.

  • models layer: for this layer it’s very simple, as it doesn’t depend on any other code, so it can be tested directly with go’s single test framework.
  • repo layer: for this layer, since we use a mysql database, we need to mock mysql so that we can test it even without connecting to mysql.
  • service layer: since the service layer depends on the repo layer, and since they are related through an interface, I use github.com/golang/mock/gomock to mock the repo layer.
  • The api layer: this layer depends on the service layer and is related through an interface, so you can also use gomock to mock the service layer. However, this is a little trickier because we are using gin for the access layer, so we need to simulate sending requests in a single test.

As we are mocking through github.com/golang/mock/gomock, we need to perform a bit of code generation and put the generated mock code into the mock package as follows

1
2
3

mockgen -destination .\mock\repo_mock.go -source .\repo\repo.go -package mock

mockgen -destination .\mock\service_mock.go -source .\service\service.go -package mock

These two commands will automatically generate the mock function for me through the interface.

repo layer testing

In the project, since we are using gorm as our orm library, we need to use github.com/DATA-DOG/go-sqlmock in conjunction with gorm to do the mock.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

func getSqlMock() (mock sqlmock.Sqlmock, gormDB *gorm.DB) {
    //创建sqlmock
    var err error
    var db *sql.DB
    db, mock, err = sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
    if err != nil {
        panic(err)
    }
    //结合gorm、sqlmock
    gormDB, err = gorm.Open(mysql.New(mysql.Config{
        SkipInitializeWithVersion: true,
        Conn:                      db,
    }), &gorm.Config{})
    if nil != err {
        log.Fatalf("Init DB with sqlmock failed, err %v", err)
    }
    return
}

func Test_mysqlArticleRepository_Fetch(t *testing.T) {
    createAt := time.Now()
    updateAt := time.Now()
    //id,title,content, updated_at, created_at
    var articles = []models.Article{
        {1, "test1", "content", updateAt, createAt},
        {2, "test2", "content2", updateAt, createAt},
    }

    limit := 2
    mock, db := getSqlMock()

    mock.ExpectQuery("SELECT id,title,content, updated_at, created_at FROM `articles` WHERE created_at > ? LIMIT 2").
        WithArgs(createAt).
        WillReturnRows(sqlmock.NewRows([]string{"id", "title", "content", "updated_at", "created_at"}).
            AddRow(articles[0].ID, articles[0].Title, articles[0].Content, articles[0].UpdatedAt, articles[0].CreatedAt).
            AddRow(articles[1].ID, articles[1].Title, articles[1].Content, articles[1].UpdatedAt, articles[1].CreatedAt))

    repository := NewMysqlArticleRepository(db)
    result, err := repository.Fetch(context.TODO(), createAt, limit)

    assert.Nil(t, err)
    assert.Equal(t, articles, result)
}

service layer testing

The main thing here is to use our gomock-generated code to mock the repo layer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

func Test_articleService_Fetch(t *testing.T) {
    ctl := gomock.NewController(t)
    defer ctl.Finish()
    now := time.Now()
    mockRepo := mock.NewMockIArticleRepo(ctl)

    gomock.InOrder(
        mockRepo.EXPECT().Fetch(context.TODO(), now, 10).Return(nil, nil),
    )

    service := NewArticleService(mockRepo)

    fetch, _ := service.Fetch(context.TODO(), now, 10)
    fmt.Println(fetch)
}

api layer testing

For this layer, we not only mock the service layer, but also send httptest to simulate the request being sent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

func TestArticleHandler_FetchArticle(t *testing.T) {

    ctl := gomock.NewController(t)
    defer ctl.Finish()
    createAt, _ := time.Parse("2006-01-02", "2021-12-26")
    mockService := mock.NewMockIArticleService(ctl)

    gomock.InOrder(
        mockService.EXPECT().Fetch(gomock.Any(), createAt, 10).Return(nil, nil),
    )

    article := NewArticleHandler(mockService)

    gin.SetMode(gin.TestMode)

    // Setup your router, just like you did in your main function, and
    // register your routes
    r := gin.Default()
    r.GET("/articles", article.FetchArticle)

    req, err := http.NewRequest(http.MethodGet, "/articles?num=10&create_date=2021-12-26", nil)
    if err != nil {
        t.Fatalf("Couldn't create request: %v\n", err)
    }

    w := httptest.NewRecorder()
    // Perform the request
    r.ServeHTTP(w, req)

    // Check to see if the response was what you expected
    if w.Code != http.StatusOK {
        t.Fatalf("Expected to get status %d but instead got %d\n", http.StatusOK, w.Code)
    }
}

Summary

The above is a little summary of the problems I found in the golang project, and I don’t care if it’s right, it’s a solution to some of our current problems. However, the project will always need to be refactored, so if there are problems next time, we can change them next time.