Forem: go-doudou

Build RAG by go-doudou And langchaingo

go-doudou — Sun, 04 May 2025 09:54:35 +0000

go-doudou

May 4 '25

go-doudou + langchaingo Microkernel Architecture RAG Large Language Model Knowledge Base Practice (Part 1)

Comments

10 min read

go-doudou + langchaingo Microkernel Architecture RAG Large Language Model Knowledge Base Practice (Part 1)

go-doudou — Sun, 04 May 2025 09:38:55 +0000

Photo by Christopher Gower on Unsplash

In modern microservice architecture design, modular and pluggable design patterns are increasingly favored by developers. go-doudou, as a domestic Go language microservice framework, provides excellent plugin mechanisms and modular architecture support. This article will explain in detail go-doudou's plugin mechanism and modular microkernel architecture implementation through a practical project based on RAG (Retrieval-Augmented Generation).

1. What are Plugin Mechanisms and Microkernel Architecture

Microkernel Architecture, also known as Plugin Architecture, is a design pattern that separates core system functions from extension functions. In this architecture:

Core System: Provides basic services and mechanisms for managing plugins
Plugin Modules: Independently developed, independently deployed, implementing specific functions

The advantages of this architecture include:

Highly Modular: Each plugin is an independent functional unit
Strong Extensibility: New functions can be added without modifying the core system
Low Coupling: Modules communicate through well-defined interfaces
Flexible Deployment: Plugins can be loaded as needed

2. Plugin Mechanism in go-doudou Framework

The go-doudou framework supports plugin mechanisms by implementing the ServicePlugin interface. Each service module is registered to the main application as a plugin, achieving decoupling between modules and the core system.

Let's first look at the core code in main/cmd/main.go of this project:

package main

import (
    "go-doudou-rag/toolkit/auth"
    "github.com/unionj-cloud/go-doudou/v2/framework/grpcx"
    "github.com/unionj-cloud/go-doudou/v2/framework/plugin"
    "github.com/unionj-cloud/go-doudou/v2/framework/rest"
    "github.com/unionj-cloud/toolkit/pipeconn"
    "github.com/unionj-cloud/toolkit/zlogger"
    "google.golang.org/grpc"

    _ "go-doudou-rag/module-auth/plugin"
    _ "go-doudou-rag/module-chat/plugin"
    _ "go-doudou-rag/module-knowledge/plugin"
)

func main() {
    srv := rest.NewRestServer()
    srv.Use(auth.Jwt)

    grpcServer := grpcx.NewGrpcServer(
        // GRPC configuration...
    )
    lis, dialCtx := pipeconn.NewPipeListener()
    plugins := plugin.GetServicePlugins()
    for _, key := range plugins.Keys() {
        value, _ := plugins.Get(key)
        value.Initialize(srv, grpcServer, dialCtx)
    }
    defer func() {
        if r := recover(); r != nil {
            zlogger.Info().Msgf("Recovered. Error: %v\n", r)
        }
        for _, key := range plugins.Keys() {
            value, _ := plugins.Get(key)
            value.Close()
        }
    }()
    go func() {
        grpcServer.RunWithPipe(lis)
    }()
    srv.AddRoutes(rest.DocRoutes(""))
    srv.Run()
}

This code demonstrates the implementation of go-doudou's microkernel architecture:

Anonymous import (_ "go-doudou-rag/module-xxx/plugin") of each module's plugin package
Get all registered service plugins plugin.GetServicePlugins()
Call the Initialize method of each plugin, passing in the REST server and gRPC server
Register resource cleanup functions, ensuring that the plugin's Close method is called when the program exits

3. Implementation and Registration of Plugins

Each module becomes a plugin by implementing the ServicePlugin interface. Taking the module-auth module as an example:

package plugin

import (
    "github.com/glebarez/sqlite"
    "github.com/unionj-cloud/go-doudou/v2/framework/grpcx"
    "github.com/unionj-cloud/go-doudou/v2/framework/plugin"
    "github.com/unionj-cloud/go-doudou/v2/framework/rest"
    "github.com/unionj-cloud/toolkit/pipeconn"
    "github.com/unionj-cloud/toolkit/stringutils"
    service "go-doudou-rag/module-auth"
    "go-doudou-rag/module-auth/config"
    "go-doudou-rag/module-auth/internal/dao"
    "go-doudou-rag/module-auth/internal/model"
    "go-doudou-rag/module-auth/transport/httpsrv"
    "google.golang.org/grpc"
    "gorm.io/gorm"
    "os"
)

var _ plugin.ServicePlugin = (*ModuleAuthPlugin)(nil)

type ModuleAuthPlugin struct {
    grpcConns []*grpc.ClientConn
}

func (receiver *ModuleAuthPlugin) Close() {
    for _, item := range receiver.grpcConns {
        item.Close()
    }
}

func (receiver *ModuleAuthPlugin) GoDoudouServicePlugin() {
}

func (receiver *ModuleAuthPlugin) GetName() string {
    name := os.Getenv("GDD_SERVICE_NAME")
    if stringutils.IsEmpty(name) {
        name = "cloud.unionj.ModuleAuth"
    }
    return name
}

func (receiver *ModuleAuthPlugin) Initialize(restServer *rest.RestServer, grpcServer *grpcx.GrpcServer, dialCtx pipeconn.DialContextFunc) {
    conf := config.LoadFromEnv()

    db, err := gorm.Open(sqlite.Open(conf.Db.Dsn), &gorm.Config{})
    if err != nil {
        panic("failed to connect database")
    }

    if err = db.AutoMigrate(&model.User{}); err != nil {
        panic(err)
    }

    dao.Use(db)
    dao.Init()

    svc := service.NewModuleAuth(conf)
    routes := httpsrv.Routes(httpsrv.NewModuleAuthHandler(svc))
    restServer.GroupRoutes("/moduleauth", routes)
    restServer.GroupRoutes("/moduleauth", rest.DocRoutes(service.Oas))
}

func init() {
    plugin.RegisterServicePlugin(&ModuleAuthPlugin{})
}

This plugin implements key interface methods:

GoDoudouServicePlugin(): Identifier interface method
GetName(): Returns the plugin name
Initialize(): Initializes the plugin, registers HTTP routes
Close(): Releases resources

Note especially the plugin.RegisterServicePlugin() in the init() function, which registers the plugin to the global plugin registry, allowing the main application to discover and load this plugin.

4. Communication Between Modules

In microkernel architecture, communication between modules is a key challenge. go-doudou provides multiple communication methods:

Direct Dependency Call: Modules can directly import interfaces from other modules
Dependency Injection: Implement dependency injection through the samber/do library

In the implementation of module-chat, we can see how to call the service of module-knowledge:

func (receiver *ModuleChatImpl) Chat(ctx context.Context, req dto.ChatRequest) (err error) {
    // ...code omitted...

    knowService := do.MustInvoke[know.ModuleKnowledge](nil)
    queryResults, err := knowService.GetQuery(ctx, kdto.QueryReq{
        Text: req.Prompt,
        Top:  10,
    })

    // ...code omitted...
}

module-knowledge registers services through dependency injection:

func init() {
    plugin.RegisterServicePlugin(&ModuleKnowledgePlugin{})

    do.Provide[service.ModuleKnowledge](nil, func(injector *do.Injector) (service.ModuleKnowledge, error) {
        conf := config.LoadFromEnv()

        db, err := gorm.Open(sqlite.Open(conf.Db.Dsn), &gorm.Config{})
        if err != nil {
            panic("failed to connect database")
        }

        if err = db.AutoMigrate(&model.File{}); err != nil {
            panic(err)
        }

        dao.Use(db)

        svc := service.NewModuleKnowledge(conf)
        return svc, nil
    })
}

5. Modular Design Practices

This project demonstrates best practices in modular design, with each module having a clear division of responsibilities:

Module-Auth: Responsible for user authentication
Module-Chat: Implements chat functionality, integrates large language models
Module-Knowledge: Knowledge base management, implements RAG retrieval

Each module follows a similar internal structure:

module-xxx/
  ├── cmd/              # Independent execution entry point
  ├── config/           # Module configuration
  ├── dto/              # Data Transfer Objects
  ├── internal/         # Internal implementation
  │   ├── dao/          # Data access
  │   └── model/        # Data models
  ├── plugin/           # Plugin implementation
  ├── transport/        # Transport layer
  │   └── httpsrv/      # HTTP service
  ├── svc.go            # Service interface definition
  └── svcimpl.go        # Service implementation

This structure ensures:

Separation of interface and implementation
Separation of concerns
Clear dependency relationships
Good encapsulation

6. Practical Case: RAG Chat System

This project implements a chat system based on RAG (Retrieval-Augmented Generation), with the overall process as follows:

Users authenticate through module-auth
After authentication, they can upload knowledge documents through module-knowledge
Users ask questions through module-chat, the system will:
- Retrieve relevant content from module-knowledge
- Call a large language model to generate answers
- Return results via SSE (Server-Sent Events) streaming

For example, the core processing logic in module-chat:

func (receiver *ModuleChatImpl) Chat(ctx context.Context, req dto.ChatRequest) (err error) {
    // ...set response headers...

    // Create LLM client
    llm, err := openai.New(
        openai.WithBaseURL(receiver.conf.Openai.BaseUrl),
        openai.WithToken(lo.Ternary(stringutils.IsNotEmpty(receiver.conf.Openai.Token), 
            receiver.conf.Openai.Token, os.Getenv("OPENAI_API_KEY"))),
        openai.WithEmbeddingModel(receiver.conf.Openai.EmbeddingModel),
        openai.WithModel(receiver.conf.Openai.Model),
    )

    // Retrieve relevant content from knowledge base
    knowService := do.MustInvoke[know.ModuleKnowledge](nil)
    queryResults, err := knowService.GetQuery(ctx, kdto.QueryReq{
        Text: req.Prompt,
        Top:  10,
    })

    // Filter results with high relevance
    queryResults = lo.Filter(queryResults, func(item kdto.QueryResult, index int) bool {
        return cast.ToFloat64(item.Similarity) >= 0.5
    })

    // Build prompt
    prompt := "Please answer the question based on the context information given below..."

    // Call LLM to generate answers and return them via streaming
    content := []llms.MessageContent{
        llms.TextParts(llms.ChatMessageTypeSystem, "You are a senior public policy researcher."),
        llms.TextParts(llms.ChatMessageTypeHuman, prompt),
    }

    _, err = llm.GenerateContent(ctx, content,
        llms.WithMaxTokens(4096),
        llms.WithTemperature(0.2),
        llms.WithStreamingFunc(func(ctx context.Context, chunk []byte) error {
            chunkResp := dto.ChatResponse{
                Content:   string(chunk),
                RequestID: requestID,
                Type:      "content",
            }
            return writeSSEMessage(w, flusher, chunkResp)
        }))

    return
}

7. System Startup and Usage Examples

7.1 Starting the System

Start this RAG system based on go-doudou plugin architecture with the following steps:

Clone the repository and enter the project directory

   git clone https://github.com/your-repo/go-doudou-rag.git
   cd go-doudou-rag

Install dependencies

   go mod tidy

Start the main application

   cd main/cmd
   go run main.go

After the system starts, all modules (auth, chat, knowledge) will be loaded as plugins, and their API endpoints will be registered to the main application.

7.2 Call Examples

The following shows how to use curl commands to send requests to the chat service for knowledge-based Q&A:

# Login
curl --location 'http://localhost:6060/moduleauth/login' \
--header 'Content-Type: application/json' \
--data '{
    "username": "admin",
    "password": "admin"
}'

# Upload PDF document
curl --location 'http://localhost:6060/moduleknowledge/upload' \
--header 'Authorization: Bearer <token obtained from login interface>' \
--form 'file=@"/Users/wubin1989/Downloads/杭州市人民政府印发关于进一步推动经济高质量发展若干政策的通知.pdf"'

# Chat
curl -w '\n' -N -X POST 'http://localhost:6060/modulechat/chat' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <token obtained from login interface>' \
--data '{
    "prompt": "What recent economic policies has Hangzhou introduced?"
}'

System response example:

First, the system returns the constructed prompt (including context information retrieved from the knowledge base):

Please answer the question based on the context information given below, the answer must be itemized for clarity. If you don't know, you can answer that you don't know, but don't make up answers:
1. — 1 —
Hangzhou Municipal People's Government Document
Hangzheng Function [2024] No. 16
Notice of Hangzhou Municipal People's Government on Issuing
Several Policies for Further Promoting High-Quality Economic Development
To all district, county (city) people's governments, municipal government departments, and directly affiliated units:
The "Several Policies for Further Promoting High-Quality Economic Development" is now issued to you, please organize implementation in conjunction with actual conditions.
Hangzhou Municipal People's Government
February 18, 2024
(This document is publicly released)
ZJAC00-2024-0001
2. — 2 —
Several Policies for Further Promoting High-Quality Economic Development
... [context partially omitted] ...

Then, the system generates a structured answer based on the retrieved context information:

According to the provided information, the Hangzhou Municipal People's Government has recently issued a series of policies to promote high-quality economic development, specifically including the following aspects:

1. **Expanding Effective Investment Policies**: Promoting the introduction of the "Hangzhou Municipal Government Investment Project Management Regulations", implementing the provincial "Thousand Projects Trillion Yuan" project for expanding effective investment, planning to complete investments of over 80 billion yuan in 2024, driving fixed asset investment growth by 3%.

2. **Stimulating Consumption Potential**: Implementing policies such as purchase tax exemptions for new energy vehicles, adding 3,000 charging facilities in public areas throughout the year, organizing over 500 consumption promotion activities, and holding more than 50 catering consumption promotion activities.

3. **Supporting High-Quality Enterprise Development**: Improving the gradient cultivation mechanism for quality enterprises, providing a one-time reward of up to 200,000 yuan for industrial enterprises that are first incorporated into statistics, and an additional one-time reward of up to 300,000 yuan for those that remain in the statistics for 3 consecutive years.

4. **Stabilizing Foreign Trade Development**: Organizing no less than 150 foreign trade groups throughout the year, participating in more than 100 overseas exhibitions, sending 3,000 enterprises to explore overseas markets, and increasing the upper limit of short-term export credit insurance premium subsidy ratio to 60% (upper limit increased to 65% for manufacturing enterprises).

5. **Building an International Exhibition City**: Successfully hosting the 3rd Global Digital Trade Expo with high quality, achieving five doublings, and attracting 30 exhibitions to settle in Hangzhou International Expo Center and Convention Center in 2024.

6. **Supporting Digital Trade Development**: Encouraging enterprises to conduct data export security assessments, participate in the formulation of various standards in the field of digital trade, and continuously promote innovation in service trade development.

7. **Leveraging E-commerce Advantages**: Promoting high-quality development of new e-commerce in Hangzhou, cumulatively building no less than 200 e-commerce live streaming "Common Prosperity Workshops" throughout the year, and deepening the construction of Hangzhou Cross-border E-commerce Comprehensive Pilot Zone.

8. **Strengthening Financial Support**: The municipal finance budget for 2024 allocates 600 million yuan to support expanding domestic demand and stimulating consumption, supporting foreign trade development, high-quality development of new e-commerce, cross-border e-commerce development, opening new international routes, etc.

7.3 Knowledge Base Retrieval Failure Handling Mechanism

An important feature of the RAG system is that it only answers questions based on information existing in the knowledge base. When a user's question has low or no relevance to documents in the knowledge base, the system will clearly inform the user that it cannot answer, rather than generating potentially inaccurate information. Here is an example:

curl -w '\n' -N -X POST 'http://localhost:6060/modulechat/chat' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3NDU4OTM4MTMsInVzZXJuYW1lIjoiYWRtaW4ifQ.EjxDfrMMHmOCvt557H8rd5sn9zX-uYOytw4OKH-jLJ8' \
--data '{
    "prompt": "What is the latest version of Java?"            
}'

System response:

I'm very sorry, no relevant information could be retrieved, unable to answer

In the code implementation, this mechanism is implemented as follows:

// Implementation from module-chat/svcimpl.go
queryResults = lo.Filter(queryResults, func(item kdto.QueryResult, index int) bool {
    return cast.ToFloat64(item.Similarity) >= 0.5
})

if len(queryResults) == 0 {
    zlogger.Error().Msgf("Knowledge not found, requestId: %s", requestID)
    chunk := dto.ChatResponse{
        Content:   "I'm very sorry, no relevant information could be retrieved, unable to answer",
        RequestID: requestID,
        Type:      "error",
    }
    writeSSEMessage(w, flusher, chunk)
    return
}

This design ensures that the system only answers questions for which it has a knowledge base, improving the reliability and accuracy of responses and avoiding the risk of generating false information. It is an important aspect of responsible AI application design, especially in fields requiring high accuracy such as policy consulting, legal advice, etc.

This example fully demonstrates the power of plugin architecture:

Module Collaboration: module-auth handles authentication, module-knowledge is responsible for knowledge retrieval, and module-chat integrates large language models to generate answers
Pluggability: Each module can be independently updated or replaced
Technical Decoupling: Each module can use different technology stacks and data storage methods

8. Summary and Outlook

This article detailed the plugin mechanism and modular pluggable microkernel architecture in the go-doudou framework through a practical RAG chat system case. We have seen that this architectural pattern not only provides good modularity and extensibility but also allows different parts of the system to work together in a loosely coupled manner, greatly improving development efficiency and system maintainability.

The plugin mechanism in the go-doudou framework provides developers with a concise yet powerful way to build modular applications through the ServicePlugin interface and dependency injection system. This approach is particularly suitable for team collaborative development of complex systems, where each team can focus on their domain module without needing to be overly concerned with implementation details of other modules.

However, understanding concepts and principles is just the first step; how to actually build such a system from scratch is what developers care about most. In the next article, "Plugin Mechanism in go-doudou Framework and Modular Pluggable Microkernel Architecture in Practice (Part 2)", we will provide a detailed practical guide, leading readers step by step to build a complete go-doudou microkernel architecture application from scratch. We will demonstrate how to use go-doudou CLI tools to create workspaces, define service interfaces, implement plugins, configure communication between modules, and other full-process operations through specific commands and code examples, helping developers quickly master the practical application methods of this powerful architectural pattern.

References

go-doudou official documentation: https://go-doudou.github.io/
Source code of this project: https://github.com/wubin1989/go-doudou-rag

Detailed Guide to go-doudou CLI Commands

go-doudou — Sun, 04 May 2025 09:36:30 +0000

go-doudou is a powerful Go language microservice development framework that provides rich command-line tools to help developers quickly build, deploy, and manage microservices. This article will detail the usage of various commands and subcommands of the go-doudou CLI tool, and explain them in conjunction with actual examples.

Installation and Upgrade

Installing go-doudou

For Go versions below 1.17:

go get -v github.com/unionj-cloud/go-doudou/v2@v2.5.8

For Go versions >= 1.17, it's recommended to use the following command to install the go-doudou command-line tool globally:

go install -v github.com/unionj-cloud/go-doudou/v2@v2.5.8

It's recommended to use the following command to download go-doudou as a project dependency:

go get -v -d github.com/unionj-cloud/go-doudou/v2@v2.5.8

::: tip
If you encounter a 410 Gone error, please execute the following command first, and then execute the above installation command:

export GOSUMDB=off

:::

Upgrading go-doudou

Execute the go-doudou version command to upgrade the globally installed go-doudou command-line tool:

go-doudou version

If a new version is detected, you will be prompted whether to upgrade. After selecting "Yes", the latest version will be automatically installed.

Command Overview

The basic usage of the go-doudou command-line tool is as follows:

go-doudou [flags]
go-doudou [command]

Main commands include:

svc: Generate or update services
- init: Initialize new project
- http: Generate HTTP routes and handlers
- client: Generate HTTP client code
- test: Generate integration test code
- grpc: Generate gRPC service code
- crud: Generate generic CRUD code from database
- run: Run the service
- push: Build Docker image and push
- deploy: Deploy the service to Kubernetes
- shutdown: Shut down the deployed service
completion: Generate autocompletion script for specified shell
enum: Generate functions for constants to implement IEnum interface
name: Bulk add or update JSON tags of struct fields
version: Show the version number of go-doudou
work: Build modular application

::: warning Note
Although the ddl command may still be visible in the help information, this command has been deprecated and is not recommended for use in new projects. Please use the svc crud command instead.
:::

You can view the help information by running go-doudou -h:

go-doudou -h

Example output:

go-doudou works like a scaffolding tool but more than that. 
it lets api providers design their apis and help them code less. 
it generates openapi 3.0 spec json document for frontend developers or other api consumers to understand what apis there, 
consumers can import it into postman to debug and test, or upload it into some code generators to download client sdk.
it provides some useful components and middleware for constructing microservice cluster like service register and discovering, 
load balancing and so on. it just begins, more features will come out soon.

Usage:
  go-doudou [flags]
  go-doudou [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  enum        Generate functions for constants to implement IEnum interface
  help        Help about any command
  name        bulk add or update json tag of struct fields
  svc         generate or update service
  version     Print the version number of go-doudou
  work        Build modular application

Flags:
  -h, --help      help for go-doudou
  -v, --version   version for go-doudou

Detailed Explanation of the completion Command

The completion command is used to generate an autocompletion script for the specified shell to improve the efficiency of using the go-doudou command line.

Basic Usage

go-doudou completion [command]

Subcommands

bash: Generate an autocompletion script for bash
fish: Generate an autocompletion script for fish
powershell: Generate an autocompletion script for powershell
zsh: Generate an autocompletion script for zsh

Examples

Generate bash autocompletion script:

go-doudou completion bash > ~/.bash_completion

Generate zsh autocompletion script:

go-doudou completion zsh > ~/.zsh_completion

Detailed Explanation of the svc Command

The svc command is the most commonly used command in the go-doudou command-line tool, used to generate or update service-related code. It contains multiple subcommands:

svc init

The svc init command is used to initialize a new go-doudou microservice project.

Basic Usage

go-doudou svc init [dir] [flags]

where [dir] is the name of the project directory to initialize.

Common Parameters

-m, --mod: Module name
--module: Initialize as a component of a modular application (boolean value). When set to true, go-doudou will automatically call go work use to add the component to the workspace.
-f, --file: Path or download link to an OpenAPI 3.0 or Swagger 2.0 specification JSON file
--case: Naming convention for protobuf message fields and JSON tags, supports "lowerCamel" and "snake" (default "lowerCamel")
-t, --type: Specify project type, value can be "grpc" or "rest" (default "grpc")
--grpc_gen_cmd: Command for generating gRPC service and message code (default uses protoc command)

Examples

Basic initialization:

go-doudou svc init myservice -m github.com/myorg/myservice

Initialize with MySQL database and generate gRPC code:

go-doudou svc init myservice --db_driver mysql --db_dsn "root:password@tcp(localhost:3306)/mydb?charset=utf8mb4&parseTime=True&loc=Local" --db_soft deleted_at --db_grpc

Initialize as a component of a modular application:

go-doudou svc init component-c -m my-workspace/component-c --module

When using the --module flag, go-doudou will automatically perform the following operations:

Create the necessary project structure
Automatically call go work use to add the newly created component to the workspace
Generate plugins and entry code for the modular application
Update the main application's import statements to automatically include the new component's plugins

svc http

The svc http command is used to generate HTTP routes and handlers.

Basic Usage

go-doudou svc http [flags]

Common Parameters

--handler: Whether to generate default handler implementations (boolean value)
-c, --client: Whether to generate default Go HTTP client code (boolean value)
-o, --omitempty: Whether to add omitempty to JSON tags in generated anonymous structs (boolean value)
--case: JSON tag naming convention applied to fields in anonymous structs in generated handlers, options are "lowerCamel" or "snake" (default "lowerCamel")
--doc: Whether to generate OpenAPI 3.0 JSON documentation (boolean value)
-e, --env: Base URL environment variable name
-r, --routePattern: Route pattern generation strategy, 0 means splitting each method of the service interface by slash / (after converting to snake_case), 1 means not splitting, just converting to lowercase
--allowGetWithReqBody: Whether to allow GET requests with request bodies (boolean value)

Examples

Generate HTTP routes and client code:

go-doudou svc http -c

Generate HTTP routes, handlers, and OpenAPI documentation:

go-doudou svc http --handler --doc

svc http client

svc http client is a subcommand of svc http, used to generate HTTP client code from an OpenAPI 3.0 specification JSON file.

Basic Usage

go-doudou svc http client [flags]

Common Parameters

-f, --file: Path or download link to an OpenAPI 3.0 or Swagger 2.0 specification JSON file
-e, --env: Base URL environment variable name
-p, --pkg: Client package name (default "client")
-o, --omit: Whether to add omitempty to JSON tags (boolean value)

Examples

Generate client code from an OpenAPI document:

go-doudou svc http client -f ./api-docs.json -e BASE_URL -p client

svc http test

svc http test is a subcommand of svc http, used to generate integration test code from a Postman Collection file.

Basic Usage

go-doudou svc http test [flags]

Common Parameters

--collection: Path to a Postman Collection v2.1 compatible file
--dotenv: Path to a dotenv format configuration file for integration tests only

Examples

Generate test code from a Postman Collection:

go-doudou svc http test --collection ./postman_collection.json --dotenv ./.env.test

svc grpc

The svc grpc command is used to generate gRPC service code.

Basic Usage

go-doudou svc grpc [flags]

Common Parameters

-o, --omitempty: Whether to add omitempty to JSON tags in generated anonymous structs (boolean value)
--case: protobuf message field naming strategy, supports "lowerCamel" and "snake" (default "lowerCamel")
--grpc_gen_cmd: Command for generating gRPC service and message code (default uses protoc command)
--http2grpc: Whether to generate RESTful API for gRPC service (boolean value)
--allow_get_body: Whether to allow GET requests with request bodies (boolean value)
--annotated_only: Whether to only generate gRPC API for methods with @grpc annotation (boolean value)

Examples

Generate basic gRPC service code:

go-doudou svc grpc

Generate gRPC service code and provide RESTful API proxy:

go-doudou svc grpc --http2grpc

Generate gRPC service code that only includes methods with @grpc annotation:

go-doudou svc grpc --annotated_only

svc crud

The svc crud command is used to generate generic CRUD code from a database. This command is the recommended choice to replace the deprecated ddl command.

Basic Usage

go-doudou svc crud [flags]

Common Parameters

--db_orm: Specify ORM, currently only supports gorm (default "gorm")
--db_driver: Database driver type, options are "mysql", "postgres", "sqlite", "sqlserver", "tidb"
--db_dsn: Database connection URL
--db_soft: Database soft delete column name (default "deleted_at")
--db_service: Generate gRPC or REST service, accepts values: grpc or rest
--db_gen_gen: Whether to generate gen.go file (boolean value)
--db_table_prefix: Table prefix or PostgreSQL schema name
--db_table_glob: For filtering tables with glob matching
--db_table_exclude_glob: For excluding tables with glob matching
--case: protobuf message field and JSON tag naming convention, supports "lowerCamel" and "snake" (default "lowerCamel")
--db_type_mapping: Specify custom column type to Go type mappings
--db_omitempty: Whether to add omitempty to JSON tags in generated model fields (boolean value)
--grpc_gen_cmd: Command for generating gRPC service and message code (default uses protoc command)

Examples

Generate CRUD code from a MySQL database:

go-doudou svc crud --db_driver mysql --db_dsn "root:password@tcp(localhost:3306)/mydb?charset=utf8mb4&parseTime=True&loc=Local" --db_soft deleted_at --db_service rest

Generate CRUD code from a PostgreSQL database, and specify schema:

go-doudou svc crud --db_driver postgres --db_dsn "host=localhost user=postgres password=postgres dbname=mydb port=5432 sslmode=disable" --db_table_prefix public --db_service grpc

Generate CRUD code for specific tables only:

go-doudou svc crud --db_driver mysql --db_dsn "root:password@tcp(localhost:3306)/mydb" --db_table_glob "user_*" --db_service rest

svc run

The svc run command is used to run a go-doudou service.

Basic Usage

go-doudou svc run [flags]

Common Parameters

-w, --watch: Enable watch mode, automatically restart the service when files change (boolean value)

Examples

Start the service:

go-doudou svc run

Start the service with watch mode enabled:

go-doudou svc run -w

svc push

The svc push command is used to build Docker images and push them to an image repository, while generating or updating K8s deployment YAML files.

Basic Usage

go-doudou svc push [flags]

Common Parameters

-r, --repo: Private Docker image repository
--pre: Image name prefix for building and pushing Docker images
--ver: Docker image version

Examples

Build an image and push it to a private repository:

go-doudou svc push -r myregistry.com/myuser

Build an image with a version tag:

go-doudou svc push -r myregistry.com/myuser --ver v1.0.0

svc deploy

The svc deploy command wraps the kubectl apply command, used to deploy a service to a Kubernetes cluster.

Basic Usage

go-doudou svc deploy [flags]

Common Parameters

-k, --k8sfile: Kubernetes YAML file for deploying the service

Examples

Deploy a service with default configuration:

go-doudou svc deploy

Deploy a service with a specified Kubernetes configuration file:

go-doudou svc deploy -k myservice_deployment.yaml

svc shutdown

The svc shutdown command wraps the kubectl delete command, used to shut down a deployed service.

Basic Usage

go-doudou svc shutdown [flags]

Common Parameters

-k, --k8sfile: Kubernetes YAML file for deploying the service

Examples

Shut down a service deployed with default configuration:

go-doudou svc shutdown

Shut down a service with a specified Kubernetes configuration file:

go-doudou svc shutdown -k myservice_deployment.yaml

Detailed Explanation of the name Command

The name command is used to bulk add or update JSON tags for struct fields.

Basic Usage

go-doudou name [flags]

Common Parameters

-f, --file: Go source file path
-c, --case: JSON tag naming convention, supports "lowerCamel", "snake", etc. (default "lowerCamel")
-s, --strategy: Naming strategy name, currently only supports "lowerCamel" and "snake" (default "lowerCamel")
-o, --omitempty: Whether to add omitempty marker (boolean value)
--form: Whether to add form tags for github.com/go-playground/form

Examples

Add snake_case JSON tags to fields in the User struct:

go-doudou name -f ./model/user.go -c snake -o

Generate both JSON and form tags:

go-doudou name -f ./model/user.go -c lowerCamel -o --form

Detailed Explanation of the enum Command

The enum command is used to generate functions implementing the IEnum interface for constants. This is useful for using enum types in Go.

Basic Usage

go-doudou enum [flags]

Common Parameters

-f, --file: Absolute path to a Go source file

Examples

Generate enum interface implementations for a file containing constant definitions:

go-doudou enum -f ./enum/status.go

Example of generated code (assuming status.go defines constants of type Status):

// Original file
type Status int

const (
    StatusPending Status = iota
    StatusActive
    StatusInactive
)

// Generated functions
func (s *Status) StringSetter(value string) {
    switch value {
    case "StatusPending":
        *s = StatusPending
    case "StatusActive":
        *s = StatusActive
    case "StatusInactive":
        *s = StatusInactive
    default:
        *s = StatusPending
    }
}

func (s *Status) StringGetter() string {
    switch *s {
    case StatusPending:
        return "StatusPending"
    case StatusActive:
        return "StatusActive"
    case StatusInactive:
        return "StatusInactive"
    default:
        return "StatusPending"
    }
}

Detailed Explanation of the version Command

The version command is used to display the version number of go-doudou and check if a new version is available.

Basic Usage

go-doudou version

Examples

go-doudou version

Example output:

go-doudou version v2.5.8

If an updated version is detected, you will be prompted whether to upgrade:

A new version is available: v2.5.9
Do you want to upgrade? [Y/n]

Detailed Explanation of the work Command

The work command is used to build modular applications, it creates a project structure with a workspace and a main entry module.

Basic Usage

go-doudou work [flags]
go-doudou work [command]

Subcommands

init: Initialize a workspace folder

work init

The work init command is used to initialize a workspace folder for developing modular applications.

Basic Usage

go-doudou work init [dir]

where [dir] is the path to the workspace directory to initialize. If not specified, the current directory is used.

Workspace Structure

After executing the work init command, go-doudou will create the following workspace structure:

workspace/              # Workspace root directory
├── go.work             # Go workspace file, automatically includes main module and other components
└── main/               # Main entry module directory
    ├── go.mod          # go.mod file for the main module
    ├── .env            # Environment variable configuration file
    └── cmd/            # Command directory
        └── main.go     # Main entry file, responsible for loading and running all components

When adding components using svc init --module, go-doudou will automatically call the go work use command to add the new component to the workspace, and automatically update the main/cmd/main.go file to import the new component's plugins.

Examples

Initialize the current directory as a workspace:

go-doudou work init

Specify a directory as a workspace:

go-doudou work init ./my-workspace

Practical Application Examples

1. Microservice Initialization and Development Process

Here is a complete microservice development process:

# Step 1: Initialize the project
go-doudou svc init myservice -m github.com/myorg/myservice

# Step 2: Edit the svc.go file to define the service interface
# Define the service interface in myservice/svc.go

# Step 3: Generate HTTP and gRPC service code
cd myservice
go-doudou svc http -c --doc
go-doudou svc grpc

# Step 4: Implement business logic
# Edit the svcimpl.go file

# Step 5: Run the service
go-doudou svc run

# Step 6: Build the image and deploy
go-doudou svc push -r myregistry.com/myuser
go-doudou svc deploy

2. Database Table-Based Microservice Generation

# Step 1: Initialize a project with database support
go-doudou svc init dbservice -m github.com/myorg/dbservice --db_driver mysql --db_dsn "root:password@tcp(localhost:3306)/mydb?charset=utf8mb4&parseTime=True&loc=Local" --db_soft deleted_at --db_grpc --db_rest

# Step 2: Run the service
cd dbservice
go-doudou svc run

3. Using the crud Command to Generate a CRUD Service from an Existing Database

# Step 1: Run the command in an existing project directory
cd myproject
go-doudou svc crud --db_driver postgres --db_dsn "host=localhost user=postgres password=postgres dbname=mydb" --db_service rest --db_soft deleted_at

# Step 2: Run the generated service
go-doudou svc run

4. Modular Application Development in a Workspace

# Step 1: Initialize the workspace
go-doudou work init my-workspace
cd my-workspace

# Step 2: Initialize modular components
# go-doudou will automatically execute "go work use" to add the component to the workspace
go-doudou svc init component-a -m my-workspace/component-a --module
go-doudou svc init component-b -m my-workspace/component-b --module

# Step 3: Define the service interface and generate code in each component
cd component-a
go-doudou svc http -c
go-doudou svc grpc

cd ../component-b
go-doudou svc http -c
go-doudou svc grpc

# Step 4: Start the main application - the main module will automatically import all components
cd ../main
go run cmd/main.go

How modular applications work:

Each component generates a plugin package during initialization, used to register itself to the main application
The main application (main module) automatically imports all component plugins and initializes them at runtime
When calling svc init --module, go-doudou automatically executes go work use to add the new component to the workspace
At the same time, it updates the main/cmd/main.go file, adding import statements for the new component's plugins

5. Implementing Enum Types

# Step 1: Define enum constants
# Define the following in a status.go file
type Status int

const (
    StatusPending Status = iota
    StatusActive
    StatusInactive
)

# Step 2: Generate enum interface implementation
go-doudou enum -f ./model/status.go

6. Generate Autocompletion Scripts

# Generate bash autocompletion script
go-doudou completion bash > ~/.bash_completion

# Generate zsh autocompletion script
go-doudou completion zsh > ~/.zsh_completion

Advanced Usage and Tips

1. Using Annotations to Control Interface Permissions

In the service interface, you can add annotations using special comments, such as:

// @role(ADMIN)
GetAdminData(ctx context.Context) (data string, err error)

Then check these annotations in middleware:

annotations := httpsrv.RouteAnnotationStore.GetParams(routeName, "@role")
if !sliceutils.StringContains(annotations, userRole) {
    // Deny access
}

2. Custom protoc Command

For complex gRPC services, you can customize the protoc command:

go-doudou svc grpc --grpc_gen_cmd "protoc --proto_path=. --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative --validate_out=lang=go,paths=source_relative:. transport/grpc/myservice.proto"

3. Environment Variables Affecting Service Behavior

go-doudou supports various environment variables to configure service behavior:

GDD_SERVICE_NAME: Service name
GDD_SERVICE_GROUP: Service group name
GDD_SERVICE_VERSION: Service version
GDD_WEIGHT: Service instance weight
GDD_REGISTER_HOST: Service registration host
GDD_HTTP_PORT: HTTP service port
GDD_GRPC_PORT: gRPC service port
GDD_LOG_LEVEL: Log level, optional values: "debug", "info", "warn", "error"
GDD_PROMETHEUS: Whether to enable Prometheus metrics collection

Example:

export GDD_SERVICE_NAME=myservice
export GDD_HTTP_PORT=8080
export GDD_LOG_LEVEL=debug
go-doudou svc run

4. Integration Testing Tips

Using the test code generated by the svc http test command, you can easily implement integration tests:

# Step 1: Generate test code from a Postman Collection
go-doudou svc http test --collection ./collection.json --dotenv ./.env.test

# Step 2: Run the tests
go test -v ./test/...

5. Modular Application Development Tips

For large projects, you can easily manage modular applications using the work command and the --module flag:

# Initialize workspace
go-doudou work init my-workspace
cd my-workspace

# Add multiple modules - go-doudou will automatically call go work use
go-doudou svc init api-gateway -m my-workspace/api-gateway --module
go-doudou svc init user-service -m my-workspace/user-service --module
go-doudou svc init product-service -m my-workspace/product-service --module

# Start the application (main module automatically imports and initializes all components)
cd main
go run cmd/main.go

Advantages of modular applications:

Clearer code organization, each component is maintained independently
Can develop and test each component independently
Shared dependencies are resolved through go.work, avoiding dependency conflicts
The main application automatically integrates all components, no need to manually write integration code
Suitable for the development and management of large microservice applications

Summary

The go-doudou command-line tool provides rich functionality to help developers quickly build, deploy, and manage microservices. Through the various commands and subcommands introduced in this article, you can easily complete the entire process from service initialization and code generation to deployment.

The biggest feature of go-doudou is simplifying the microservice development process, eliminating the need to write a lot of boilerplate code, allowing you to focus on implementing business logic. It supports the generation of RESTful API and gRPC services, as well as integration with databases, making it an ideal choice for building modern Go microservices.

In addition, go-doudou also provides powerful modular application development support. Through the work command and the --module flag, you can easily manage multi-module projects. go-doudou will automatically execute go work use to add components to the workspace, and automatically import and initialize all components in the main application, greatly simplifying the development and maintenance work of modular applications.

::: warning Important Note
Please note that the ddl command has been deprecated and is no longer recommended for use. If you need to generate code from a database or synchronize Go structs to a database, please use the svc crud command instead.
:::

I hope this article helps you understand and use the go-doudou CLI tool. For more details, please refer to the official documentation and example code repositories.

go-doudou series 02: How to develop gRPC service with go-doudou

go-doudou — Mon, 21 Nov 2022 00:44:07 +0000

Since v2.0.0 version, go-doudou has been supported to develop gRPC service. The approach or workflow is the same as previous versions that is define methods of a golang interface in svc.go file first, then let go-doudou generate as much as code for you. go-doudou has a smooth learning curve, which make it very friendly to novice, especially to developers with other programming language background like Java, Nodejs or Python. go-doudou is easy, but very productive.

Here I will use a small demo to show you how the developping workflow looks like including some best practices. We will implement a Largest Remainder Method gRPC service with go-doudou, and then we will see how to test it. Full code can be cloned from https://github.com/unionj-cloud/go-doudou-tutorials/tree/master/go-stats.

Preparation

Intall Go

You should install go v1.16 or above.

Install gRPC Compiler and Plugins

Install Compiler protoc

To install the Protobuf compiler protoc, please refer to official documentation, here are the installation commands for common operating systems:

Ubuntu system:

$ apt install -y protobuf-compiler
$ protoc --version # Make sure to install v3 and above

Mac system, you need to install Homebrew:

$ brew install protobuf
$ protoc --version # Make sure to install v3 and above

If Homebrew fails to be installed on Windows systems or Mac systems, you need to download the installation package from github, unzip it, and configure the environment variables yourself.

The latest protoc download address for Windows system: https://github.com/protocolbuffers/protobuf/releases/download/v21.7/protoc-21.7-win64.zip

Mac system Intel latest protoc download address: https://github.com/protocolbuffers/protobuf/releases/download/v21.7/protoc-21.7-osx-x86_64.zip

Please find other packages in github releases.

Install Plugins

Install the plugin:

$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2

Configure environment variables:

$ export PATH="$PATH:$(go env GOPATH)/bin"

Please go to https://grpc.io/docs/languages/go/quickstart/ to find the latest version number.

Install go-doudou

To install go-doudou CLI

If go version < 1.17,

go get -v github.com/unionj-cloud/go-doudou/v2@v2.0.3

If go version >= 1.17, recommend to use below command to install go-doudou cli globally

go install -v github.com/unionj-cloud/go-doudou/v2@v2.0.3

To download go-doudou as dependency for your module

go get -v -d github.com/unionj-cloud/go-doudou/v2@v2.0.3

If you meet 410 Gone error, try to run below command, then run install command again:

export GOSUMDB=off

After installation, if you meet go-doudou: command not found error, please configure $HOME/go/bin to ~/.bash_profile file, for example:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:/usr/local/go/bin
PATH=$PATH:$HOME/go/bin

export PATH

Verify go-doudou has been installed, run below command:

➜  go-doudou -v
go-doudou version v2.0.3

Initialize Project

Now we are ready to start coding. First we should initialize our project.

go-doudou svc init go-stats -m go-doudou-tutorials/go-stats

go-stats is project root folder path, go-doudou will create directories recursively if any directory not exists. go-doudou-tutorials/go-stats is module name, specified by -m flag.

go-doudou generated these folder and files as below.

➜  go-stats git:(master) ✗ tree -L 2
.
├── Dockerfile
├── go.mod
├── svc.go
└── vo
    └── vo.go

1 directory, 4 files

In svc.go file there has been declared an interface for you to define methods to map protobuf rpcs. vo package is for you to define structs to map protobuf messages.

Design Service

Then we have a look the svc.go file.

There are two //go:generate directives, line 12 is for RESTful service, line 13 is for gRPC service, for convenience. If you use goland (my favourite IDE), you can just single click the green arrow to run go-doudou command.

PageUsers is an example method, now we get rid of it and write our own.

We also need to define PercentageReqVo and PercentageRespVo in vo package. Note, struct type input parameters and result parameters here in each methods must be declared in vo package as go-doudou CLI only scan the package to fetch informations for generating protobuf messages.

go-doudou name in line 7 is a small CLI tool to create or update json tag of each fields in each structs, default is lower camel case.

Generate Code

Now we can run go-doudou svc grpc command to generate proto file, server and client stub code.

➜  go-stats git:(master) ✗ tree -L 3 -a
.
├── .dockerignore
├── .env
├── .gitignore
├── Dockerfile
├── cmd
│   └── main.go
├── config
│   └── config.go
├── db
│   └── db.go
├── go.mod
├── svc.go
├── svcimpl.go
├── transport
│   └── grpc
│       ├── annotation.go
│       ├── gostats.pb.go
│       ├── gostats.proto
│       └── gostats_grpc.pb.go
└── vo
    └── vo.go

13 directories, 16 files

We can see that there are some more folders and files generated.

.dockerignore : ignore **/*.local files
.env : dotenv configuration file
cmd : there is main.go file as entry point
config : mapping environment variables to config struct
db : for database connection, useless here
svcimpl.go : implement your own business logic
transport : proto file, server and client stub code

Before we dive into the code, we should run go mod tidy to download dependencies. Then we start the program to make sure everything is working correctly until now.

go-doudou is relying on go.uber.org/automaxprocs for effectly using resources, so you can see line 2.

Let's open transport/grpc/gostats.proto file and explain the code.

As we can see, all of the field names in each messages are lower camel case, which is not conventional snake case, because all our json tags in vo package are lower camel case. Of course you can use snake case both sides, but you must make them consistent with each other, otherwise you maybe have to convert protoc generated struct to vo struct manually (if you will use vo structs).

go-doudou will wrap all result parameters except error into one message like LargestRemainderRpcResponse when generating proto file and copy all comments into it.

go-doudou only supports protobuf v3 grammer.

Implement Stub

We will implement our business logic in svcimpl.go file now. Let's see what's in it now.

Code line 13 assigns a *GoStatsImpl type nil to interface type pb.GoStatsServiceServer to make sure pointer type GoStatsImpl is always implementing pb.GoStatsServiceServer interface.

We can add any fields to GoStatsImpl if necessary like external service client instance, database connection instance, any queue or pool instance, etc. There is a package level factory function NewGoStats for you to inject any dependencies and create a pointer type GoStatsImpl instance for later use.

Now let's implement LargestRemainderRpc method. Here is a piece of best practice that you don't have to implement pb.GoStatsServiceServer directly, you can choose to implement GoStats at first, then call each implemented GoStats methods in corresponding pb.GoStatsServiceServer method implementations as go-doudou doesn't support grpc-gateway and grpc-web, implementing GoStats first can let you reuse code to add RESTful endpoints when necessary in the future.

So we prefer to run go-doudou svc http -c to generate RESTful related code at first even if you don't need RESTful at the moment.

The current project structure looks like this.

➜  go-stats git:(master) ✗ tree -L 3   
.
├── Dockerfile
├── client
│   ├── client.go
│   ├── clientproxy.go
│   └── iclient.go
├── cmd
│   └── main.go
├── config
│   └── config.go
├── db
│   └── db.go
├── go.mod
├── go.sum
├── gostats_openapi3.go
├── gostats_openapi3.json
├── svc.go
├── svcimpl.go
├── transport
│   ├── grpc
│   │   ├── annotation.go
│   │   ├── gostats.pb.go
│   │   ├── gostats.proto
│   │   └── gostats_grpc.pb.go
│   └── httpsrv
│       ├── handler.go
│       ├── handlerimpl.go
│       └── middleware.go
└── vo
    └── vo.go

8 directories, 21 files

We can see go-doudou generated httpsrv package that is for implementing RESTful. We can simply remove it or leave it there. Then let's see what changed in svcimpl.go file.

We can see line 17 and line 36-42 are new code. Line 17 is for making sure pointer type GoStatsImpl is always implementing GoStats interface. Line 36-42 are stub code for us to finish.

go-doudou provides us a helper function LargestRemainder in github.com/unionj-cloud/go-doudou/v2/toolkit/numberutils package. We omit the algorithm explain here as it is not the concern.

Now we can implement LargestRemainderRpc method like this:

We don't need to worry about manually converting from protoc generated struct to vo struct as each fields in one has the same json tag with the other, so that we can use DeepCopy function in github.com/unionj-cloud/go-doudou/v2/toolkit/copier package to help us do the conversion.

Test Service

We start our service again. Don't forget to run go mod tidy as we generated new code and imported new dependencies while we were coding.

➜  go-stats git:(master) ✗ go mod tidy && go run cmd/main.go
2022/11/23 13:18:13 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
                           _                    _
                          | |                  | |
  __ _   ___   ______   __| |  ___   _   _   __| |  ___   _   _
 / _` | / _ \ |______| / _` | / _ \ | | | | / _` | / _ \ | | | |
| (_| || (_) |        | (_| || (_) || |_| || (_| || (_) || |_| |
 \__, | \___/          \__,_| \___/  \__,_| \__,_| \___/  \__,_|
  __/ |
 |___/
2022-11-23 13:18:13 INF ================ Registered Services ================
2022-11-23 13:18:13 INF +------------------------------------------+----------------------+
2022-11-23 13:18:13 INF |                 SERVICE                  |         RPC          |
2022-11-23 13:18:13 INF +------------------------------------------+----------------------+
2022-11-23 13:18:13 INF | go_stats.GoStatsService                  | LargestRemainderRpc  |
2022-11-23 13:18:13 INF | grpc.reflection.v1alpha.ServerReflection | ServerReflectionInfo |
2022-11-23 13:18:13 INF +------------------------------------------+----------------------+
2022-11-23 13:18:13 INF ===================================================
2022-11-23 13:18:13 INF Grpc server is listening at [::]:50051
2022-11-23 13:18:13 INF Grpc server started in 1.001238ms

Personally, I prefer to use evans to test gRPC.

➜  go-stats git:(master) ✗ evans -r repl -p 50051

  ______
 |  ____|
 | |__    __   __   __ _   _ __    ___
 |  __|   \ \ / /  / _. | | '_ \  / __|
 | |____   \ V /  | (_| | | | | | \__ \
 |______|   \_/    \__,_| |_| |_| |___/

 more expressive universal gRPC client


go_stats.GoStatsService@127.0.0.1:50051> show service
+----------------+---------------------+-----------------+-----------------------------+
|    SERVICE     |         RPC         |  REQUEST TYPE   |        RESPONSE TYPE        |
+----------------+---------------------+-----------------+-----------------------------+
| GoStatsService | LargestRemainderRpc | PercentageReqVo | LargestRemainderRpcResponse |
+----------------+---------------------+-----------------+-----------------------------+

go_stats.GoStatsService@127.0.0.1:50051> service GoStatsService

go_stats.GoStatsService@127.0.0.1:50051> call LargestRemainderRpc
<repeated> data::value (TYPE_INT32) => 20
<repeated> data::key (TYPE_STRING) => apple
<repeated> data::value (TYPE_INT32) => 30
<repeated> data::key (TYPE_STRING) => banana
<repeated> data::value (TYPE_INT32) => 40
<repeated> data::key (TYPE_STRING) => pear
<repeated> data::value (TYPE_INT32) => 
places (TYPE_INT32) => 2
{
  "data": [
    {
      "key": "apple",
      "percent": 22.22,
      "percentFormatted": "22.22%",
      "value": 20
    },
    {
      "key": "banana",
      "percent": 33.33,
      "percentFormatted": "33.33%",
      "value": 30
    },
    {
      "key": "pear",
      "percent": 44.45,
      "percentFormatted": "44.45%",
      "value": 40
    }
  ]
}

See, we input apple 20kg, banana 30kg, pear 40kg and we want 2 digits after dot, then we get expected result that 22.22 + 33.33 + 44.45 = 100.

Summary

In this tutorial, we learned basic skill about how to develop gRPC service with go-doudou microservice framework. go-doudou not only can help you build your gRPC service to implement your business logic, but also contains full of service government features to help you build entire microservice system. go-doudou is young, but very promising, hope more and more developers can join us to make contributions.

go-doudou series 01: How to develop a monolithic RESTful service with go-doudou

go-doudou — Mon, 24 Jan 2022 11:56:08 +0000

TODO
Learning Goals
Dev Requirements
Install go-doudou
Init Project
Define Apis
Generate Code
Start Service
Init Database
Generate Domain and CRUD Code
User SignUp
- Fix Domain
- PublicSignUp Method
- Test by Postman
User LogIn
- PublicLogIn Method
- Test by Postman
Upload Avatar
- Fix Domain
- Fix .env
- JWT Middleware
- UploadAvatar Method
Download Avatar
- GetPublicDownloadAvatar Method
User Pagination
- Import Test Data
- PageUsers Method
- Test by Postman
User Detail
- GetUser Method
- Test by Postman
Deploy Service
Source Code

In this tutorial, I will introduce go-doudou microservice framework to you. Go-doudou has built-in service register, discover and fault tolerance features based on gossip protocol, and it uses gorilla/mux as http router, and uses golang interface as IDL(Interface Definition Language). Go-doudou is an IDL compiler and server/client code generator tool at first, then it becomes a microservice framework. Its RESTful version is stable and production ready, while grpc version is in early development.

After brief introduction, now I will use a user management service as an example to demonstrate how to develop a monolithic service with go-doudou. This will be the first post about go-doudou, we will begin from monolithic and then go to microservice step by step.

TODO

User SignUp
User LogIn
User Detail
User Pagination
Upload Avatar
Download Avatar

Learning Goals

Protect user detail, user pagination and upload avatar endpoints by jwt token
Online OpenAPI 3.0 api documentation
Golang http client code
Fake response for frontend
Use built-in ddl tool for syncing structs to tables and generating structs from tables
Use generated dao layer code for single table CRUD

Dev Requirements

docker: recommend to install Desktop from official website
IDE：goland

Install go-doudou

If go version is below 1.16：

GO111MODULE=on  go get -v github.com/unionj-cloud/go-doudou@v0.9.6

if go version is 1.16 or above：

go get -v github.com/unionj-cloud/go-doudou@v0.9.6

Try to execute command go-doudou -v, if it printed below content, installation is successful：

➜  ~ go-doudou -v
go-doudou version v0.9.6

if it printed command not found, you should try to check GOPATH, and the go-doudou binary should be in $GOPATH/bin directory.

➜  ~ go env | grep GOPATH
GOPATH="/Users/wubin1989/go"

I can find go-doudou binary in /Users/wubin1989/go/bin directory for example. Then you should add one line export PATH=/Users/wubin1989/go/bin:$PATH to .zshrc or .bashrc file. And then execute source ~/.zshrc or source ~/.bashrc or open a new command line tab to make it work.

Init Project

Execute command in arbitrary directory. Here is in tutorials directory：

go-doudou svc init usersvc

cd usersvc, you will see below structure：

➜  tutorials ll
total 0
drwxr-xr-x  9 wubin1989  staff   288B 10 24 20:05 usersvc
➜  tutorials cd usersvc  
➜  usersvc git:(master) ✗ ll
total 24
-rw-r--r--  1 wubin1989  staff   707B 10 24 20:05 Dockerfile
-rw-r--r--  1 wubin1989  staff   439B 10 24 20:05 go.mod
-rw-r--r--  1 wubin1989  staff   247B 10 24 20:05 svc.go
drwxr-xr-x  3 wubin1989  staff    96B 10 24 20:05 vo

svc.go file：IDL file for defining api endpoints by interface methods
vo folder：define structs as api input and output parameters. Vo is named from view object.
Dockerfile：for building docker image

Define Apis

Have a look at svc.go file：

package service

import (
    "context"
    v3 "github.com/unionj-cloud/go-doudou/openapi/v3"
    "os"
    "usersvc/vo"
)

// Usersvc is user management service
// You should set Bearer Token header when you request protected endpoints such as user detail, user pagination and upload avatar.
// You can add doc for whole service here
type Usersvc interface {
    // PageUsers is user pagination api
    // demo how to define post request api which accepts application/json content-type
    PageUsers(ctx context.Context,
        // pagination parameter
        query vo.PageQuery) (
        // pagination result
        data vo.PageRet,
        // error
        err error)

    // GetUser is user detail api
    // demo how to define get http request with query string parameters
    GetUser(ctx context.Context,
        // user id
        userId int) (
        // user detail
        data vo.UserVo,
        // error
        err error)

    // PublicSignUp is user signup api
    // demo how to define post request api which accepts application/x-www-form-urlencoded content-type
    PublicSignUp(ctx context.Context,
        // username
        username string,
        // password
        password string,
        // image code
        code string,
    ) (
        // return OK if success
        data string, err error)

    // PublicLogIn is user login api
    // demo how to do authentication and issue token
    PublicLogIn(ctx context.Context,
        // username
        username string,
        // password
        password string) (
        // token
        data string, err error)

    // UploadAvatar is avatar upload api
    // demo how to define file upload api
    // NOTE: there must be at least one []*v3.FileModel or *v3.FileModel input parameter
    UploadAvatar(ctx context.Context,
        // user avatar
        avatar *v3.FileModel) (
        // return OK if success
        data string, err error)

    // GetPublicDownloadAvatar is avatar download api
    // demo how to define file download api
    // NOTE: there must be one and at most one *os.File output parameter
    GetPublicDownloadAvatar(ctx context.Context,
        // user id
        userId int) (
        // avatar file
        data *os.File, err error)
}

All comments in above code will be displayed in online OpenAPI 3.0 documentation, so recommend to add comments to let your clients use your apis more easily. For simplicity, no support for fancy symbol like @, only support golang // syntax.

Generate Code

Execute below command to generate server and client code:

go-doudou svc http --handler -c go --doc

Explain：

--handler: to generate http handler implementation
-c：to generate client http request code. Currently only support go.
--doc：to generate OpenAPI 3.0 doc file in json format This is the most used command for me. Recommend to all go-doudou users. When you made some changes in svc.go file, execute this command, and you will get incrementally generated new code. The rule is:
handler.go file, *_openapi3.json file and client.go file will always be fully overwritten. So you should not edit these files manually.
handlerimpl.go file, svcimpl.go file and clientproxy.go file will always be incrementally appended new code. Existing code will not be changed by go-doudou. So you can edit these files to fit your needs and still get all benefits from go-doudou.
Other files will be skipped if already exist.

You'd better execute below command to make sure all dependencies have been downloaded:

go mod tidy

Let's see project structure now:

➜  usersvc git:(master) ✗ ll
total 296
-rw-r--r--  1 wubin1989  staff   707B 10 24 20:05 Dockerfile
drwxr-xr-x  3 wubin1989  staff    96B 10 24 23:10 client
drwxr-xr-x  3 wubin1989  staff    96B 10 24 23:10 cmd
drwxr-xr-x  3 wubin1989  staff    96B 10 24 23:10 config
drwxr-xr-x  3 wubin1989  staff    96B 10 24 23:10 db
-rw-r--r--  1 wubin1989  staff   514B 10 24 23:10 go.mod
-rw-r--r--  1 wubin1989  staff   115K 10 24 23:10 go.sum
-rw-r--r--  1 wubin1989  staff   1.7K 10 24 23:21 svc.go
-rw-r--r--  1 wubin1989  staff   1.6K 10 25 09:18 svcimpl.go
drwxr-xr-x  3 wubin1989  staff    96B 10 24 23:10 transport
-rwxr-xr-x  1 wubin1989  staff   5.9K 10 25 09:18 usersvc_openapi3.go
-rwxr-xr-x  1 wubin1989  staff   5.7K 10 25 09:18 usersvc_openapi3.json
drwxr-xr-x  3 wubin1989  staff    96B 10 24 23:07 vo

client folder: generated go client code
cmd folder: main.go file in it for starting the service
config folder: used for populating configs
db folder: used for connecting database
svcimpl.go file: there are mock implementations initially to return fake response data for clients. You should replace them with your own implementation later.
transport folder：there are http handlers in it.
usersvc_openapi3.go file：used for online doc
usersvc_openapi3.json file：used for online doc

Start Service

go-doudou svc run

You will see:

➜  usersvc git:(master) ✗ go-doudou svc run
INFO[2022-01-23 15:55:07] Initializing logging reporter                
INFO[2022-01-23 15:55:07] ================ Registered Routes ================ 
INFO[2022-01-23 15:55:07] +----------------------+--------+-------------------------+ 
INFO[2022-01-23 15:55:07] |         NAME         | METHOD |         PATTERN         | 
INFO[2022-01-23 15:55:07] +----------------------+--------+-------------------------+ 
INFO[2022-01-23 15:55:07] | PageUsers            | POST   | /page/users             | 
INFO[2022-01-23 15:55:07] | User                 | GET    | /user                   | 
INFO[2022-01-23 15:55:07] | PublicSignUp         | POST   | /public/sign/up         | 
INFO[2022-01-23 15:55:07] | PublicLogIn          | POST   | /public/log/in          | 
INFO[2022-01-23 15:55:07] | UploadAvatar         | POST   | /upload/avatar          | 
INFO[2022-01-23 15:55:07] | PublicDownloadAvatar | GET    | /public/download/avatar | 
INFO[2022-01-23 15:55:07] | GetDoc               | GET    | /go-doudou/doc          | 
INFO[2022-01-23 15:55:07] | GetOpenAPI           | GET    | /go-doudou/openapi.json | 
INFO[2022-01-23 15:55:07] | Prometheus           | GET    | /go-doudou/prometheus   | 
INFO[2022-01-23 15:55:07] | GetRegistry          | GET    | /go-doudou/registry     | 
INFO[2022-01-23 15:55:07] +----------------------+--------+-------------------------+ 
INFO[2022-01-23 15:55:07] =================================================== 
INFO[2022-01-23 15:55:07] Started in 233.424µs                        
INFO[2022-01-23 15:55:07] Http server is listening on :6060

When you see Http server is listening on :6060, it means service has been started and we also have a mock server. For example, we can send a request to /user api, to see what will be sent back(I use httpie):

➜  usersvc git:(master) ✗ http http://localhost:6060/user
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 109
Content-Type: application/json; charset=UTF-8
Date: Sun, 23 Jan 2022 07:59:28 GMT
Vary: Accept-Encoding

{
    "data": {
        "Dept": "ZkkCmcLU",
        "Id": -1941954111002502016,
        "Name": "aiMtQ",
        "Phone": "XMAqXf"
    }
}

You may notice that all attribute names are capitalised, and that is not what you want. There is one line go generate command in vo.go file:

//go:generate go-doudou name --file $GOFILE

The command uses a built-in small tool called name. It can add or modify json tag to every exported field of each struct in a file. Default mode is camelcase, and it also supports snakecase.

go generate ./...

Then restart the service, resend a request to /user api, you can see it works.

➜  usersvc git:(master) ✗ http http://localhost:6060/user
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 114
Content-Type: application/json; charset=UTF-8
Date: Sun, 23 Jan 2022 08:00:40 GMT
Vary: Accept-Encoding

{
    "data": {
        "dept": "wGAEEeveHp",
        "id": -816946940349962228,
        "name": "hquwOKl",
        "phone": "AriWmKYB"
    }
}

If you want to know more about name command, please refer to README.md.
At the moment, as json tag of each exported field has been changed, you should regenerate OpenAPI 3.0 doc json file:

go-doudou svc http --doc

Then restart the service, open browser and navigate to http://localhost:6060/go-doudou/doc, input http basic username admin, password admin to see online doc.

Init Database

For supporting utf8/utf8mb4, you should create my/custom.cnf file in project root path and paste below content:

[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4

[mysqld]
character_set_server=utf8mb4
collation-server=utf8mb4_general_ci
default-authentication-plugin=mysql_native_password
init_connect='SET NAMES utf8mb4'

Then create sqlscripts/init.sql file for initialising database, paste below sqls：

CREATE SCHEMA `tutorial` DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci;

CREATE TABLE `tutorial`.`t_user`
(
    `id`        INT         NOT NULL AUTO_INCREMENT,
    `username`  VARCHAR(45) NOT NULL COMMENT 'username',
    `password`  VARCHAR(60) NOT NULL COMMENT 'password',
    `name`      VARCHAR(45) NOT NULL COMMENT 'real name',
    `phone`     VARCHAR(45) NOT NULL COMMENT 'phone number',
    `dept`      VARCHAR(45) NOT NULL COMMENT 'department',
    `create_at` DATETIME    NULL DEFAULT current_timestamp,
    `update_at` DATETIME    NULL DEFAULT current_timestamp on update current_timestamp,
    `delete_at` DATETIME    NULL,
    PRIMARY KEY (`id`)
);

Create docker-compose.yml file, paste below content:

version: '3.9'

services:
  db:
    container_name: db
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 1234
    ports:
      - 3306:3306
    volumes:
      - $PWD/my:/etc/mysql/conf.d
      - $PWD/sqlscripts:/docker-entrypoint-initdb.d
    networks:
      - tutorial

networks:
  tutorial:
    driver: bridge

Execute docker compose command to start mysql docker container:

docker-compose -f docker-compose.yml up -d

Execute docker ps to make sure it started.

➜  usersvc git:(master) ✗ docker ps        
CONTAINER ID   IMAGE       COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
df6af6362c41   mysql:5.7   "docker-entrypoint.s…"   13 minutes ago   Up 13 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   db

Generate Domain and CRUD Code

As our mysql schema name is tutorial, so we should change the value of environment variable DB_SCHEMA to tutorial in .env file.

DB_SCHEMA=tutorial

Execute below command to generate domain and dao layer(CRUD) code:

go-doudou ddl -r --dao --pre=t_

Flag explain：

-r：short for reverse, generate structs from tables
--dao：generate dao layer code, only support single table CRUD operations based on sqlx
--pre：table name has prefix t_ At the moment, you can see two more folders:

More about the built-in ddl tool, please refer to README.md
Here we should see what CRUD operation methods in dao/base.go file, we will use them to implement our business logic.

package dao

import (
    "context"
    "github.com/unionj-cloud/go-doudou/ddl/query"
)

type Base interface {
    Insert(ctx context.Context, data interface{}) (int64, error)
    Upsert(ctx context.Context, data interface{}) (int64, error)
    UpsertNoneZero(ctx context.Context, data interface{}) (int64, error)
    DeleteMany(ctx context.Context, where query.Q) (int64, error)
    Update(ctx context.Context, data interface{}) (int64, error)
    UpdateNoneZero(ctx context.Context, data interface{}) (int64, error)
    UpdateMany(ctx context.Context, data interface{}, where query.Q) (int64, error)
    UpdateManyNoneZero(ctx context.Context, data interface{}, where query.Q) (int64, error)
    Get(ctx context.Context, id interface{}) (interface{}, error)
    SelectMany(ctx context.Context, where ...query.Q) (interface{}, error)
    CountMany(ctx context.Context, where ...query.Q) (int, error)
    PageMany(ctx context.Context, page query.Page, where ...query.Q) (query.PageRet, error)
}

We will add *sqlx.DB instance as field to UsersvcImpl struct in svcimpl.go file.

type UsersvcImpl struct {
    conf *config.Config
    db   *sqlx.DB
}

Then make a small fix in factory function NewUsersvc.

func NewUsersvc(conf *config.Config, db *sqlx.DB) Usersvc {
    return &UsersvcImpl{
        conf,
        db,
    }
}

In our main function, it has already injected *sqlx.DB instance conn into NewUsersvc.

svc := service.NewUsersvc(conf, conn)

Afterwards, we can use db field of UsersvcImpl for CRUD operations.

User SignUp

Fix Domain

Normally username should be unique, so we should fix domain/user.go to add unique constraint to User struct

Username string     `dd:"type:varchar(45);extra:comment 'username';unique"`

Here we can see we add ;unique to dd tag.
Then execute go-doudou ddl command:

go-doudou ddl --pre=t_

There is no -r flag because we need to sync User struct to t_user table.

PublicSignUp Method

We should add CheckUsernameExists method to UserDao interface for checking if the username has already been used.

package dao

import "context"

type UserDao interface {
    Base
    CheckUsernameExists(ctx context.Context, username string) (bool, error)
}

Then we create a new file dao/userdaoimplext.go to add extra method implementations.

package dao

import (
    "context"
    "github.com/unionj-cloud/go-doudou/ddl/query"
    "usersvc/domain"
)

func (receiver UserDaoImpl) CheckUsernameExists(ctx context.Context, username string) (bool, error) {
    many, err := receiver.SelectMany(ctx, query.C().Col("username").Eq(username))
    if err != nil {
        return false, err
    }
    users := many.([]domain.User)
    if len(users) > 0 {
        return true, nil
    }
    return false, nil
}

In this way, we can fulfill more complex needs by adding custom CRUD operation methods to generated dao layer interface. If you add or delete fields of a domain struct, you just need to remove *daosql.go file, and generate new one by executing command go-doudou ddl --dao --pre=t_, existing dao layer files won't be changed anything.

Then we implement PublicSignUp method.

func (receiver *UsersvcImpl) PublicSignUp(ctx context.Context, username string, password string, code string) (data string, err error) {
    hashPassword, _ := lib.HashPassword(password)
    userDao := dao.NewUserDao(receiver.db)
    var exists bool
    exists, err = userDao.CheckUsernameExists(ctx, username)
    if err != nil {
        panic(err)
    }
    if exists {
        panic(lib.ErrUsernameExists)
    }
    _, err = userDao.Insert(ctx, domain.User{
        Username: username,
        Password: hashPassword,
    })
    if err != nil {
        panic(err)
    }
    return "OK", nil
}

If error ocurred, you can simply let program panic, or return "", lib.ErrUsernameExists because we add a recover middleware ddhttp.Recover. It can let our service recover and send error message to clients. If any non-nil error returned, http status code is 500 by default. If you don't like it, you can change generated http handler code manually. Remember existing code in handlerimpl.go file won't be changed when execute go-doudou command, only be incrementally appended new code.

Test by Postman

User LogIn

PublicLogIn Method

func (receiver *UsersvcImpl) PublicLogIn(ctx context.Context, username string, password string) (data string, err error) {
    userDao := dao.NewUserDao(receiver.db)
    many, err := userDao.SelectMany(ctx, query.C().Col("username").Eq(username).And(query.C().Col("delete_at").IsNull()))
    if err != nil {
        return "", err
    }
    users := many.([]domain.User)
    if len(users) == 0 || !lib.CheckPasswordHash(password, users[0].Password) {
        panic(lib.ErrUsernameOrPasswordIncorrect)
    }
    now := time.Now()
    token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.MapClaims{
        "userId": users[0].Id,
        "exp":    now.Add(10 * time.Minute).Unix(),
        //"iat":    now.Unix(),
        //"nbf":    now.Unix(),
    })
    return token.SignedString(receiver.conf.JWTConf.Secret)
}

The code logic is query user record from database by input parameter username, if not found, return Incorrect username or password error, if password was correct, issue token. The jwt library used here is golang-jwt/jwt

Test by Postman

Upload Avatar

Fix Domain

We add avatar field to user struct now:

Avatar   string     `dd:"type:varchar(255);extra:comment 'user avatar'"`

Before continue, we should remove dao/userdaosql.go file first, then execute ddl command:

go-doudou ddl --dao --pre=t_

If multiple domain structs were changed, we can use wildcard to delete all *sql.go files in order to generate new ones.

rm -rf dao/*sql.go

Fix .env

Add three lines. JWT_ prefixed ones are jwt related configs, while Biz_ prefixed ones are business related configs.

JWT_SECRET=secret
JWT_IGNORE_URL=/public/sign/up,/public/log/in,/public/get/download/avatar,/public/**
BIZ_OUTPUT=out

You can simply set JWT_IGNORE_URL to /public/** only. I add so many values here to show what kind of values can be used.

At the same time, config/config.go should be changed, too. Of course, you can directly use os.Getenv.

package config

import (
    "github.com/kelseyhightower/envconfig"
    "github.com/sirupsen/logrus"
)

type Config struct {
    DbConf  DbConfig
    JWTConf JWTConf
    BizConf BizConf
}

type BizConf struct {
    Output string
}

type JWTConf struct {
    Secret    []byte
    IgnoreUrl []string `split_words:"true"`
}

type DbConfig struct {
    Driver  string `default:"mysql"`
    Host    string `default:"localhost"`
    Port    string `default:"3306"`
    User    string
    Passwd  string
    Schema  string
    Charset string `default:"utf8mb4"`
}

func LoadFromEnv() *Config {
    var dbconf DbConfig
    err := envconfig.Process("db", &dbconf)
    if err != nil {
        logrus.Panicln("Error processing env", err)
    }
    var jwtConf JWTConf
    err = envconfig.Process("jwt", &jwtConf)
    if err != nil {
        logrus.Panicln("Error processing env", err)
    }
    var bizConf BizConf
    err = envconfig.Process("biz", &bizConf)
    if err != nil {
        logrus.Panicln("Error processing env", err)
    }
    return &Config{
        dbconf,
        jwtConf,
        bizConf,
    }
}

JWT Middleware

As go-doudou is relying on gorilla/mux, so if you can write middlewares for gorilla/mux, you can also write ones for go-doudou.

package middleware

import (
    "context"
    "fmt"
    "github.com/dgrijalva/jwt-go"
    "github.com/gobwas/glob"
    "net/http"
    "os"
    "strings"
)

type ctxKey int

const userIdKey ctxKey = ctxKey(0)

func NewContext(ctx context.Context, id int) context.Context {
    return context.WithValue(ctx, userIdKey, id)
}

func FromContext(ctx context.Context) (int, bool) {
    userId, ok := ctx.Value(userIdKey).(int)
    return userId, ok
}

func Jwt(g glob.Glob) func(inner http.Handler) http.Handler {
    return func(inner http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if g.Match(r.RequestURI) {
                inner.ServeHTTP(w, r)
                return
            }
            authHeader := r.Header.Get("Authorization")
            tokenString := strings.TrimSpace(strings.TrimPrefix(authHeader, "Bearer "))

            token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
                if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
                    return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
                }
                return []byte(os.Getenv("BIZ_JWT_SECRET")), nil
            })
            if err != nil || !token.Valid {
                w.WriteHeader(401)
                w.Write([]byte("Unauthorised.\n"))
                return
            }

            claims := token.Claims.(jwt.MapClaims)
            if userId, exists := claims["userId"]; !exists {
                w.WriteHeader(401)
                w.Write([]byte("Unauthorised.\n"))
                return
            } else {
                inner.ServeHTTP(w, r.WithContext(NewContext(r.Context(), int(userId.(float64)))))
            }
        })
    }
}

Then you can use it like this srv.AddMiddleware(middleware.Jwt(glob.MustCompile(fmt.Sprintf("{%s}", conf.BizConf.JwtIgnoreUrl)))).

UploadAvatar Method

func (receiver *UsersvcImpl) UploadAvatar(ctx context.Context, avatar *v3.FileModel) (data string, err error) {
    defer avatar.Close()
    _ = os.MkdirAll(receiver.conf.BizConf.Output, os.ModePerm)
    out := filepath.Join(receiver.conf.BizConf.Output, avatar.Filename)
    var f *os.File
    f, err = os.OpenFile(out, os.O_WRONLY|os.O_CREATE, os.ModePerm)
    if err != nil {
        panic(err)
    }
    defer f.Close()
    _, err = io.Copy(f, avatar.Reader)
    if err != nil {
        panic(err)
    }
    userId, _ := middleware.FromContext(ctx)
    userDao := dao.NewUserDao(receiver.db)
    _, err = userDao.UpdateNoneZero(ctx, domain.User{
        Id:     userId,
        Avatar: out,
    })
    if err != nil {
        panic(err)
    }
    return "OK", nil
}

NOTE: don't forget to add defer avatar.Close() to release resource.

Download Avatar

GetPublicDownloadAvatar Method

func (receiver *UsersvcImpl) GetPublicDownloadAvatar(ctx context.Context, userId int) (data *os.File, err error) {
    userDao := dao.NewUserDao(receiver.db)
    var get interface{}
    get, err = userDao.Get(ctx, userId)
    if err != nil {
        panic(err)
    }
    return os.Open(get.(domain.User).Avatar)
}

User Pagination

Import Test Data

INSERT INTO tutorial.t_user (id, username, password, name, phone, dept, create_at, update_at, delete_at, avatar) VALUES (2, 'crazyboy', '$2a$14$VaQLa/GbLAhRZvvTlgE8OOQgsBY4RDAJC5jkz13kjP9RlntdKBZVW', 'John Snow', '13552053960', 'IT dept.', '2021-12-28 06:41:00', '2021-12-28 14:59:20', null, 'out/wolf-wolves-snow-wolf-landscape-985ca149f06cd03b9f0ed8dfe326afdb.jpg');
INSERT INTO tutorial.t_user (id, username, password, name, phone, dept, create_at, update_at, delete_at, avatar) VALUES (4, 'david', '$2a$14$AKCs.u9vFUOCe5VwcmdfwOAkeiDtQYEgIB/nSU8/eemYwd91.qU.i', 'David Li', '13552053961', 'Admin dept.', '2021-12-28 12:12:32', '2021-12-28 14:59:20', null, '');
INSERT INTO tutorial.t_user (id, username, password, name, phone, dept, create_at, update_at, delete_at, avatar) VALUES (5, 'lucy', '$2a$14$n0.l54axUqnKGagylQLu7ee.yDrtLubxzM1qmOaHK9Ft2P09YtQUS', 'Lucy Zhang', '13552053962', 'Sales dept.', '2021-12-28 12:13:17', '2021-12-28 14:59:20', null, '');
INSERT INTO tutorial.t_user (id, username, password, name, phone, dept, create_at, update_at, delete_at, avatar) VALUES (6, 'jack', '$2a$14$jFCwiZHcD7.DL/teao.Dl.HAFwk8wM2f1riH1fG2f52WYKqSiGZlC', 'Jack Chen', '', 'CEO Office', '2021-12-28 12:14:19', '2021-12-28 14:59:20', null, '');

PageUsers Method

func (receiver *UsersvcImpl) PageUsers(ctx context.Context, pageQuery vo.PageQuery) (data vo.PageRet, err error) {
    userDao := dao.NewUserDao(receiver.db)
    var q query.Q
    q = query.C().Col("delete_at").IsNull()
    if stringutils.IsNotEmpty(pageQuery.Filter.Name) {
        q = q.And(query.C().Col("name").Like(fmt.Sprintf(`%s%%`, pageQuery.Filter.Name)))
    }
    if stringutils.IsNotEmpty(pageQuery.Filter.Dept) {
        q = q.And(query.C().Col("dept").Eq(pageQuery.Filter.Dept))
    }
    var page query.Page
    if len(pageQuery.Page.Orders) > 0 {
        for _, item := range pageQuery.Page.Orders {
            page = page.Order(query.Order{
                Col:  item.Col,
                Sort: sortenum.Sort(item.Sort),
            })
        }
    }
    if pageQuery.Page.PageNo == 0 {
        pageQuery.Page.PageNo = 1
    }
    page = page.Limit((pageQuery.Page.PageNo-1)*pageQuery.Page.Size, pageQuery.Page.Size)
    var ret query.PageRet
    ret, err = userDao.PageMany(ctx, page, q)
    if err != nil {
        panic(err)
    }
    var items []vo.UserVo
    for _, item := range ret.Items.([]domain.User) {
        var userVo vo.UserVo
        _ = copier.DeepCopy(item, &userVo)
        items = append(items, userVo)
    }
    data = vo.PageRet{
        Items:    items,
        PageNo:   ret.PageNo,
        PageSize: ret.PageSize,
        Total:    ret.Total,
        HasNext:  ret.HasNext,
    }
    return data, nil
}

Test by Postman

User Detail

GetUser Method

func (receiver *UsersvcImpl) GetUser(ctx context.Context, userId int) (data vo.UserVo, err error) {
    userDao := dao.NewUserDao(receiver.db)
    var get interface{}
    get, err = userDao.Get(ctx, userId)
    if err != nil {
        if errors.Is(err, sql.ErrNoRows) {
            return vo.UserVo{}, lib.ErrUserNotFound
        } else {
            panic(err)
        }
    }
    user := get.(domain.User)
    return vo.UserVo{
        Id:       user.Id,
        Username: user.Username,
        Name:     user.Name,
        Phone:    user.Phone,
        Dept:     user.Dept,
    }, nil
}

Test by Postman

Before test, you should add token to postman.

Then send a request with fake user id.

Then send a request with real user id.

Deploy Service

I want to show you how to deploy our service by docker-compose.
Dockerfile

FROM golang:1.16.6-alpine AS builder

ENV GO111MODULE=on
ARG user
ENV HOST_USER=$user
ENV GOPROXY=https://goproxy.cn,direct

WORKDIR /repo

ADD go.mod .
ADD go.sum .

ADD . ./

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk add --no-cache bash tzdata

ENV TZ="Asia/Shanghai"

RUN go mod vendor

RUN export GDD_VER=$(go list -mod=vendor -m -f '{{ .Version }}' github.com/unionj-cloud/go-doudou) && \
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -ldflags="-X 'github.com/unionj-cloud/go-doudou/svc/config.BuildUser=$HOST_USER' -X 'github.com/unionj-cloud/go-doudou/svc/config.BuildTime=$(date)' -X 'github.com/unionj-cloud/go-doudou/svc/config.GddVer=$GDD_VER'" -mod vendor -o api cmd/main.go

ENTRYPOINT ["/repo/api"]

docker-compose.yml file

version: '3.9'

services:
  db:
    container_name: db
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 1234
    ports:
      - 3306:3306
    volumes:
      - $PWD/my:/etc/mysql/conf.d
      - $PWD/sqlscripts:/docker-entrypoint-initdb.d
    networks:
      - tutorial

  usersvc:
    container_name: usersvc
    build:
      context: .
    environment:
      - GDD_BANNER=off
      - GDD_PORT=6060
      - DB_HOST=db
    expose:
      - "6060"
    ports:
      - "6060:6060"
    networks:
      - tutorial
    depends_on:
      - db

networks:
  tutorial:
    driver: bridge

Then execute below command:

docker-compose -f docker-compose.yml up -d

Source Code

Here is full source code with postman collection file Click Me.