core

README

English | 中文

CORE

Package core is a service container that elegantly bootstrap and coordinate twelve-factor apps in Go.

Background

The twelve-factor methodology has proven its worth over the years. Since its invention many fields in technology have changed, many among them are shining and exciting. In the age of Kubernetes, service mesh and serverless architectures, the twelve-factor methodology has not faded away, but rather has happened to be a good fit for nearly all of those powerful platforms.

Scaffolding a twelve-factor go app may not be a difficult task for experienced engineers, but certainly presents some challenges to juniors. For those who are capable of setting things up, there are still many decisions left to make, and choices to be agreed upon within the team.

Package core was created to bootstrap and coordinate such services.

Feature

Package core shares the common concerns of your application:

• Configuration management: env, flags, files, etc.
• Pluggable transports: HTTP, gRPC, etc.
• Dependency injection
• Job management: Cron, long-running, one-off commandline, etc.
• Events and Queues
• Metrics
• Distributed Tracing
• Database migrations and seedings
• Distributed transactions
• Leader election

Overview

Whatever the app is, the bootstrapping phase is roughly composed by:

• Read the configuration from out of the binary. Namely, flags, environment variables, and/or configuration files.

• Initialize dependencies. Databases, message queues, service discoveries, etc.

• Define how to run the app. HTTP, RPC, command-lines, cronjobs, or more often mixed.

Package core abstracts those repeated steps, keeping them concise, portable yet explicit. Let's see the following snippet:

package main

import (
  "context"
  "net/http"

  "github.com/DoNewsCode/core"
  "github.com/DoNewsCode/core/observability"
  "github.com/DoNewsCode/core/otgorm"
  "github.com/gorilla/mux"
)

func main() {
  // Phase One: create a core from a configuration file
  c := core.New(core.WithYamlFile("config.yaml"))

  // Phase two: bind dependencies
  c.Provide(otgorm.Providers())

  // Phase three: define service
  c.AddModule(core.HttpFunc(func(router *mux.Router) {
    router.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
      writer.Write([]byte("hello world"))
    })
  }))

  // Phase Four: run!
  c.Serve(context.Background())
}

In a few lines, an HTTP service is bootstrapped in the style outlined above. It is simple, explicit, and to some extent, declarative.

The service demonstrated above uses an inline handler function to highlight the point. Normally, for real projects, we will use modules instead. The "module" in package Core's glossary is not necessarily a go module (though it can be). It is simply a group of services.

You may note that the HTTP service doesn't really consume the dependency. That's true.

Let's rewrite the HTTP service to consume the above dependencies.

package main

import (
  "context"
  "net/http"

  "github.com/DoNewsCode/core"
  "github.com/DoNewsCode/core/otgorm"
  "github.com/DoNewsCode/core/srvhttp"
  "github.com/gorilla/mux"
  "gorm.io/gorm"
)

type User struct {
  Id   string
  Name string
}

type Repository struct {
  DB *gorm.DB
}

func (r Repository) Find(id string) (*User, error) {
  var user User
  if err := r.DB.First(&user, id).Error; err != nil {
    return nil, err
  }
  return &user, nil
}

type Handler struct {
  R Repository
}

func (h Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
  encoder := srvhttp.NewResponseEncoder(writer)
  encoder.Encode(h.R.Find(request.URL.Query().Get("id")))
}

type Module struct {
  H Handler
}

func New(db *gorm.DB) Module {
  return Module{Handler{Repository{db}}}
}

func (m Module) ProvideHTTP(router *mux.Router) {
  router.Handle("/", m.H)
}

func main() {
  // Phase One: create a core from a configuration file
  c := core.New(core.WithYamlFile("config.yaml"))

  // Phase two: bind dependencies
  c.Provide(otgorm.Providers())

  // Phase three: define service
  c.AddModuleFunc(New)

  // Phase four: run!
  c.Serve(context.Background())
}

Phase three has been replaced by the c.AddModuleFunc(New). AddModuleFunc populates the arguments to New from dependency containers and add the returned module instance to the internal module registry.

When c.Serve() is called, all registered modules will be scanned for implemented interfaces. The module in the example implements interface:

type HTTPProvider interface {
	ProvideHTTP(router *mux.Router)
}

Therefore, the core knows this module wants to expose HTTP service and subsequently invokes the ProvideHTTP with a router. You can register multiple modules, and each module can implement one or more services.

Now we have a fully workable project, with layers of handler, repository, and entity. Had this been a DDD workshop, we would be expanding the example even further.

That being said, let's redirect our attention to other goodies package core has offered:

• Package core natively supports multiplexing modules. You could start you project as a monolith with multiple modules, and gradually migrate them into microservices.

• Package core doesn't lock in transport or framework. For instance, you can use go kit to construct your services, and bring in transports like gRPC, AMQP, thrift, etc. Non-network services like CLI and Cron are also supported. See all available interfaces a module can expose. You can also invent your own.

• Package core also babysits the services after initialization. The duty includes but not limited to distributed tracing, metrics exporting, error handling, event-dispatching, and leader election.

Be sure to checkout the documentation section to learn more.

Documentation

• Tutorial
• GoDoc
• Starter Template
• Contributing

Design Principles

• No package global state.
• Promote dependency injection.
• Testable code.
• Minimalist interface design. Easy to decorate and replace.
• Work with the Go ecosystem rather than reinventing the wheel.
• End to end Context passing.

Non-Goals

• Tries to be a Spring, Laravel, or Ruby on Rails.
• Tries to care about service details.
• Tries to reimplement the functionality provided by modern platforms.

Works well with

• Gin (if HTTP only)
• Go Kit (if multiple transports)
• Kratos v2

Thanks for JetBrains OS licenses

Documentation

Overview

Package core is a service container that elegantly bootstrap and coordinate twelve-factor apps in Go.

Checkout the README.md for an overview of this package.

Example (Minimal)

package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"net/http"
	"time"

	"github.com/DoNewsCode/core"

	"github.com/gorilla/mux"
)

func main() {
	// Spin up a real server
	c := core.Default(core.WithInline("log.level", "none"))
	c.AddModule(core.HttpFunc(func(router *mux.Router) {
		router.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
			writer.Write([]byte("hello world"))
		})
	}))
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	go c.Serve(ctx)

	// Giver server sometime to be ready.
	time.Sleep(time.Second)

	// Let's try if the server works.
	resp, err := http.Get("http://localhost:8080/")
	if err != nil {
		return
	}
	defer resp.Body.Close()
	bytes, _ := ioutil.ReadAll(resp.Body)
	cancel()

	fmt.Println(string(bytes))
}

Output:

hello world

Index

• Constants
• func NewServeModule(in serveIn) serveModule
• func ProvideAppName(conf contract.ConfigUnmarshaler) contract.AppName
• func ProvideConfig(configStack []config.ProviderSet, configWatcher contract.ConfigWatcher) contract.ConfigUnmarshaler
• func ProvideDi(conf contract.ConfigUnmarshaler) *dig.Container
• func ProvideEnv(conf contract.ConfigUnmarshaler) contract.Env
• func ProvideLogger(conf contract.ConfigUnmarshaler, appName contract.AppName, env contract.Env) log.Logger
• func WithYamlFile(path string) (CoreOption, CoreOption)
• type AppNameProvider
• type C
  • func Default(opts ...CoreOption) *C
  • func New(opts ...CoreOption) *C
  • func (c *C) AddModule(module any)
  • func (c *C) AddModuleFunc(constructor any)
  • func (c *C) ApplyRootCommand(command *cobra.Command)
  • func (c *C) Invoke(function any)
  • func (c *C) Provide(deps di.Deps)
  • func (c *C) ProvideEssentials()
  • func (c *C) Serve(ctx context.Context) error
  • func (c *C) Shutdown()
• type CloserProvider
• type CommandProvider
• type ConfParser
• type ConfProvider
• type ConfigProvider
• type CoreOption
  • func SetAppNameProvider(provider AppNameProvider) CoreOption
  • func SetConfigProvider(provider ConfigProvider) CoreOption
  • func SetDiProvider(provider DiProvider) CoreOption
  • func SetEnvProvider(provider EnvProvider) CoreOption
  • func SetLoggerProvider(provider LoggerProvider) CoreOption
  • func WithConfigStack(provider ConfProvider, parser ConfParser) CoreOption
  • func WithConfigWatcher(watcher contract.ConfigWatcher) CoreOption
  • func WithInline(key string, entry any) CoreOption
• type CronProvider
• type DiProvider
• type EnvProvider
• type GRPCProvider
• type HTTPProvider
• type HttpFunc
  • func (h HttpFunc) ProvideHTTP(router *mux.Router)
• type LoggerProvider
• type OnGRPCServerShutdownPayload
• type OnGRPCServerStartPayload
• type OnHTTPServerShutdownPayload
• type OnHTTPServerStartPayload
• type RunProvider
• type Runnable

Examples

• Package (Minimal)
• C (Stack)
• C.AddModule (Inject)
• C.AddModule (Simple)
• C.AddModuleFunc
• C.Provide

Constants

const (
	// OnHTTPServerStart is an event triggered when the http server is ready to serve
	// traffic. At this point the module is already wired up. This event is useful to
	// register service to service discovery.
	OnHTTPServerStart event = "onHTTPServerStart"

	// OnHTTPServerShutdown is an event triggered when the http server is shutting down.
	// traffic. At this point The traffic can no longer reach the server, but the
	// database and other infrastructures are not closed yet. This event is useful
	// to unregister service to service discovery.
	OnHTTPServerShutdown event = "onHTTPServerShutdown"

	// OnGRPCServerStart is an event triggered when the grpc server is ready to serve
	// traffic. At this point the module is already wired up. This event is useful to
	// register service to service discovery.
	OnGRPCServerStart event = "onGRPCServerStart"

	// OnGRPCServerShutdown is an event triggered when the http server is shutting down.
	// traffic. At this point The traffic can no longer reach the server, but the
	// database and other infrastructures are not closed yet. This event is useful
	// to unregister service to service discovery.
	OnGRPCServerShutdown event = "onGRPCServerShutdown"
)

Functions

func NewServeModule

func NewServeModule(in serveIn) serveModule

func ProvideAppName

func ProvideAppName(conf contract.ConfigUnmarshaler) contract.AppName

ProvideAppName is the default AppNameProvider for package Core.

func ProvideConfig

func ProvideConfig(configStack []config.ProviderSet, configWatcher contract.ConfigWatcher) contract.ConfigUnmarshaler

ProvideConfig is the default ConfigProvider for package Core.

func ProvideDi

func ProvideDi(conf contract.ConfigUnmarshaler) *dig.Container

ProvideDi is the default DiProvider for package Core.

func ProvideEnv

func ProvideEnv(conf contract.ConfigUnmarshaler) contract.Env

ProvideEnv is the default EnvProvider for package Core.

func ProvideLogger

func ProvideLogger(conf contract.ConfigUnmarshaler, appName contract.AppName, env contract.Env) log.Logger

ProvideLogger is the default LoggerProvider for package Core.

func WithYamlFile

func WithYamlFile(path string) (CoreOption, CoreOption)

WithYamlFile is a two-in-one coreOption. It uses the configuration file as the source of configuration, and watches the change of that file for hot reloading.

Types

type AppNameProvider

type AppNameProvider func(conf contract.ConfigUnmarshaler) contract.AppName

AppNameProvider provides the contract.AppName to the core.

type C

type C struct {
	AppName contract.AppName
	Env     contract.Env
	contract.ConfigAccessor
	logging.LevelLogger
	*container.Container
	// contains filtered or unexported fields
}

C stands for the core of the application. It contains service definitions and dependencies. C means to be used in the boostrap phase of the application. Do not pass C into services and use it as a service locator.

Example (Stack)

package main

import (
	"context"
	"flag"
	"net/http"
	"strings"

	"github.com/DoNewsCode/core"

	"github.com/gorilla/mux"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/basicflag"
	"github.com/knadh/koanf/providers/env"
	"github.com/knadh/koanf/providers/file"
)

func main() {
	fs := flag.NewFlagSet("example", flag.ContinueOnError)
	fs.String("log.level", "error", "the log level")
	// Spin up a real server
	c := core.New(
		core.WithConfigStack(basicflag.Provider(fs, "."), nil),
		core.WithConfigStack(env.Provider("APP_", ".", func(s string) string {
			return strings.ToLower(strings.Replace(s, "APP_", "", 1))
		}), nil),
		core.WithConfigStack(file.Provider("./config/testdata/mock.json"), json.Parser()),
	)
	c.AddModule(core.HttpFunc(func(router *mux.Router) {
		router.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
			writer.Write([]byte("hello world"))
		})
	}))
	c.Serve(context.Background())
}

func Default

func Default(opts ...CoreOption) *C

Default creates a core.C under its default state. Core dependencies are already provided, and the config module and serve module are bundled.

func New

func New(opts ...CoreOption) *C

New creates a new bare-bones C.

func (*C) AddModule

func (c *C) AddModule(module any)

AddModule adds a module to the core.

A Module is a group of functionality. It must provide some runnable stuff: http handlers, grpc handlers, cron jobs, one-time command, etc.

Optionally if the module embeds di.In, its fields will be injected by DI container. The semantics of injection follows the same rule of dig.Invoke. Note that the module added in this way will not retain any original field values, i.e. the module will only contain fields populated by DI container.

Example (Inject)

package main

import (
	"os"

	"github.com/DoNewsCode/core"
	"github.com/DoNewsCode/core/di"

	"github.com/go-kit/log"
)

func main() {
	type Foo struct {
		di.In
		Logger log.Logger
	}
	var f Foo
	c := core.New()
	c.Provide(di.Deps{func() log.Logger {
		return log.NewLogfmtLogger(os.Stdout)
	}})
	// If the module embeds di.In (directly or indirectly via pointer), then DI will
	// populate its fields.
	c.AddModule(&f)
	f.Logger.Log("msg", "hello")
}

Output:

msg=hello

Example (Simple)

package main

import (
	"fmt"

	"github.com/DoNewsCode/core"
)

func main() {
	type Foo struct{}
	c := core.New()
	c.AddModule(Foo{})
	fmt.Printf("%T\n", c.Modules()...)
}

Output:

core_test.Foo

func (*C) AddModuleFunc

func (c *C) AddModuleFunc(constructor any)

AddModuleFunc add the module after Invoking its constructor. Clean up functions and errors are handled automatically.

Example

package main

import (
	"fmt"

	"github.com/DoNewsCode/core"
)

func main() {
	type Foo struct{}
	c := core.New()
	c.AddModuleFunc(func() (Foo, func(), error) {
		return Foo{}, func() {}, nil
	})
	fmt.Printf("%T %T\n", c.Modules()...)
}

Output:

core.cleanup core_test.Foo

func (*C) ApplyRootCommand

func (c *C) ApplyRootCommand(command *cobra.Command)

ApplyRootCommand iterates through every CommandProvider registered in the container, and introduce the root *cobra.Command to everyone.

func (*C) Invoke

func (c *C) Invoke(function any)

Invoke runs the given function after instantiating its dependencies. Any arguments that the function has are treated as its dependencies. The dependencies are instantiated in an unspecified order along with any dependencies that they might have. The function may return an error to indicate failure. The error will be returned to the caller as-is.

It internally calls uber's dig library. Consult dig's documentation for details. (https://pkg.go.dev/go.uber.org/dig)

func (*C) Provide

func (c *C) Provide(deps di.Deps)

Provide adds dependencies provider to the core. Note the dependency provider must be a function in the form of:

func(foo Foo) Bar

where foo is the upstream dependency and Bar is the provided type. The order for providers doesn't matter. They are only executed lazily when the Invoke is called.

This method internally calls uber's dig library. Consult dig's documentation for details. (https://pkg.go.dev/go.uber.org/dig)

The difference is, core.Provide has been made to accommodate the convention from google/wire (https://github.com/google/wire). All "func()" returned by constructor are treated as clean up functions. It also examines if the dependency implements the modular interface. If so, this dependency will be added the module collection.

Example

package main

import (
	"fmt"

	"github.com/DoNewsCode/core"
	"github.com/DoNewsCode/core/di"
)

func main() {
	type Foo struct {
		Value string
	}
	type Bar struct {
		foo Foo
	}
	c := core.New()
	c.Provide(di.Deps{
		func() (foo Foo, cleanup func(), err error) {
			return Foo{
				Value: "test",
			}, func() {}, nil
		},
		func(foo Foo) Bar {
			return Bar{foo: foo}
		},
	})
	c.Invoke(func(bar Bar) {
		fmt.Println(bar.foo.Value)
	})
}

Output:

test

func (*C) ProvideEssentials

func (c *C) ProvideEssentials()

ProvideEssentials adds the default core dependencies to the core.

func (*C) Serve

func (c *C) Serve(ctx context.Context) error

Serve runs the serve command bundled in the core. For larger projects, consider use full-featured ServeModule instead of calling serve directly.

func (*C) Shutdown

func (c *C) Shutdown()

Shutdown iterates through every CloserProvider registered in the container, and calls them in the reversed order of registration.

type CloserProvider

type CloserProvider interface {
	ProvideCloser()
}

CloserProvider provides a shutdown function that will be called when service exits.

type CommandProvider

type CommandProvider interface {
	ProvideCommand(command *cobra.Command)
}

CommandProvider provides cobra.Command.

type ConfParser

type ConfParser interface {
	Unmarshal([]byte) (map[string]any, error)
	Marshal(map[string]any) ([]byte, error)
}

ConfParser models a parser for configuration. For example, yaml.Parser.

type ConfProvider

type ConfProvider interface {
	ReadBytes() ([]byte, error)
	Read() (map[string]any, error)
}

ConfProvider models a configuration provider. For example, file.Provider.

type ConfigProvider

type ConfigProvider func(configStack []config.ProviderSet, configWatcher contract.ConfigWatcher) contract.ConfigUnmarshaler

ConfigProvider provides contract.ConfigAccessor to the core.

type CoreOption

type CoreOption func(*coreValues)

CoreOption is the option to modify core attribute.

func SetAppNameProvider

func SetAppNameProvider(provider AppNameProvider) CoreOption

SetAppNameProvider is a CoreOption to replaces the default AppNameProvider.

func SetConfigProvider

func SetConfigProvider(provider ConfigProvider) CoreOption

SetConfigProvider is a CoreOption to replaces the default ConfigProvider.

func SetDiProvider

func SetDiProvider(provider DiProvider) CoreOption

SetDiProvider is a CoreOption to replaces the default DiContainer.

func SetEnvProvider

func SetEnvProvider(provider EnvProvider) CoreOption

SetEnvProvider is a CoreOption to replaces the default EnvProvider.

func SetLoggerProvider

func SetLoggerProvider(provider LoggerProvider) CoreOption

SetLoggerProvider is a CoreOption to replaces the default LoggerProvider.

func WithConfigStack

func WithConfigStack(provider ConfProvider, parser ConfParser) CoreOption

WithConfigStack is a CoreOption that defines a configuration layer. See package config for details.

func WithConfigWatcher

func WithConfigWatcher(watcher contract.ConfigWatcher) CoreOption

WithConfigWatcher is a CoreOption that adds a config watcher to the core (for hot reloading configs).

func WithInline

func WithInline(key string, entry any) CoreOption

WithInline is a CoreOption that creates a inline config in the configuration stack.

type CronProvider

type CronProvider interface {
	ProvideCron(cron *cron.Cron)
}

CronProvider provides cron jobs.

type DiProvider

type DiProvider func(conf contract.ConfigUnmarshaler) *dig.Container

DiProvider provides the *dig.Container to the core.

type EnvProvider

type EnvProvider func(conf contract.ConfigUnmarshaler) contract.Env

EnvProvider provides the contract.Env to the core.

type GRPCProvider

type GRPCProvider interface {
	ProvideGRPC(server *grpc.Server)
}

GRPCProvider provides gRPC services.

type HTTPProvider

type HTTPProvider interface {
	ProvideHTTP(router *mux.Router)
}

HTTPProvider provides http services.

type HttpFunc

type HttpFunc func(router *mux.Router)

HttpFunc converts a function to a module provides http.

func (HttpFunc) ProvideHTTP

func (h HttpFunc) ProvideHTTP(router *mux.Router)

ProvideHTTP implements container.HTTPProvider

type LoggerProvider

type LoggerProvider func(conf contract.ConfigUnmarshaler, appName contract.AppName, env contract.Env) log.Logger

LoggerProvider provides the log.Logger to the core.

type OnGRPCServerShutdownPayload

type OnGRPCServerShutdownPayload struct {
	GRPCServer *grpc.Server
	Listener   net.Listener
}

OnGRPCServerShutdownPayload is the payload of OnGRPCServerShutdown

type OnGRPCServerStartPayload

type OnGRPCServerStartPayload struct {
	GRPCServer *grpc.Server
	Listener   net.Listener
}

OnGRPCServerStartPayload is the payload of OnGRPCServerStart

type OnHTTPServerShutdownPayload

type OnHTTPServerShutdownPayload struct {
	HTTPServer *http.Server
	Listener   net.Listener
}

OnHTTPServerShutdownPayload is the payload of OnHTTPServerShutdown

type OnHTTPServerStartPayload

type OnHTTPServerStartPayload struct {
	HTTPServer *http.Server
	Listener   net.Listener
}

OnHTTPServerStartPayload is the payload of OnHTTPServerStart

type RunProvider

type RunProvider interface {
	ProvideRunGroup(group *run.Group)
}

RunProvider provides a runnable actor. Use it to register any server-like actions. For example, kafka consumer can be started here.

type Runnable

type Runnable interface {
	Run(ctx context.Context) error
}

Runnable provides a runnable actor. The core will call Run in an exclusive goroutine, so it is safe for Run to block the execution. Return only when run completes. The received context.Context is canceled at shutdown.