dynamic

README

Dynamic module

Provide loading of object file and invokes at runtime with goloader

Documentation

Overview

Package dynamic is a hot loader toolkit based on goloader.

License

Source codes are under Apache License Version 2.0.

Underwater

1. Can load and link relocatable object files at runtime, also unload or reload them, in other words, goload is a runtime linker.
2. Module codes are loaded into an executable Mapping memory section ( same as how other JIT solutions work ).
3. Having lower footprint as go official plugin.

Notes

1. This project is in WIP stage. Current only target on go 1.21+.
2. User must be careful when use global symbols of those not ship with the host executable, other dynamics may depend on them. also there free sequence is important. current user should do it by themselves.
3. For goloader's limitation, current only exported function can link and use.
4. Sym is a function entry address, use AsOnce for a one-shot convert or As for a reusable convert.

Compile tool

This is a module compile tool to compile go files into relocatable object file (extension as .o), which can be loaded and execute at runtime as a Dynamic module. The compile tool can be installed by:

go install github.com/ZenLiuCN/dynamic/compiler@latest

It also can inspect the .o file's imports, prepare go sdk to compile the host executable and so on ... . For more details see the cli help:

compiler -h

Use this library on develop stage or compile distribution binaries

• 1. Prepare GO sdk

  use compile cli tool via `compiler prepare` or use shell script named as `patch.sh`.

• 2. Work around with the dynamics

• 3. Compile modules

  use `compiler compile` or `compiler module ` to compile modules

• 3. Restore the GO SDK

  use compile cli tool via `compiler clean` or use shell script named as `patch.sh`.

Samples

See testdata and tests.

Index

• Variables
• func As[T any](ptr *Sym) (x T)
• func AsOnce[T any](ptr Sym) (x T)
• func Compile(debug bool, o []string, remove bool) (err error)
• func CopyDir(src string, dest string, si fs.FileInfo) (err error)
• func CopyFile(src string, dest string, si fs.FileInfo) (err error)
• func Imports(debug bool, f []string) (err error)
• func Inspect(file, pkg string) ([]string, error)
• func Packs(dbg bool, sources []string, pkgPath string, noPkg bool, includes []string, ...) (err error)
• func Use[T any](dyn Dynamic, sym string) func(func(t T, err error))
• type Dependency
• type Dynamic
  • func NewDynamic(sym Symbols, debug ...bool) (d Dynamic)
• type Info
  • func ObjectImportsIter(file, pkgPath string) (err error, info *Info)
  • func (i Info) String() string
• type Infos
  • func LinkerImportsIter(link *goloader.Linker) (infos Infos)
  • func (i Infos) String() string
• type Proto
• type Sym
• type Symbols
  • func NewSymbols() (t Symbols, err error)
  • func (s Symbols) ExistsSymbols() []string

Variables

var (
	// ErrMissingSymbol occurs when can't found a symbol.
	ErrMissingSymbol = errors.New("missing symbol")
	// ErrAlreadyInitialized occurs when a Dynamic reinitializing.
	ErrAlreadyInitialized = errors.New("already initialized dynamic")
	// ErrLinked occurs when a Dynamic relinking.
	ErrLinked = errors.New("already linked")
	// ErrUninitialized occurs use or link a Dynamic before initialized.
	ErrUninitialized = errors.New("module not initialized")
)

var (
	// TestDebug for testing purpose.
	TestDebug = true
)

Functions

func As

func As[T any](ptr *Sym) (x T)

As convert fetched Sym to contract function type, this result can use as many times before ptr been GC.

func AsOnce

func AsOnce[T any](ptr Sym) (x T)

AsOnce convert fetched Sym to contract function type, this result can only use once for value passed ptr

func Compile

func Compile(debug bool, o []string, remove bool) (err error)

Compile an object file output to working directory

func CopyDir

func CopyDir(src string, dest string, si fs.FileInfo) (err error)

CopyDir from src to dest with optional src file info

func CopyFile

func CopyFile(src string, dest string, si fs.FileInfo) (err error)

CopyFile from src to dest with optional src file info

func Imports

func Imports(debug bool, f []string) (err error)

Imports generate import cfg as importcfg file in current working directory.

func Inspect

func Inspect(file, pkg string) ([]string, error)

Inspect display symbols inside an object file

func Packs

func Packs(dbg bool, sources []string, pkgPath string, noPkg bool, includes []string, excludes []string) (err error)

func Use

func Use[T any](dyn Dynamic, sym string) func(func(t T, err error))

Use create a function to fetch and use symbol on the fly

Types

type Dependency

type Dependency struct {
	File    string
	PkgPath string
	Symbols []string
}

Dependency contains a dependency package information

type Dynamic

type Dynamic interface {
	InitializeMany(file, pkg []string, types ...any) (err error) //Initialize from many object files
	Initialize(file, pkg string, types ...any) (err error)       //Initialize from one object file
	InitializeSerialized(in io.Reader, types ...any) (err error) //Initialize from serialized linker

	LoadDependencies(dependency Dependency, dependencies ...Dependency) error //load dependencies, must use before [Dynamic.Link]
	ExistsSymbols() []string                                                  //runtime symbols, valid after creation
	MissingSymbols() []string                                                 //dump the missing symbols, valid after initialize.

	Link() (err error) //link and create code module, must use after [Dynamic.Initialize], [Dynamic.InitializeSerialized] or [Dynamic.InitializeMany],throws ErrUninitialized

	Serialize(out io.Writer) error     //write linker data to an output binary format [gob] which may loaded by [Dynamic.InitializeSerialized]
	Fetch(sym string) (u Sym, ok bool) //fetch a symbol as unsafe.Pointer, which can cast to the desired type, throws ErrUninitialized
	MustFetch(sym string) (u Sym)      // fetch a symbol as unsafe.Pointer, which can cast to the desired type, throws ErrUninitialized or ErrMissingSymbol
	Exports() []string                 //exports symbols of the module. nil if not link
	Free(sync bool)                    //resources of dynamic module, sync parameter to sync the stdout or not
	GetLinker() *goloader.Linker       //fetch the internal [goloader.Linker], it is nil before initial stage by invoke one of [Dynamic.Initialize], [Dynamic.InitializeSerialized] or [Dynamic.InitializeMany]
	GetModule() *goloader.CodeModule   //fetch the internal [goloader.CodeModule], it is nil before link by invoke [Dynamic.Link]
	// contains filtered or unexported methods
}

Dynamic module from object files or serialized Liner file, this interface can not be implement outside this package.

Stage

- Initialized : load modules, current can check imports symbols and so on. - Linked : link with runtime, current can fetch the exposed symbols for use.

Use Steps:

1. InitializeMany or Initialize or InitializeSerialized to initialize this dynamic module.
2. [Dynamic.Link] to link the code to runtime and other global dependencies.
3. Use this module.
3. Call [Dynamic.Free] to release the resources.

Note:

1. Must fetch and use one symbol as desired type inside one specific goroutine.
2. Dynamic itself can be used safe between goroutines, but not thread-safe.

func NewDynamic

func NewDynamic(sym Symbols, debug ...bool) (d Dynamic)

NewDynamic create new dynamic with provided Symbols, an optional debug parameter will enable debug logging inside Dynamic

type Info

type Info struct {
	File    string
	PkgPath string
	Imports map[string]string // with pairs of package import path and version
}

Info contains the import information of a linker

func ObjectImportsIter

func ObjectImportsIter(file, pkgPath string) (err error, info *Info)

ObjectImportsIter resolve all imported packages and version (only if it's a module).

this use for parse dependencies

func (Info) String

func (i Info) String() string

type Infos

type Infos []*Info

Infos is a stringer slice of Info

func LinkerImportsIter

func LinkerImportsIter(link *goloader.Linker) (infos Infos)

LinkerImportsIter resolve all imported packages and version if it's a module.

this use for parse dependencies

func (Infos) String

func (i Infos) String() string

type Proto

type Proto interface {
	Name() string
	Action() string
}

Proto for testing purpose.

type Sym

type Sym uintptr

Sym is a simple alias of uintptr, which is a pointer to symbol entry address.

type Symbols

type Symbols map[string]uintptr

Symbols contains global resolved symbols

If two Dynamic shares the same Symbols instance, it may depend on each other after make.

func NewSymbols

func NewSymbols() (t Symbols, err error)

func (Symbols) ExistsSymbols

func (s Symbols) ExistsSymbols() []string