goloader

README

Goloader/JIT Compiler for Go

The goloader/jit package can compile and load Go code from text, file, folder or remote package (including code with package imports).

It automatically resolves package dependencies recursively, and provides a type safe way of interacting with the built functions.

Forked from dearplain and pkujhd.

Usage

Build

Make sure you're using go >= 1.18.

First, execute the following command. This is because Goloader relies on the internal package, which is forbidden by the Go compiler.

cp -r $GOROOT/src/cmd/internal $GOROOT/src/cmd/objfile

Go compiler patch

To allow the loader to know the types of exported functions, this package will attempt to patch the Go compiler (gc) to emit these if not already patched.

The effect of the patch can be found in jit/gc.patch.

go install github.com/eh-steve/goloader/jit/patchgc@latest
# You may need to run patchgc as sudo if your $GOROOT is owned by root
# (alternatively `chown -R $USER:$USER $GOROOT`)
patchgc

Example Usage

package main

import (
	"fmt"
	"github.com/eh-steve/goloader/jit"
)

func main() {
	conf := jit.BuildConfig{
		KeepTempFiles:   false,          // Files are copied/written to a temp dir to ensure it is writable. This retains the temporary copies
		ExtraBuildFlags: []string{"-x"}, // Flags passed to go build command
		BuildEnv:        nil,            // Env vars to set for go build toolchain
		TmpDir:          "",             // To control where temporary files are copied
		DebugLog:        true,           //
	}

	loadable, err := jit.BuildGoFiles(conf, "./path/to/file1.go", "/path/to/file2.go")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoPackage(conf, "./path/to/package")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoPackageRemote(conf, "github.com/some/package/v4", "latest")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoText(conf, `
package mypackage

import "encoding/json"

func MyFunc(input []byte) (interface{}, error) {
	var output interface{}
	err := json.Unmarshal(input, &output)
	return output, err
}
`)

	if err != nil {
		panic(err)
	}

	module, err := loadable.Load()
	// module.SymbolsByPkg is a map[string]map[string]interface{} of all packages and their exported functions and global vars
	symbols := module.SymbolsByPkg[loadable.ImportPath]
	if err != nil {
		panic(err)
	}
	defer func() {
		err = module.Unload()
		if err != nil {
			panic(err)
		}
	}()
	switch f := symbols["MyFunc"].(type) {
	case func([]byte) (interface{}, error):
		result, err := f([]byte(`{"k":"v"}`))
		if err != nil {
			panic(err)
		}
		fmt.Println(result)
	default:
		panic("Function signature was not what was expected")
	}
}

How does it work?

Goloader works like a linker, it relocates the addresses of symbols in an object file, generates runnable code, and then reuses the runtime functions and the type pointers of the loader where available.

Goloader provides some information to the runtime and garbage collector of Go, which allows it to work correctly with them.

Please note that Goloader is not a scripting engine. It reads the archives emitted from the Go compiler and makes them runnable. All features of Go are supported, and run just as fast and lightweight as native Go code.

Comparison with plugin

Plugin:

• Can't load plugins not built with exact same versions of packages that host binary uses (plugin was built with a different version of package) - this makes them basically unusable in most large projects
• Introduces dependency on libdl/CGo (and doesn't work on Windows)
• Prevents linker deadcode elimination for unreachable methods (increases host binary size with unused methods)
• Can't be unloaded/dynamically updated
• Duplicates a lot of the go runtime (large binary sizes)

Goloader:

• Can build/load any packages (somewhat unsafely - it attempts to verify that types across JIT packages and host packages match, but doesn't do the same checks for function signatures)
• Pure Go - no dependency on libdl/Cgo
• Patches host itabs containing unreachable methods instead of preventing linker deadcode elimination
• Can be unloaded, and objects from one version of a JIT package can be converted at runtime to those from another version, to allow dynamic adjustment of functions/methods without losing state
• Reuses the runtime from the host binary (much smaller binaries)

Goloader supports pprof tool (yes, you can see code loaded by Goloader in pprof), but does not (yet) support debugging with delve.

OS/Arch Compatibility

JIT compiler tested/passing on:

OS/Arch	amd64/+CGo	arm64/+CGo	amd64/-CGo	arm64/-CGo
Linux/go-1.20.3	✔	✔	✔	✔
Darwin/go-1.20.3	✔	✔	✔	✔
Windows/go-1.20.3	✔	⁉	✔	⁉
Linux/go-1.19.4	✔	✔	✔	✔
Darwin/go-1.19.4	✔	✔	✔	✔
Windows/go-1.19.4	✔	⁉	✔	⁉
Linux/go-1.18.8	✔	✔	✔	✔
Darwin/go-1.18.8	✔	✔	✔	✔
Windows/go-1.18.8	❌	⁉	✔	⁉

Warning

Don't use "-s -w" compile argument, It strips symbol table.

Documentation

Index

• Constants
• Variables
• func AsRType(_typ *_type) reflect.Type
• func CanAttemptConversion(oldValue interface{}, newType reflect.Type) bool
• func ConvertTypesAcrossModules(oldModule, newModule *CodeModule, oldValue interface{}, newType reflect.Type) (res interface{}, err error)
• func FuncPCsABI0(abiInternalPCs []uintptr) []uintptr
• func MakeThreadJITCodeExecutable(ptr uintptr, len int)
• func Mmap(size int) ([]byte, error)
• func MmapData(size int) ([]byte, error)
• func Munmap(b []byte) (err error)
• func Parse(f *os.File, pkgpath *string) ([]string, error)
• func RegSymbol(symPtr map[string]uintptr, pkgSet map[string]struct{}) error
• func RegSymbolWithSo(symPtr map[string]uintptr, pkgSet map[string]struct{}, path string) error
• func RegTypes(symPtr map[string]uintptr, interfaces ...interface{})
• func WithDumpTextBeforeAndAfterRelocs() func(*LinkerOptions)
• func WithForceTestRelocationEpilogues() func(*LinkerOptions)
• func WithNoRelocationEpilogues() func(*LinkerOptions)
• func WithRandomSymbolNameOrder() func(*LinkerOptions)
• func WithRelocationDebugWriter(writer io.Writer) func(*LinkerOptions)
• func WithSkipTypeDeduplicationForPackages(packages []string) func(*LinkerOptions)
• func WithSymbolNameOrder(symNames []string) func(*LinkerOptions)
• type CodeModule
  • func Load(linker *Linker, symPtr map[string]uintptr) (codeModule *CodeModule, err error)
  • func (cm *CodeModule) DataAddr() (start, end uintptr)
  • func (cm *CodeModule) TextAddr() (start, end uintptr)
  • func (cm *CodeModule) Unload() error
• type Linker
  • func ReadObjs(files []string, pkgPath []string, globalSymPtr map[string]uintptr, ...) (*Linker, error)
  • func (linker *Linker) Autolib() []string
  • func (linker *Linker) Opts(linkerOpts ...LinkerOptFunc)
  • func (linker *Linker) SymbolOrder() []string
  • func (linker *Linker) UnloadStrings()
  • func (linker *Linker) UnresolvedExternalSymbolUsers(symbolMap map[string]uintptr) map[string][]string
  • func (linker *Linker) UnresolvedExternalSymbols(symbolMap map[string]uintptr, ignorePackages []string, ...) map[string]*obj.Sym
  • func (linker *Linker) UnresolvedPackageReferences(existingPkgs []string) []string
• type LinkerOptFunc
• type LinkerOptions
• type Type
  • func AsType(_typ *_type) Type
• type Value

Constants

const (
	PtrSize    = 4 << (^uintptr(0) >> 63)
	Uint32Size = int(unsafe.Sizeof(uint32(0)))
	IntSize    = int(unsafe.Sizeof(int(0)))
	UInt64Size = int(unsafe.Sizeof(uint64(0)))

	FindFuncBucketSize = int(unsafe.Sizeof(findfuncbucket{}))
	InvalidHandleValue = ^uintptr(0)
	InvalidOffset      = int(-1)
	PageSize           = 1 << 12 //4096
)

size

const (
	EmptyString    = ""
	DefaultPkgPath = "main"
	ZeroByte       = byte(0x00)
)

const (
	FileSymPrefix              = "gofile.."
	MainPkgPrefix              = "main."
	OsStdout                   = "os.Stdout"
	FirstModulePrefix          = "firstmodule."
	DefaultStringContainerSize = 1024 * 1024 * 16
)

string match prefix/suffix

const (
	TypeImportPathPrefix = "type:.importpath."
	TypeDoubleDotPrefix  = "type:."
	TypePrefix           = "type:"
	ItabPrefix           = "go:itab."
	TypeStringPrefix     = "go:string."
	ObjSymbolSeparator   = ":"
)

const (
	Invalid       = reflectlite.Invalid
	Bool          = reflectlite.Bool
	Int           = reflectlite.Int
	Int8          = reflectlite.Int8
	Int16         = reflectlite.Int16
	Int32         = reflectlite.Int32
	Int64         = reflectlite.Int64
	Uint          = reflectlite.Uint
	Uint8         = reflectlite.Uint8
	Uint16        = reflectlite.Uint16
	Uint32        = reflectlite.Uint32
	Uint64        = reflectlite.Uint64
	Uintptr       = reflectlite.Uintptr
	Float32       = reflectlite.Float32
	Float64       = reflectlite.Float64
	Complex64     = reflectlite.Complex64
	Complex128    = reflectlite.Complex128
	Array         = reflectlite.Array
	Chan          = reflectlite.Chan
	Func          = reflectlite.Func
	Interface     = reflectlite.Interface
	Map           = reflectlite.Map
	Pointer       = reflectlite.Pointer
	Slice         = reflectlite.Slice
	String        = reflectlite.String
	Struct        = reflectlite.Struct
	UnsafePointer = reflectlite.UnsafePointer
	Ptr           = reflectlite.Ptr
)

const (
	EmptyPkgPath = "<unlinkable>"
)

const (
	KindGCProg = 1 << 6
)

const (
	RuntimeDeferReturn = "runtime.deferreturn"
)

runtime symbol

const (
	TLSNAME = "(TLS)"
)

Variables

var Indirect = reflectlite.Indirect

var MakeMapWithSize = reflectlite.MakeMapWithSize

var New = reflectlite.New

var NewAt = reflectlite.NewAt

var TypeOf = reflectlite.TypeOf

var ValueOf = reflectlite.ValueOf

Functions

func AsRType

func AsRType(_typ *_type) reflect.Type

func CanAttemptConversion

func CanAttemptConversion(oldValue interface{}, newType reflect.Type) bool

func ConvertTypesAcrossModules

func ConvertTypesAcrossModules(oldModule, newModule *CodeModule, oldValue interface{}, newType reflect.Type) (res interface{}, err error)

func FuncPCsABI0

func FuncPCsABI0(abiInternalPCs []uintptr) []uintptr

func MakeThreadJITCodeExecutable

func MakeThreadJITCodeExecutable(ptr uintptr, len int)

func Mmap

func Mmap(size int) ([]byte, error)

func MmapData

func MmapData(size int) ([]byte, error)

func Munmap

func Munmap(b []byte) (err error)

func Parse

func Parse(f *os.File, pkgpath *string) ([]string, error)

func RegSymbol

func RegSymbol(symPtr map[string]uintptr, pkgSet map[string]struct{}) error

func RegSymbolWithSo

func RegSymbolWithSo(symPtr map[string]uintptr, pkgSet map[string]struct{}, path string) error

func RegTypes

func RegTypes(symPtr map[string]uintptr, interfaces ...interface{})

func WithDumpTextBeforeAndAfterRelocs

func WithDumpTextBeforeAndAfterRelocs() func(*LinkerOptions)

func WithForceTestRelocationEpilogues

func WithForceTestRelocationEpilogues() func(*LinkerOptions)

func WithNoRelocationEpilogues

func WithNoRelocationEpilogues() func(*LinkerOptions)

func WithRandomSymbolNameOrder

func WithRandomSymbolNameOrder() func(*LinkerOptions)

func WithRelocationDebugWriter

func WithRelocationDebugWriter(writer io.Writer) func(*LinkerOptions)

func WithSkipTypeDeduplicationForPackages

func WithSkipTypeDeduplicationForPackages(packages []string) func(*LinkerOptions)

func WithSymbolNameOrder

func WithSymbolNameOrder(symNames []string) func(*LinkerOptions)

WithSymbolNameOrder allows you to control the sequence (placement in memory) of symbols from an object file. When not set, the order as parsed from the archive file is used.

Types

type CodeModule

type CodeModule struct {
	SymbolsByPkg map[string]map[string]interface{}
	Syms         map[string]uintptr
	// contains filtered or unexported fields
}

func Load

func Load(linker *Linker, symPtr map[string]uintptr) (codeModule *CodeModule, err error)

func (*CodeModule) DataAddr

func (cm *CodeModule) DataAddr() (start, end uintptr)

func (*CodeModule) TextAddr

func (cm *CodeModule) TextAddr() (start, end uintptr)

func (*CodeModule) Unload

func (cm *CodeModule) Unload() error

type Linker

type Linker struct {
	Arch *sys.Arch
	// contains filtered or unexported fields
}

func ReadObjs

func ReadObjs(files []string, pkgPath []string, globalSymPtr map[string]uintptr, linkerOpts ...LinkerOptFunc) (*Linker, error)

func (*Linker) Autolib

func (linker *Linker) Autolib() []string

func (*Linker) Opts

func (linker *Linker) Opts(linkerOpts ...LinkerOptFunc)

func (*Linker) SymbolOrder

func (linker *Linker) SymbolOrder() []string

func (*Linker) UnloadStrings

func (linker *Linker) UnloadStrings()

func (*Linker) UnresolvedExternalSymbolUsers

func (linker *Linker) UnresolvedExternalSymbolUsers(symbolMap map[string]uintptr) map[string][]string

func (*Linker) UnresolvedExternalSymbols

func (linker *Linker) UnresolvedExternalSymbols(symbolMap map[string]uintptr, ignorePackages []string, stdLibPkgs map[string]struct{}, unsafeBlindlyUseFirstModuleTypes bool) map[string]*obj.Sym

func (*Linker) UnresolvedPackageReferences

func (linker *Linker) UnresolvedPackageReferences(existingPkgs []string) []string

type LinkerOptFunc

type LinkerOptFunc func(options *LinkerOptions)

type LinkerOptions

type LinkerOptions struct {
	SymbolNameOrder                  []string
	RandomSymbolNameOrder            bool
	RelocationDebugWriter            io.Writer
	DumpTextBeforeAndAfterRelocs     bool
	NoRelocationEpilogues            bool
	SkipTypeDeduplicationForPackages []string
	ForceTestRelocationEpilogues     bool
}

type Type

type Type reflectlite.Type

func AsType

func AsType(_typ *_type) Type

type Value

type Value struct {
	reflectlite.Value
}