goloader

package module
v0.0.0-...-d48c41c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 23, 2024 License: Apache-2.0 Imports: 36 Imported by: 0

README

Goloader/JIT Compiler for Go

Build Status

The goloader/jit package can compile and load Go code from text, file, folder or remote package (including code with package imports).

It automatically resolves package dependencies recursively, and provides a type safe way of interacting with the built functions.

Forked from dearplain and pkujhd.

Usage

Build

Make sure you're using go >= 1.18.

First, execute the following command. This is because Goloader relies on the internal package, which is forbidden by the Go compiler.

cp -r $GOROOT/src/cmd/internal $GOROOT/src/cmd/objfile

Go compiler patch

To allow the loader to know the types of exported functions, this package will attempt to patch the Go compiler (gc) to emit these if not already patched.

The effect of the patch can be found in jit/gc.patch.

go install github.com/eh-steve/goloader/jit/patchgc@latest
# You may need to run patchgc as sudo if your $GOROOT is owned by root
# (alternatively `chown -R $USER:$USER $GOROOT`)
patchgc

Example Usage

package main

import (
	"fmt"
	"github.com/eh-steve/goloader/jit"
)

func main() {
	conf := jit.BuildConfig{
		KeepTempFiles:   false,          // Files are copied/written to a temp dir to ensure it is writable. This retains the temporary copies
		ExtraBuildFlags: []string{"-x"}, // Flags passed to go build command
		BuildEnv:        nil,            // Env vars to set for go build toolchain
		TmpDir:          "",             // To control where temporary files are copied
		DebugLog:        true,           //
	}

	loadable, err := jit.BuildGoFiles(conf, "./path/to/file1.go", "/path/to/file2.go")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoPackage(conf, "./path/to/package")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoPackageRemote(conf, "github.com/some/package/v4", "latest")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoText(conf, `
package mypackage

import "encoding/json"

func MyFunc(input []byte) (interface{}, error) {
	var output interface{}
	err := json.Unmarshal(input, &output)
	return output, err
}
`)

	if err != nil {
		panic(err)
	}

	module, err := loadable.Load()
	// module.SymbolsByPkg is a map[string]map[string]interface{} of all packages and their exported functions and global vars
	symbols := module.SymbolsByPkg[loadable.ImportPath]
	if err != nil {
		panic(err)
	}
	defer func() {
		err = module.Unload()
		if err != nil {
			panic(err)
		}
	}()
	switch f := symbols["MyFunc"].(type) {
	case func([]byte) (interface{}, error):
		result, err := f([]byte(`{"k":"v"}`))
		if err != nil {
			panic(err)
		}
		fmt.Println(result)
	default:
		panic("Function signature was not what was expected")
	}
}

How does it work?

Goloader works like a linker, it relocates the addresses of symbols in an object file, generates runnable code, and then reuses the runtime functions and the type pointers of the loader where available.

Goloader provides some information to the runtime and garbage collector of Go, which allows it to work correctly with them.

Please note that Goloader is not a scripting engine. It reads the archives emitted from the Go compiler and makes them runnable. All features of Go are supported, and run just as fast and lightweight as native Go code.

Comparison with plugin

Plugin:

  • Can't load plugins not built with exact same versions of packages that host binary uses (plugin was built with a different version of package) - this makes them basically unusable in most large projects
  • Introduces dependency on libdl/CGo (and doesn't work on Windows)
  • Prevents linker deadcode elimination for unreachable methods (increases host binary size with unused methods)
  • Can't be unloaded/dynamically updated
  • Duplicates a lot of the go runtime (large binary sizes)

Goloader:

  • Can build/load any packages (somewhat unsafely - it attempts to verify that types across JIT packages and host packages match, but doesn't do the same checks for function signatures)
  • Pure Go - no dependency on libdl/Cgo
  • Patches host itabs containing unreachable methods instead of preventing linker deadcode elimination
  • Can be unloaded, and objects from one version of a JIT package can be converted at runtime to those from another version, to allow dynamic adjustment of functions/methods without losing state
  • Reuses the runtime from the host binary (much smaller binaries)

Goloader supports pprof tool (yes, you can see code loaded by Goloader in pprof), but does not (yet) support debugging with delve.

OS/Arch Compatibility

JIT compiler tested/passing on:

OS/Arch amd64/+CGo arm64/+CGo amd64/-CGo arm64/-CGo
Linux/go-1.20.3
Darwin/go-1.20.3
Windows/go-1.20.3
Linux/go-1.19.4
Darwin/go-1.19.4
Windows/go-1.19.4
Linux/go-1.18.8
Darwin/go-1.18.8
Windows/go-1.18.8

Warning

Don't use "-s -w" compile argument, It strips symbol table.

Documentation

Index

Constants

View Source
const (
	PtrSize    = 4 << (^uintptr(0) >> 63)
	Uint32Size = int(unsafe.Sizeof(uint32(0)))
	IntSize    = int(unsafe.Sizeof(int(0)))
	UInt64Size = int(unsafe.Sizeof(uint64(0)))

	FindFuncBucketSize = int(unsafe.Sizeof(findfuncbucket{}))
	InvalidHandleValue = ^uintptr(0)
	InvalidOffset      = int(-1)
	PageSize           = 1 << 12 //4096
)

size

View Source
const (
	EmptyString    = ""
	DefaultPkgPath = "main"
	ZeroByte       = byte(0x00)
)
View Source
const (
	FileSymPrefix              = "gofile.."
	MainPkgPrefix              = "main."
	OsStdout                   = "os.Stdout"
	FirstModulePrefix          = "firstmodule."
	DefaultStringContainerSize = 1024 * 1024 * 16
)

string match prefix/suffix

View Source
const (
	TypeImportPathPrefix = "type:.importpath."
	TypeDoubleDotPrefix  = "type:."
	TypePrefix           = "type:"
	ItabPrefix           = "go:itab."
	TypeStringPrefix     = "go:string."
	ObjSymbolSeparator   = ":"
)
View Source
const (
	Invalid       = reflectlite.Invalid
	Bool          = reflectlite.Bool
	Int           = reflectlite.Int
	Int8          = reflectlite.Int8
	Int16         = reflectlite.Int16
	Int32         = reflectlite.Int32
	Int64         = reflectlite.Int64
	Uint          = reflectlite.Uint
	Uint8         = reflectlite.Uint8
	Uint16        = reflectlite.Uint16
	Uint32        = reflectlite.Uint32
	Uint64        = reflectlite.Uint64
	Uintptr       = reflectlite.Uintptr
	Float32       = reflectlite.Float32
	Float64       = reflectlite.Float64
	Complex64     = reflectlite.Complex64
	Complex128    = reflectlite.Complex128
	Array         = reflectlite.Array
	Chan          = reflectlite.Chan
	Func          = reflectlite.Func
	Interface     = reflectlite.Interface
	Map           = reflectlite.Map
	Pointer       = reflectlite.Pointer
	Slice         = reflectlite.Slice
	String        = reflectlite.String
	Struct        = reflectlite.Struct
	UnsafePointer = reflectlite.UnsafePointer
	Ptr           = reflectlite.Ptr
)
View Source
const (
	EmptyPkgPath = "<unlinkable>"
)
View Source
const (
	KindGCProg = 1 << 6
)
View Source
const (
	RuntimeDeferReturn = "runtime.deferreturn"
)

runtime symbol

View Source
const (
	TLSNAME = "(TLS)"
)

Variables

View Source
var Indirect = reflectlite.Indirect
View Source
var MakeMapWithSize = reflectlite.MakeMapWithSize
View Source
var New = reflectlite.New
View Source
var NewAt = reflectlite.NewAt
View Source
var TypeOf = reflectlite.TypeOf
View Source
var ValueOf = reflectlite.ValueOf

Functions

func AsRType

func AsRType(_typ *_type) reflect.Type

func CanAttemptConversion

func CanAttemptConversion(oldValue interface{}, newType reflect.Type) bool

func CastToFuncUnsafe

func CastToFuncUnsafe[T any](addr uintptr) T

func ConvertTypesAcrossModules

func ConvertTypesAcrossModules(oldModule, newModule *CodeModule, oldValue interface{}, newType reflect.Type) (res interface{}, err error)

func FuncPCsABI0

func FuncPCsABI0(abiInternalPCs []uintptr) []uintptr

func MakeThreadJITCodeExecutable

func MakeThreadJITCodeExecutable(ptr uintptr, len int)

func Mmap

func Mmap(size int) ([]byte, error)

func MmapData

func MmapData(size int) ([]byte, error)

func Munmap

func Munmap(b []byte) (err error)

func Parse

func Parse(f *os.File, pkgpath *string) ([]string, error)

func RegSymbol

func RegSymbol(symPtr map[string]uintptr, pkgSet map[string]struct{}) error

func RegSymbolWithSo

func RegSymbolWithSo(symPtr map[string]uintptr, pkgSet map[string]struct{}, path string) error

func RegTypes

func RegTypes(symPtr map[string]uintptr, interfaces ...interface{})

func WithDumpTextBeforeAndAfterRelocs

func WithDumpTextBeforeAndAfterRelocs() func(*LinkerOptions)

func WithForceTestRelocationEpilogues

func WithForceTestRelocationEpilogues() func(*LinkerOptions)

func WithNoRelocationEpilogues

func WithNoRelocationEpilogues() func(*LinkerOptions)

func WithRandomSymbolNameOrder

func WithRandomSymbolNameOrder() func(*LinkerOptions)

func WithRelocationDebugWriter

func WithRelocationDebugWriter(writer io.Writer) func(*LinkerOptions)

func WithSkipTypeDeduplicationForPackages

func WithSkipTypeDeduplicationForPackages(packages []string) func(*LinkerOptions)

func WithSymbolNameOrder

func WithSymbolNameOrder(symNames []string) func(*LinkerOptions)

WithSymbolNameOrder allows you to control the sequence (placement in memory) of symbols from an object file. When not set, the order as parsed from the archive file is used.

Types

type CodeModule

type CodeModule struct {
	SymbolsByPkg map[string]map[string]interface{}
	Syms         map[string]uintptr
	// contains filtered or unexported fields
}

func Load

func Load(linker *Linker, symPtr map[string]uintptr) (codeModule *CodeModule, err error)

func (*CodeModule) DataAddr

func (cm *CodeModule) DataAddr() (start, end uintptr)

func (*CodeModule) TextAddr

func (cm *CodeModule) TextAddr() (start, end uintptr)

func (*CodeModule) Unload

func (cm *CodeModule) Unload() error

type Linker

type Linker struct {
	Arch *sys.Arch
	// contains filtered or unexported fields
}

func ReadObjs

func ReadObjs(files []string, pkgPath []string, globalSymPtr map[string]uintptr, linkerOpts ...LinkerOptFunc) (*Linker, error)

func (*Linker) Autolib

func (linker *Linker) Autolib() []string

func (*Linker) Opts

func (linker *Linker) Opts(linkerOpts ...LinkerOptFunc)

func (*Linker) SymbolOrder

func (linker *Linker) SymbolOrder() []string

func (*Linker) UnloadStrings

func (linker *Linker) UnloadStrings()

func (*Linker) UnresolvedExternalSymbolUsers

func (linker *Linker) UnresolvedExternalSymbolUsers(symbolMap map[string]uintptr) map[string][]string

func (*Linker) UnresolvedExternalSymbols

func (linker *Linker) UnresolvedExternalSymbols(symbolMap map[string]uintptr, ignorePackages []string, stdLibPkgs map[string]struct{}, unsafeBlindlyUseFirstModuleTypes bool) map[string]*obj.Sym

func (*Linker) UnresolvedPackageReferences

func (linker *Linker) UnresolvedPackageReferences(existingPkgs []string) []string

type LinkerOptFunc

type LinkerOptFunc func(options *LinkerOptions)

type LinkerOptions

type LinkerOptions struct {
	SymbolNameOrder                  []string
	RandomSymbolNameOrder            bool
	RelocationDebugWriter            io.Writer
	DumpTextBeforeAndAfterRelocs     bool
	NoRelocationEpilogues            bool
	SkipTypeDeduplicationForPackages []string
	ForceTestRelocationEpilogues     bool
}

type Type

type Type reflectlite.Type

func AsType

func AsType(_typ *_type) Type

type Value

type Value struct {
	reflectlite.Value
}

Directories

Path Synopsis
jit
objabi
tls
reflectlite
internal/goarch
package goarch contains GOARCH-specific constants.
package goarch contains GOARCH-specific constants.
internal/unsafeheader
Package unsafeheader contains header declarations for the Go runtime's slice and string implementations.
Package unsafeheader contains header declarations for the Go runtime's slice and string implementations.
reflectlite1.18
Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types.
Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types.
unload

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL