goloader

package module
v0.0.0-...-d48c41c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 23, 2024 License: Apache-2.0 Imports: 36 Imported by: 0

README

Goloader/JIT Compiler for Go

Build Status

The goloader/jit package can compile and load Go code from text, file, folder or remote package (including code with package imports).

It automatically resolves package dependencies recursively, and provides a type safe way of interacting with the built functions.

Forked from dearplain and pkujhd.

Usage

Build

Make sure you're using go >= 1.18.

First, execute the following command. This is because Goloader relies on the internal package, which is forbidden by the Go compiler.

cp -r $GOROOT/src/cmd/internal $GOROOT/src/cmd/objfile

Go compiler patch

To allow the loader to know the types of exported functions, this package will attempt to patch the Go compiler (gc) to emit these if not already patched.

The effect of the patch can be found in jit/gc.patch.

go install github.com/eh-steve/goloader/jit/patchgc@latest
# You may need to run patchgc as sudo if your $GOROOT is owned by root
# (alternatively `chown -R $USER:$USER $GOROOT`)
patchgc

Example Usage

package main

import (
	"fmt"
	"github.com/eh-steve/goloader/jit"
)

func main() {
	conf := jit.BuildConfig{
		KeepTempFiles:   false,          // Files are copied/written to a temp dir to ensure it is writable. This retains the temporary copies
		ExtraBuildFlags: []string{"-x"}, // Flags passed to go build command
		BuildEnv:        nil,            // Env vars to set for go build toolchain
		TmpDir:          "",             // To control where temporary files are copied
		DebugLog:        true,           //
	}

	loadable, err := jit.BuildGoFiles(conf, "./path/to/file1.go", "/path/to/file2.go")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoPackage(conf, "./path/to/package")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoPackageRemote(conf, "github.com/some/package/v4", "latest")
	if err != nil {
		panic(err)
	}
	// or
	loadable, err = jit.BuildGoText(conf, `
package mypackage

import "encoding/json"

func MyFunc(input []byte) (interface{}, error) {
	var output interface{}
	err := json.Unmarshal(input, &output)
	return output, err
}
`)

	if err != nil {
		panic(err)
	}

	module, err := loadable.Load()
	// module.SymbolsByPkg is a map[string]map[string]interface{} of all packages and their exported functions and global vars
	symbols := module.SymbolsByPkg[loadable.ImportPath]
	if err != nil {
		panic(err)
	}
	defer func() {
		err = module.Unload()
		if err != nil {
			panic(err)
		}
	}()
	switch f := symbols["MyFunc"].(type) {
	case func([]byte) (interface{}, error):
		result, err := f([]byte(`{"k":"v"}`))
		if err != nil {
			panic(err)
		}
		fmt.Println(result)
	default:
		panic("Function signature was not what was expected")
	}
}

How does it work?

Goloader works like a linker, it relocates the addresses of symbols in an object file, generates runnable code, and then reuses the runtime functions and the type pointers of the loader where available.

Goloader provides some information to the runtime and garbage collector of Go, which allows it to work correctly with them.

Please note that Goloader is not a scripting engine. It reads the archives emitted from the Go compiler and makes them runnable. All features of Go are supported, and run just as fast and lightweight as native Go code.

Comparison with plugin

Plugin:

  • Can't load plugins not built with exact same versions of packages that host binary uses (plugin was built with a different version of package) - this makes them basically unusable in most large projects
  • Introduces dependency on libdl/CGo (and doesn't work on Windows)
  • Prevents linker deadcode elimination for unreachable methods (increases host binary size with unused methods)
  • Can't be unloaded/dynamically updated
  • Duplicates a lot of the go runtime (large binary sizes)

Goloader:

  • Can build/load any packages (somewhat unsafely - it attempts to verify that types across JIT packages and host packages match, but doesn't do the same checks for function signatures)
  • Pure Go - no dependency on libdl/Cgo
  • Patches host itabs containing unreachable methods instead of preventing linker deadcode elimination
  • Can be unloaded, and objects from one version of a JIT package can be converted at runtime to those from another version, to allow dynamic adjustment of functions/methods without losing state
  • Reuses the runtime from the host binary (much smaller binaries)

Goloader supports pprof tool (yes, you can see code loaded by Goloader in pprof), but does not (yet) support debugging with delve.

OS/Arch Compatibility

JIT compiler tested/passing on:

OS/Arch amd64/+CGo arm64/+CGo amd64/-CGo arm64/-CGo
Linux/go-1.20.3 ✔ ✔ ✔ ✔
Darwin/go-1.20.3 ✔ ✔ ✔ ✔
Windows/go-1.20.3 ✔ ⁉ ✔ ⁉
Linux/go-1.19.4 ✔ ✔ ✔ ✔
Darwin/go-1.19.4 ✔ ✔ ✔ ✔
Windows/go-1.19.4 ✔ ⁉ ✔ ⁉
Linux/go-1.18.8 ✔ ✔ ✔ ✔
Darwin/go-1.18.8 ✔ ✔ ✔ ✔
Windows/go-1.18.8 ❌ ⁉ ✔ ⁉

Warning

Don't use "-s -w" compile argument, It strips symbol table.

Documentation

Index

Constants

View Source 
const (
	PtrSize    = 4 << (^uintptr(0) >> 63)
	Uint32Size = int(unsafe.Sizeof(uint32(0)))
	IntSize    = int(unsafe.Sizeof(int(0)))
	UInt64Size = int(unsafe.Sizeof(uint64(0)))

	FindFuncBucketSize = int(unsafe.Sizeof(findfuncbucket{}))
	InvalidHandleValue = ^uintptr(0)
	InvalidOffset      = int(-1)
	PageSize           = 1 << 12 //4096
)

size

View Source 
const (
	EmptyString    = ""
	DefaultPkgPath = "main"
	ZeroByte       = byte(0x00)
)
View Source 
const (
	FileSymPrefix              = "gofile.."
	MainPkgPrefix              = "main."
	OsStdout                   = "os.Stdout"
	FirstModulePrefix          = "firstmodule."
	DefaultStringContainerSize = 1024 * 1024 * 16
)

string match prefix/suffix

View Source 
const (
	TypeImportPathPrefix = "type:.importpath."
	TypeDoubleDotPrefix  = "type:."
	TypePrefix           = "type:"
	ItabPrefix           = "go:itab."
	TypeStringPrefix     = "go:string."
	ObjSymbolSeparator   = ":"
)
View Source 
const (
	Invalid       = reflectlite.Invalid
	Bool          = reflectlite.Bool
	Int           = reflectlite.Int
	Int8          = reflectlite.Int8
	Int16         = reflectlite.Int16
	Int32         = reflectlite.Int32
	Int64         = reflectlite.Int64
	Uint          = reflectlite.Uint
	Uint8         = reflectlite.Uint8
	Uint16        = reflectlite.Uint16
	Uint32        = reflectlite.Uint32
	Uint64        = reflectlite.Uint64
	Uintptr       = reflectlite.Uintptr
	Float32       = reflectlite.Float32
	Float64       = reflectlite.Float64
	Complex64     = reflectlite.Complex64
	Complex128    = reflectlite.Complex128
	Array         = reflectlite.Array
	Chan          = reflectlite.Chan
	Func          = reflectlite.Func
	Interface     = reflectlite.Interface
	Map           = reflectlite.Map
	Pointer       = reflectlite.Pointer
	Slice         = reflectlite.Slice
	String        = reflectlite.String
	Struct        = reflectlite.Struct
	UnsafePointer = reflectlite.UnsafePointer
	Ptr           = reflectlite.Ptr
)
View Source 
const (
	EmptyPkgPath = "<unlinkable>"
)
View Source 
const (
	KindGCProg = 1 << 6
)
View Source 
const (
	RuntimeDeferReturn = "runtime.deferreturn"
)

runtime symbol

View Source 
const (
	TLSNAME = "(TLS)"
)

Variables

View Source 
var Indirect = reflectlite.Indirect
View Source 
var MakeMapWithSize = reflectlite.MakeMapWithSize
View Source 
var New = reflectlite.New
View Source 
var NewAt = reflectlite.NewAt
View Source 
var TypeOf = reflectlite.TypeOf
View Source 
var ValueOf = reflectlite.ValueOf

Functions

func AsRType 

func AsRType(_typ *_type) reflect.Type

func CanAttemptConversion 

func CanAttemptConversion(oldValue interface{}, newType reflect.Type) bool

func CastToFuncUnsafe 

func CastToFuncUnsafe[T any](addr uintptr) T

func ConvertTypesAcrossModules 

func ConvertTypesAcrossModules(oldModule, newModule *CodeModule, oldValue interface{}, newType reflect.Type) (res interface{}, err error)

func FuncPCsABI0 

func FuncPCsABI0(abiInternalPCs []uintptr) []uintptr

func MakeThreadJITCodeExecutable 

func MakeThreadJITCodeExecutable(ptr uintptr, len int)

func Mmap 

func Mmap(size int) ([]byte, error)

func MmapData 

func MmapData(size int) ([]byte, error)

func Munmap 

func Munmap(b []byte) (err error)

func Parse 

func Parse(f *os.File, pkgpath *string) ([]string, error)

func RegSymbol 

func RegSymbol(symPtr map[string]uintptr, pkgSet map[string]struct{}) error

func RegSymbolWithSo 

func RegSymbolWithSo(symPtr map[string]uintptr, pkgSet map[string]struct{}, path string) error

func RegTypes 

func RegTypes(symPtr map[string]uintptr, interfaces ...interface{})

func WithDumpTextBeforeAndAfterRelocs 

func WithDumpTextBeforeAndAfterRelocs() func(*LinkerOptions)

func WithForceTestRelocationEpilogues 

func WithForceTestRelocationEpilogues() func(*LinkerOptions)

func WithNoRelocationEpilogues 

func WithNoRelocationEpilogues() func(*LinkerOptions)

func WithRandomSymbolNameOrder 

func WithRandomSymbolNameOrder() func(*LinkerOptions)

func WithRelocationDebugWriter 

func WithRelocationDebugWriter(writer io.Writer) func(*LinkerOptions)

func WithSkipTypeDeduplicationForPackages 

func WithSkipTypeDeduplicationForPackages(packages []string) func(*LinkerOptions)

func WithSymbolNameOrder 

func WithSymbolNameOrder(symNames []string) func(*LinkerOptions)

WithSymbolNameOrder allows you to control the sequence (placement in memory) of symbols from an object file. When not set, the order as parsed from the archive file is used.

Types

type CodeModule 

type CodeModule struct {
	SymbolsByPkg map[string]map[string]interface{}
	Syms         map[string]uintptr
	// contains filtered or unexported fields
}

func Load 

func Load(linker *Linker, symPtr map[string]uintptr) (codeModule *CodeModule, err error)

func (*CodeModule) DataAddr 

func (cm *CodeModule) DataAddr() (start, end uintptr)

func (*CodeModule) TextAddr 

func (cm *CodeModule) TextAddr() (start, end uintptr)

func (*CodeModule) Unload 

func (cm *CodeModule) Unload() error

type Linker 

type Linker struct {
	Arch *sys.Arch
	// contains filtered or unexported fields
}

func ReadObjs 

func ReadObjs(files []string, pkgPath []string, globalSymPtr map[string]uintptr, linkerOpts ...LinkerOptFunc) (*Linker, error)

func (*Linker) Autolib 

func (linker *Linker) Autolib() []string

func (*Linker) Opts 

func (linker *Linker) Opts(linkerOpts ...LinkerOptFunc)

func (*Linker) SymbolOrder 

func (linker *Linker) SymbolOrder() []string

func (*Linker) UnloadStrings 

func (linker *Linker) UnloadStrings()

func (*Linker) UnresolvedExternalSymbolUsers 

func (linker *Linker) UnresolvedExternalSymbolUsers(symbolMap map[string]uintptr) map[string][]string

func (*Linker) UnresolvedExternalSymbols 

func (linker *Linker) UnresolvedExternalSymbols(symbolMap map[string]uintptr, ignorePackages []string, stdLibPkgs map[string]struct{}, unsafeBlindlyUseFirstModuleTypes bool) map[string]*obj.Sym

func (*Linker) UnresolvedPackageReferences 

func (linker *Linker) UnresolvedPackageReferences(existingPkgs []string) []string

type LinkerOptFunc 

type LinkerOptFunc func(options *LinkerOptions)

type LinkerOptions 

type LinkerOptions struct {
	SymbolNameOrder                  []string
	RandomSymbolNameOrder            bool
	RelocationDebugWriter            io.Writer
	DumpTextBeforeAndAfterRelocs     bool
	NoRelocationEpilogues            bool
	SkipTypeDeduplicationForPackages []string
	ForceTestRelocationEpilogues     bool
}

type Type 

type Type reflectlite.Type

func AsType 

func AsType(_typ *_type) Type

type Value 

type Value struct {
	reflectlite.Value
}

Source Files

View all Source files

Directories

Path Synopsis
jit
patchgc
libc_cgo
mapping
vmmap
objabi
dataindex
reloctype
symkind
tls
reflectlite
internal/goarch
package goarch contains GOARCH-specific constants.
 package goarch contains GOARCH-specific constants.
internal/itoa
internal/unsafeheader
Package unsafeheader contains header declarations for the Go runtime's slice and string implementations.
 Package unsafeheader contains header declarations for the Go runtime's slice and string implementations.
reflectlite1.18
Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types.
 Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types.
reflectlite1.19
unload
jsonunload
protobufunload

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.