extism

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2024 License: BSD-3-Clause Imports: 19 Imported by: 20

README

Extism Go SDK

This repo houses the Go SDK for integrating with the Extism runtime. Install this library into your host Go applications to run Extism plugins.

Join the Extism Discord and chat with us!

Installation

Install via go get:

go get github.com/extism/go-sdk

Reference Docs

You can find the reference docs at https://pkg.go.dev/github.com/extism/go-sdk.

Getting Started

This guide should walk you through some of the concepts in Extism and this Go library.

Creating A Plug-in

The primary concept in Extism is the plug-in. You can think of a plug-in as a code module stored in a .wasm file.

Plug-in code can come from a file on disk, object storage or any number of places. Since you may not have one handy let's load a demo plug-in from the web. Let's start by creating a main func and loading an Extism Plug-in:

package main

import (
	"context"
	"fmt"
	"github.com/extism/go-sdk"
	"os"
)

func main() {
	manifest := extism.Manifest{
		Wasm: []extism.Wasm{
			extism.WasmUrl{
				Url: "https://github.com/extism/plugins/releases/latest/download/count_vowels.wasm",
			},
		},
	}

	ctx := context.Background()
	config := extism.PluginConfig{}
	plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})

	if err != nil {
		fmt.Printf("Failed to initialize plugin: %v\n", err)
		os.Exit(1)
	}
}

Note: See the Manifest docs as it has a rich schema and a lot of options.

Calling A Plug-in's Exports

This plug-in was written in Rust and it does one thing, it counts vowels in a string. As such, it exposes one "export" function: count_vowels. We can call exports using extism.Plugin.Call. Let's add that code to our main func:

func main() {
    // ...

	data := []byte("Hello, World!")
	exit, out, err := plugin.Call("count_vowels", data)
	if err != nil {
		fmt.Println(err)
		os.Exit(int(exit))
	}

	response := string(out)
	fmt.Println(response)
    // => {"count": 3, "total": 3, "vowels": "aeiouAEIOU"}
}

Running this should print out the JSON vowel count report:

$ go run main.go
# => {"count":3,"total":3,"vowels":"aeiouAEIOU"}

All exports have a simple interface of optional bytes in, and optional bytes out. This plug-in happens to take a string and return a JSON encoded string with a report of results.

Plug-in State

Plug-ins may be stateful or stateless. Plug-ins can maintain state between calls by the use of variables. Our count vowels plug-in remembers the total number of vowels it's ever counted in the "total" key in the result. You can see this by making subsequent calls to the export:

func main () {
    // ...

    exit, out, err := plugin.Call("count_vowels", []byte("Hello, World!"))
    if err != nil {
        fmt.Println(err)
        os.Exit(int(exit))
    }
    fmt.Println(string(out))
    // => {"count": 3, "total": 6, "vowels": "aeiouAEIOU"}

    exit, out, err = plugin.Call("count_vowels", []byte("Hello, World!"))
    if err != nil {
        fmt.Println(err)
        os.Exit(int(exit))
    }
    fmt.Println(string(out))
    // => {"count": 3, "total": 9, "vowels": "aeiouAEIOU"}
}

These variables will persist until this plug-in is freed or you initialize a new one.

Configuration

Plug-ins may optionally take a configuration object. This is a static way to configure the plug-in. Our count-vowels plugin takes an optional configuration to change out which characters are considered vowels. Example:

func main() {
    manifest := extism.Manifest{
        Wasm: []extism.Wasm{
            extism.WasmUrl{
                Url: "https://github.com/extism/plugins/releases/latest/download/count_vowels.wasm",
            },
        },
        Config: map[string]string{
            "vowels": "aeiouyAEIOUY",
        },
    }

    ctx := context.Background()
    config := extism.PluginConfig{}

    plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})

    if err != nil {
        fmt.Printf("Failed to initialize plugin: %v\n", err)
        os.Exit(1)
    }

    exit, out, err := plugin.Call("count_vowels", []byte("Yellow, World!"))
    if err != nil {
        fmt.Println(err)
        os.Exit(int(exit))
    }

    fmt.Println(string(out))
    // => {"count": 4, "total": 4, "vowels": "aeiouAEIOUY"}
}
Host Functions

Let's extend our count-vowels example a little bit: Instead of storing the total in an ephemeral plug-in var, let's store it in a persistent key-value store!

Wasm can't use our KV store on it's own. This is where Host Functions come in.

Host functions allow us to grant new capabilities to our plug-ins from our application. They are simply some Go functions you write which can be passed down and invoked from any language inside the plug-in.

Let's load the manifest like usual but load up this count_vowels_kvstore plug-in:

manifest := extism.Manifest{
    Wasm: []extism.Wasm{
        extism.WasmUrl{
            Url: "https://github.com/extism/plugins/releases/latest/download/count_vowels_kvstore.wasm",
        },
    },
}

Note: The source code for this is here and is written in rust, but it could be written in any of our PDK languages.

Unlike our previous plug-in, this plug-in expects you to provide host functions that satisfy our its import interface for a KV store.

We want to expose two functions to our plugin, kv_write(key string, value []bytes) which writes a bytes value to a key and kv_read(key string) []byte which reads the bytes at the given key.

// pretend this is Redis or something :)
kvStore := make(map[string][]byte)

kvRead := extism.NewHostFunctionWithStack(
    "kv_read",
    func(ctx context.Context, p *extism.CurrentPlugin, stack []uint64) {
        key, err := p.ReadString(stack[0])
        if err != nil {
            panic(err)
        }

        value, success := kvStore[key]
        if !success {
            value = []byte{0, 0, 0, 0}
        }

        stack[0], err = p.WriteBytes(value)
    },
    []ValueType{ValueTypePTR},
    []ValueType{ValueTypePTR},
)

kvWrite := extism.NewHostFunctionWithStack(
    "kv_write",
    func(ctx context.Context, p *extism.CurrentPlugin, stack []uint64) {
        key, err := p.ReadString(stack[0])
        if err != nil {
            panic(err)
        }

        value, err := p.ReadBytes(stack[1])
        if err != nil {
            panic(err)
        }

        kvStore[key] = value
    },
    []ValueType{ValueTypePTR, ValueTypePTR},
    []ValueType{},
)

Note: In order to write host functions you should get familiar with the methods on the extism.CurrentPlugin type. The p parameter is an instance of this type.

We need to pass these imports to the plug-in to create them. All imports of a plug-in must be satisfied for it to be initialized:

plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{kvRead, kvWrite});

Now we can invoke the event:

exit, out, err := plugin.Call("count_vowels", []byte("Hello, World!"))
// => Read from key=count-vowels"
// => Writing value=3 from key=count-vowels"
// => {"count": 3, "total": 3, "vowels": "aeiouAEIOU"}

exit, out, err = plugin.Call("count_vowels", []byte("Hello, World!"))
// => Read from key=count-vowels"
// => Writing value=6 from key=count-vowels"
// => {"count": 3, "total": 6, "vowels": "aeiouAEIOU"}
Enabling Compilation Cache

While Wazero (the underlying Wasm runtime) is very fast in initializing modules, you can make subsequent initializations even faster by enabling the compilation cache:

ctx := context.Background()
cache := wazero.NewCompilationCache()
defer cache.Close(ctx)

manifest := Manifest{Wasm: []Wasm{WasmFile{Path: "wasm/noop.wasm"}}}

config := PluginConfig{
    EnableWasi:    true,
    ModuleConfig:  wazero.NewModuleConfig(),
    RuntimeConfig: wazero.NewRuntimeConfig().WithCompilationCache(cache),
}

_, err := NewPlugin(ctx, manifest, config, []HostFunction{})

Build example plugins

Since our example plugins are also written in Go, for compiling them we use TinyGo:

cd plugins/config
tinygo build -target wasi -o ../wasm/config.wasm main.go

Documentation

Index

Constants

View Source 
const (
	// ValueTypeI32 is a 32-bit integer.
	ValueTypeI32 = api.ValueTypeI32
	// ValueTypeI64 is a 64-bit integer.
	ValueTypeI64 = api.ValueTypeI64
	// ValueTypeF32 is a 32-bit floating point number.
	ValueTypeF32 = api.ValueTypeF32
	// ValueTypeF64 is a 64-bit floating point number.
	ValueTypeF64 = api.ValueTypeF64
	// ValueTypePTR represents a pointer to an Extism memory block. Alias for ValueTypeI64
	ValueTypePTR = ValueTypeI64
)
View Source 
const (
	None runtimeType = iota
	Haskell
	Wasi
)

Variables

This section is empty.

Functions

func DecodeF32 

func DecodeF32(input uint64) float32

DecodeF32 decodes the input as a ValueTypeF32.

See EncodeF32

func DecodeF64 

func DecodeF64(input uint64) float64

DecodeF64 decodes the input as a ValueTypeF64.

See EncodeF64

func DecodeI32 

func DecodeI32(input uint64) int32

DecodeI32 decodes the input as a ValueTypeI32.

func DecodeU32 

func DecodeU32(input uint64) uint32

DecodeU32 decodes the input as a ValueTypeI32.

func EncodeF32 

func EncodeF32(input float32) uint64

EncodeF32 encodes the input as a ValueTypeF32.

See DecodeF32

func EncodeF64 

func EncodeF64(input float64) uint64

EncodeF64 encodes the input as a ValueTypeF64.

See EncodeF32

func EncodeI32 

func EncodeI32(input int32) uint64

EncodeI32 encodes the input as a ValueTypeI32.

func EncodeI64 

func EncodeI64(input int64) uint64

EncodeI64 encodes the input as a ValueTypeI64.

func EncodeU32 

func EncodeU32(input uint32) uint64

EncodeU32 encodes the input as a ValueTypeI32.

Types

type CurrentPlugin 

type CurrentPlugin struct {
	// contains filtered or unexported fields
}

func (*CurrentPlugin) Alloc 

func (p *CurrentPlugin) Alloc(n uint64) (uint64, error)

Alloc a new memory block of the given length, returning its offset

func (*CurrentPlugin) Free 

func (p *CurrentPlugin) Free(offset uint64) error

Free the memory block specified by the given offset

func (*CurrentPlugin) Length 

func (p *CurrentPlugin) Length(offs uint64) (uint64, error)

Length returns the number of bytes allocated at the specified offset

func (*CurrentPlugin) Log 

func (p *CurrentPlugin) Log(level LogLevel, message string)

func (*CurrentPlugin) Logf 

func (p *CurrentPlugin) Logf(level LogLevel, format string, args ...any)

func (*CurrentPlugin) Memory 

func (p *CurrentPlugin) Memory() api.Memory

Memory returns the plugin's WebAssembly memory interface.

func (*CurrentPlugin) ReadBytes 

func (p *CurrentPlugin) ReadBytes(offset uint64) ([]byte, error)

ReadBytes reads a byte array from memory

func (*CurrentPlugin) ReadString 

func (p *CurrentPlugin) ReadString(offset uint64) (string, error)

ReadString reads a string from wasm memory

func (*CurrentPlugin) WriteBytes 

func (p *CurrentPlugin) WriteBytes(b []byte) (uint64, error)

WriteBytes writes a string to wasm memory and return the offset

func (*CurrentPlugin) WriteString 

func (p *CurrentPlugin) WriteString(s string) (uint64, error)

Write a string to wasm memory and return the offset

type HostFunction 

type HostFunction struct {
	Name      string
	Namespace string
	Params    []api.ValueType
	Returns   []api.ValueType
	// contains filtered or unexported fields
}

HostFunction represents a custom function defined by the host.

func NewHostFunctionWithStack 

func NewHostFunctionWithStack(
	name string,
	callback HostFunctionStackCallback,
	params []ValueType,
	returnTypes []ValueType) HostFunction

NewHostFunctionWithStack creates a new instance of a HostFunction, which is designed to provide custom functionality in a given host environment. Here's an example multiplication function that loads operands from memory: 

 mult := NewHostFunctionWithStack(
	"mult",
	func(ctx context.Context, plugin *CurrentPlugin, stack []uint64) {
		a := DecodeI32(stack[0])
		b := DecodeI32(stack[1])

		stack[0] = EncodeI32(a * b)
	},
	[]ValueType{ValueTypeI64, ValueTypeI64},
	ValueTypeI64
 )

func (*HostFunction) SetNamespace 

func (f *HostFunction) SetNamespace(namespace string)

type HostFunctionStackCallback 

type HostFunctionStackCallback func(ctx context.Context, p *CurrentPlugin, stack []uint64)

HostFunctionStackCallback is a Function implemented in Go instead of a wasm binary. The plugin parameter is the calling plugin, used to access memory or exported functions and logging.

The stack is includes any parameters encoded according to their ValueType. Its length is the max of parameter or result length. When there are results, write them in order beginning at index zero. Do not use the stack after the function returns.

Here's a typical way to read three parameters and write back one. 

// read parameters in index order
argv, argvBuf := DecodeU32(inputs[0]), DecodeU32(inputs[1])

// write results back to the stack in index order
stack[0] = EncodeU32(ErrnoSuccess)

This function can be non-deterministic or cause side effects. It also has special properties not defined in the WebAssembly Core specification. Notably, this uses the caller's memory (via Module.Memory). See https://www.w3.org/TR/wasm-core-1/#host-functions%E2%91%A0

To safely decode/encode values from/to the uint64 inputs/ouputs, users are encouraged to use Extism's EncodeXXX or DecodeXXX functions.

type HttpRequest 

type HttpRequest struct {
	Url     string
	Headers map[string]string
	Method  string
}

HttpRequest represents an HTTP request to be made by the plugin.

type LogLevel 

type LogLevel uint8

LogLevel defines different log levels. 

const (
	LogLevelOff LogLevel
	LogLevelError
	LogLevelWarn
	LogLevelInfo
	LogLevelDebug
	LogLevelTrace
)

func (LogLevel) String 

func (l LogLevel) String() string

type Manifest 

type Manifest struct {
	Wasm   []Wasm `json:"wasm"`
	Memory struct {
		MaxPages uint32 `json:"max_pages,omitempty"`
	} `json:"memory,omitempty"`
	Config       map[string]string `json:"config,omitempty"`
	AllowedHosts []string          `json:"allowed_hosts,omitempty"`
	AllowedPaths map[string]string `json:"allowed_paths,omitempty"`
	Timeout      uint64            `json:"timeout_ms,omitempty"`
}

Manifest represents the plugin's manifest, including Wasm modules and configuration. See https://extism.org/docs/concepts/manifest for schema.

func (*Manifest) UnmarshalJSON 

func (m *Manifest) UnmarshalJSON(data []byte) error

type Plugin 

type Plugin struct {
	Runtime *Runtime
	Modules map[string]api.Module
	Main    api.Module
	Timeout time.Duration
	Config  map[string]string
	// NOTE: maybe we can have some nice methods for getting/setting vars
	Var            map[string][]byte
	AllowedHosts   []string
	AllowedPaths   map[string]string
	LastStatusCode int
	// contains filtered or unexported fields
}

Plugin is used to call WASM functions

func NewPlugin 

func NewPlugin(
	ctx context.Context,
	manifest Manifest,
	config PluginConfig,
	functions []HostFunction) (*Plugin, error)

NewPlugin creates a new Extism plugin with the given manifest, configuration, and host functions. The returned plugin can be used to call WebAssembly functions and interact with the plugin.

func (*Plugin) Call 

func (plugin *Plugin) Call(name string, data []byte) (uint32, []byte, error)

Call a function by name with the given input, returning the output

func (*Plugin) Close 

func (p *Plugin) Close() error

Close closes the plugin by freeing the underlying resources.

func (*Plugin) FunctionExists 

func (plugin *Plugin) FunctionExists(name string) bool

FunctionExists returns true when the named function is present in the plugin's main module

func (*Plugin) GetError 

func (plugin *Plugin) GetError() string

GetError retrieves the error message from the last WebAssembly function call, if any.

func (*Plugin) GetOutput 

func (plugin *Plugin) GetOutput() ([]byte, error)

GetOutput retrieves the output data from the last WebAssembly function call.

func (*Plugin) Log 

func (p *Plugin) Log(level LogLevel, message string)

func (*Plugin) Logf 

func (p *Plugin) Logf(level LogLevel, format string, args ...any)

func (*Plugin) Memory 

func (plugin *Plugin) Memory() api.Memory

Memory returns the plugin's WebAssembly memory interface.

func (*Plugin) SetInput 

func (plugin *Plugin) SetInput(data []byte) (uint64, error)

SetInput sets the input data for the plugin to be used in the next WebAssembly function call.

func (*Plugin) SetLogLevel 

func (p *Plugin) SetLogLevel(level LogLevel)

SetLogLevel sets the minim logging level, applies to custom logging callbacks too

func (*Plugin) SetLogger 

func (p *Plugin) SetLogger(logger func(LogLevel, string))

SetLogger sets a custom logging callback

type PluginConfig 

type PluginConfig struct {
	ModuleConfig  wazero.ModuleConfig
	RuntimeConfig wazero.RuntimeConfig
	EnableWasi    bool
	LogLevel      LogLevel
}

PluginConfig contains configuration options for the Extism plugin.

type Runtime 

type Runtime struct {
	Wazero wazero.Runtime
	Extism api.Module
	Env    api.Module
	// contains filtered or unexported fields
}

Runtime represents the Extism plugin's runtime environment, including the underlying Wazero runtime and modules.

type ValueType 

type ValueType = byte

type Wasm 

type Wasm interface {
	ToWasmData(ctx context.Context) (WasmData, error)
}

Wasm is an interface that represents different ways of providing WebAssembly data.

type WasmData 

type WasmData struct {
	Data []byte `json:"data"`
	Hash string `json:"hash,omitempty"`
	Name string `json:"name,omitempty"`
}

WasmData represents in-memory WebAssembly data, including its content, hash, and name.

func (WasmData) ToWasmData 

func (d WasmData) ToWasmData(ctx context.Context) (WasmData, error)

type WasmFile 

type WasmFile struct {
	Path string `json:"path"`
	Hash string `json:"hash,omitempty"`
	Name string `json:"name,omitempty"`
}

WasmFile represents WebAssembly data that needs to be loaded from a file.

func (WasmFile) ToWasmData 

func (f WasmFile) ToWasmData(ctx context.Context) (WasmData, error)

type WasmUrl 

type WasmUrl struct {
	Url     string            `json:"url"`
	Hash    string            `json:"hash,omitempty"`
	Headers map[string]string `json:"headers,omitempty"`
	Name    string            `json:"name,omitempty"`
	Method  string            `json:"method,omitempty"`
}

WasmUrl represents WebAssembly data that needs to be fetched from a URL.

func (WasmUrl) ToWasmData 

func (u WasmUrl) ToWasmData(ctx context.Context) (WasmData, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.