yomo

README

YoMo

YoMo is an open-source LLM Function Calling Framework for building Geo-distributed AI applications. Built atop QUIC Transport Protocol and Stateful Serverless architecture, makes your AI application low-latency, reliable, secure, and easy.

💚 We care about: Customer Experience in the Age of AI

🌶 Features

	Features
⚡️	Low-latency Guaranteed by implementing atop QUIC QUIC
🔐	Security TLS v1.3 on every data packet by design
📸	Stateful Serverless Make your GPU serverless 10x faster
🌎	Geo-Distributed Architecture Brings AI inference closer to end users
🚀	Y3 a faster than real-time codec

🚀 Getting Started

Let's implement a function calling with sfn-currency-converter:

Step 1. Install CLI

curl -fsSL https://get.yomo.run | sh

Verify if the CLI was installed successfully

yomo version

Step 2. Start the server

Prepare the configuration as my-agent.yaml

name: ai-zipper
host: 0.0.0.0
port: 9000

auth:
  type: token
  token: SECRET_TOKEN

bridge:
  ai:
    server:
      addr: 0.0.0.0:8000 ## Restful API endpoint
      provider: openai ## LLM API Service we will use

    providers:
      azopenai:
        api_endpoint: https://<RESOURCE>.openai.azure.com
        deployment_id: <DEPLOYMENT_ID>
        api_key: <API_KEY>
        api_version: <API_VERSION>

      openai:
        api_key: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxx
        model: gpt-4-1106-preview

      gemini:
        api_key: <GEMINI_API_KEY>

      cloudflare_azure:
        endpoint: https://gateway.ai.cloudflare.com/v1/<CF_GATEWAY_ID>/<CF_GATEWAY_NAME>
        api_key: <AZURE_API_KEY>
        resource: <AZURE_OPENAI_RESOURCE>
        deployment_id: <AZURE_OPENAI_DEPLOYMENT_ID>
        api_version: 2023-12-01-preview

Start the server:

YOMO_LOG_LEVEL=debug yomo serve -c my-agent.yaml

Step 3. Write the function

First, let's define what this function do and how's the parameters required, these will be combined to prompt when invoking LLM.

type Parameter struct {
	Domain string `json:"domain" jsonschema:"description=Domain of the website,example=example.com"`
}

func Description() string {
	return `if user asks ip or network latency of a domain, you should return the result of the giving domain. try your best to dissect user expressions to infer the right domain names`
}

func InputSchema() any {
	return &Parameter{}
}

Create a Stateful Serverless Function to get the IP and Latency of a domain:

func Handler(ctx serverless.Context) {
	var msg Parameter
	ctx.ReadLLMArguments(&msg)

	// get ip of the domain
	ips, _ := net.LookupIP(msg.Domain)

	// get ip[0] ping latency
	pinger, _ := ping.NewPinger(ips[0].String())
	pinger.Count = 3
	pinger.Run()
	stats := pinger.Statistics()

	val := fmt.Sprintf("domain %s has ip %s with average latency %s", msg.Domain, ips[0], stats.AvgRtt)
	ctx.WriteLLMResult(val)
}

Finally, let's run it

$ yomo run app.go

time=2024-03-19T21:43:30.583+08:00 level=INFO msg="connected to zipper" component=StreamFunction sfn_id=B0ttNSEKLSgMjXidB11K1 sfn_name=fn-get-ip-from-domain zipper_addr=localhost:9000
time=2024-03-19T21:43:30.584+08:00 level=INFO msg="register ai function success" component=StreamFunction sfn_id=B0ttNSEKLSgMjXidB11K1 sfn_name=fn-get-ip-from-domain zipper_addr=localhost:9000 name=fn-get-ip-from-domain tag=16

Done, let's have a try

$ curl -i http://127.0.0.1:9000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [
    {
      "role": "system",
      "content": "You are a test assistant."
    },
    {
      "role": "user",
      "content": "Compare website speed between Nike and Puma"
    }
  ],
  "stream": false
}'

HTTP/1.1 200 OK
Content-Length: 944
Connection: keep-alive
Content-Type: application/json
Date: Tue, 19 Mar 2024 13:30:14 GMT
Keep-Alive: timeout=4
Proxy-Connection: keep-alive

{
  "Content": "Based on the data provided for the domains nike.com and puma.com which include IP addresses and average latencies, we can infer the following about their website speeds:
  - Nike.com has an IP address of 13.225.183.84 with an average latency of 65.568333 milliseconds.
  - Puma.com has an IP address of 151.101.194.132 with an average latency of 54.563666 milliseconds.
  
  Comparing these latencies, Puma.com is faster than Nike.com as it has a lower average latency. 
  
  Please be aware, however, that website speed can be influenced by many factors beyond latency, such as server processing time, content size, and delivery networks among others. To get a more comprehensive understanding of website speed, you would need to consider additional metrics and possibly conductreal-time speed tests.",
  "FinishReason": "stop"
}

Full Example Code

Full LLM Function Calling Codes

📚 Documentation

Read more about YoMo at yomo.run/docs.

YoMo ❤️ Vercel, our documentation website is

🎯 Focuses on Geo-distributed AI Inference Infra

It’s no secret that today’s users want instant AI inference, every AI application is more powerful when it response quickly. But, currently, when we talk about distribution, it represents distribution in data center. The AI model is far away from their users from all over the world.

If an application can be deployed anywhere close to their end users, solve the problem, this is Geo-distributed System Architecture:

🦸 Contributing

First off, thank you for considering making contributions. It's people like you that make YoMo better. There are many ways in which you can participate in the project, for example:

• File a bug report. Be sure to include information like what version of YoMo you are using, what your operating system is, and steps to recreate the bug.
• Suggest a new feature.
• Read our contributing guidelines to learn about what types of contributions we are looking for.
• We have also adopted a code of conduct that we expect project participants to adhere to.

License

Apache License 2.0

Documentation

Index

• Variables
• func RunZipper(ctx context.Context, configPath string) error
• type ClientOption
• type SfnOption
• type Source
  • func NewSource(name, zipperAddr string, opts ...SourceOption) Source
• type SourceOption
• type StreamFunction
  • func NewStreamFunction(name, zipperAddr string, opts ...SfnOption) StreamFunction
• type Zipper
  • func NewZipper(name string, meshConfig map[string]config.Mesh, options ...ZipperOption) (Zipper, error)
• type ZipperOption

Variables

var (
	// WithCredential sets the credential method for the Source.
	WithCredential = func(payload string) SourceOption { return SourceOption(core.WithCredential(payload)) }

	// WithSourceTLSConfig sets tls config for the Source.
	WithSourceTLSConfig = func(tc *tls.Config) SourceOption { return SourceOption(core.WithClientTLSConfig(tc)) }

	// WithSourceQuicConfig sets quic config for the Source.
	WithSourceQuicConfig = func(qc *quic.Config) SourceOption { return SourceOption(core.WithClientQuicConfig(qc)) }

	// WithLogger sets logger for the Source.
	WithLogger = func(l *slog.Logger) SourceOption { return SourceOption(core.WithLogger(l)) }

	// WithSourceReConnect makes source Connect until success, unless authentication fails.
	WithSourceReConnect = func() SourceOption { return SourceOption(core.WithReConnect()) }
)

SourceOption Options.

var (
	// WithSfnCredential sets the credential method for the Sfn.
	WithSfnCredential = func(payload string) SfnOption { return SfnOption(core.WithCredential(payload)) }

	// WithSfnTLSConfig sets tls config for the Sfn.
	WithSfnTLSConfig = func(tc *tls.Config) SfnOption { return SfnOption(core.WithClientTLSConfig(tc)) }

	// WithSfnQuicConfig sets quic config for the Sfn.
	WithSfnQuicConfig = func(qc *quic.Config) SfnOption { return SfnOption(core.WithClientQuicConfig(qc)) }

	// WithSfnLogger sets logger for the Sfn.
	WithSfnLogger = func(l *slog.Logger) SfnOption { return SfnOption(core.WithLogger(l)) }

	// WithSfnReConnect makes sfn Connect until success, unless authentication fails.
	WithSfnReConnect = func() SfnOption { return SfnOption(core.WithReConnect()) }

	// WithSfnAIFunctionDefinition sets AI function definition for the Sfn.
	WithSfnAIFunctionDefinition = func(description string, inputModel any) SfnOption {
		return SfnOption(core.WithAIFunctionDefinition(description, inputModel))
	}

	// WithAIFunctionJsonDefinition sets AI function definition for the Sfn in the form of jsonschema string.
	WithAIFunctionJsonDefinition = func(jsonschema string) SfnOption {
		return SfnOption(core.WithAIFunctionJsonDefinition(jsonschema))
	}

	// DisableOtelTrace determines whether to disable otel trace.
	DisableOtelTrace = func() SfnOption { return SfnOption(core.DisableOtelTrace()) }
)

Sfn Options.

var (
	// WithAuth sets the zipper authentication method.
	WithAuth = func(name string, args ...string) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithAuth(name, args...))
		}
	}

	// WithZipperTLSConfig sets the TLS configuration for the zipper.
	WithZipperTLSConfig = func(tc *tls.Config) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithServerTLSConfig(tc))
		}
	}

	// WithZipperQuicConfig sets the QUIC configuration for the zipper.
	WithZipperQuicConfig = func(qc *quic.Config) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithServerQuicConfig(qc))
		}
	}

	// WithZipperLogger sets logger for the zipper.
	WithZipperLogger = func(l *slog.Logger) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithServerLogger(l))
		}
	}

	// WithRouter sets router for the zipper.
	WithRouter = func(r router.Router) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithRouter(r))
		}
	}

	// WithConnector sets connector for the zipper.
	WithConnector = func(c core.Connector) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithConnector(c))
		}
	}

	// WithVersionNegotiateFunc sets the version negotiate function for the zipper
	WithVersionNegotiateFunc = func(f core.VersionNegotiateFunc) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithVersionNegotiateFunc(f))
		}
	}

	// WithUpstreamOption provides upstream zipper options for Zipper.
	WithUpstreamOption = func(opts ...ClientOption) ZipperOption {
		return func(o *zipperOptions) {
			o.clientOption = opts
		}
	}

	// WithZipperConnMiddleware sets conn middleware for the zipper.
	WithZipperConnMiddleware = func(mw ...core.ConnMiddleware) ZipperOption {
		return func(o *zipperOptions) {
			o.serverOption = append(o.serverOption, core.WithConnMiddleware(mw...))
		}
	}

	// WithZipperFrameMiddleware sets frame middleware for the zipper.
	WithZipperFrameMiddleware = func(mw ...core.FrameMiddleware) ZipperOption {
		return func(o *zipperOptions) {
			o.serverOption = append(o.serverOption, core.WithFrameMiddleware(mw...))
		}
	}
)

Functions

func RunZipper

func RunZipper(ctx context.Context, configPath string) error

RunZipper run a zipper from a config file.

Types

type ClientOption

type ClientOption = core.ClientOption

ClientOption is option for the upstream Zipper.

type SfnOption

type SfnOption core.ClientOption

SfnOption is option for the SFN.

type Source

type Source interface {
	// Close will close the connection to YoMo-Zipper.
	Close() error
	// Connect to YoMo-Zipper.
	Connect() error
	// Write the data to directed downstream.
	Write(tag uint32, data []byte) error
	// WriteWithTarget writes data to sfn instance with specified target.
	WriteWithTarget(tag uint32, data []byte, target string) error
	// SetErrorHandler set the error handler function when server error occurs
	SetErrorHandler(fn func(err error))
}

Source is responsible for sending data to yomo.

func NewSource

func NewSource(name, zipperAddr string, opts ...SourceOption) Source

NewSource create a yomo-source

type SourceOption

type SourceOption core.ClientOption

SourceOption is option for the Source.

type StreamFunction

type StreamFunction interface {
	// SetWantedTarget sets target for sfn that to receive data carrying the same target.
	// This function is optional and it should be called before Connect().
	SetWantedTarget(string)
	// SetObserveDataTags set the data tag list that will be observed
	SetObserveDataTags(tag ...uint32)
	// Init will initialize the stream function
	Init(fn func() error) error
	// SetHandler set the handler function, which accept the raw bytes data and return the tag & response
	SetHandler(fn core.AsyncHandler) error
	// SetErrorHandler set the error handler function when server error occurs
	SetErrorHandler(fn func(err error))
	// SetPipeHandler set the pipe handler function
	SetPipeHandler(fn core.PipeHandler) error
	// SetCronHandler set the cron handler function.
	//  Examples:
	//  sfn.SetCronHandler("0 30 * * * *", func(ctx serverless.CronContext) {})
	//  sfn.SetCronHandler("@hourly",      func(ctx serverless.CronContext) {})
	//  sfn.SetCronHandler("@every 1h30m", func(ctx serverless.CronContext) {})
	// more spec style see: https://pkg.go.dev/github.com/robfig/cron#hdr-Usage
	SetCronHandler(spec string, fn core.CronHandler) error
	// Connect create a connection to the zipper
	Connect() error
	// Close will close the connection
	Close() error
	// Wait waits sfn to finish.
	Wait()
}

StreamFunction defines serverless streaming functions.

func NewStreamFunction

func NewStreamFunction(name, zipperAddr string, opts ...SfnOption) StreamFunction

NewStreamFunction create a stream function.

type Zipper

type Zipper interface {
	// Logger returns the logger of zipper.
	Logger() *slog.Logger

	// ListenAndServe start zipper as server.
	ListenAndServe(context.Context, string) error

	// Close will close the zipper.
	Close() error
}

Zipper is the orchestrator of yomo. There are two types of zipper: one is Upstream Zipper, which is used to connect to multiple downstream zippers, another one is Downstream Zipper (will call it as Zipper directly), which is used to connected by `Upstream Zipper`, `Source` and `Stream Function`.

func NewZipper

func NewZipper(name string, meshConfig map[string]config.Mesh, options ...ZipperOption) (Zipper, error)

NewZipper returns a zipper.

type ZipperOption

type ZipperOption func(*zipperOptions)

ZipperOption is option for the Zipper.