yomo

README

YoMo

YoMo is an open-source Streaming Serverless Framework for building Low-latency Geo-Distributed System. Built atop QUIC Transport Protocol and Functional Reactive Programming interface, it makes real-time collaborative applications reliable, secure, and easy.

Read the docs: 🦖https://yomo.run

💚 We care about: The Demand For Real-Time Digital User Experiences

It’s no secret that today’s users want instant gratification, every productivity application is more powerful when it's collaborative. But, currently, when we talk about distribution, it represents distribution in data center. API is far away from their users from all over the world.

If an application can be deployed anywhere close to their end users, solve the problem, this is Geo-distributed System Architecture:

🌶 Features

	Features
⚡️	Low-latency Guaranteed by implementing atop QUIC QUIC
🔐	Security TLS v1.3 on every data packet by design
📱	5G/WiFi-6 Reliable networking in Cellular/Wireless
🌎	Geo-Distributed Edge Mesh Edge-Mesh Native architecture makes your services close to end users
📸	Event-First Architecture leverages serverless service to be event driven and elastic
🦖	Streaming Serverless Write only a few lines of code to build applications and microservices
🚀	Y3 a faster than real-time codec
📨	Reactive stream processing based on Rx

🚀 Getting Started

Prerequisite

Install Go

Step 1. Install CLI

curl -fsSL https://get.yomo.run | sh

Verify if the CLI was installed successfully

yomo version

Step 2. Init your first stream function, in WebAssembly way

In this demo, we will create a go project observing a data stream and count bytes received.

yomo init try-yomo

The yomo CLI will generate codes in folder try-yomo.

Step 3. Build

This Stream Function is written in Go, before compiling to WebAssembly, you need to install tinygo first.

$ yomo build app.go

ℹ️ YoMo Stream Function file: app.go
⌛ YoMo Stream Function building...
✅ Success! YoMo Stream Function build.

Now, we get the sfn.wasm file, only 190K bytes.

$ exa -l
.rw-r--r--  359 fanweixiao 14 Apr 01:02 app.go
.rwxr-xr-x 190k fanweixiao 14 Apr 01:08 sfn.wasm

Note: you can implement Stream Function in Rust, Zig, C or other languages can be compiled to WebAssembly, more examples can be found at example/7-wasm/sfn.

Step 4. Run

There is an public test Zipper service tap.yomo.dev:9140 which is provided by our community, you can test your StreamFunction quickly by connecting to it.

$ yomo dev sfn.wasm

ℹ️ YoMo Stream Function file: sfn.wasm
⌛  Create YoMo Stream Function instance...
ℹ️ Starting YoMo Stream Function instance with executable file: sfn.wasm. Zipper: [tap.yomo.dev:9140].
ℹ️ YoMo Stream Function is running...
time=2023-04-14T00:05:25.073+08:00 level=INFO msg="use credential" component="Stream Function" client_id=7IwpRofCpPp-AcVV2qUFc client_name=yomo-app-demo credential_name=none
time=2023-04-14T00:05:26.297+08:00 level=INFO msg="connected to zipper" component="Stream Function" client_id=7IwpRofCpPp-AcVV2qUFc client_name=yomo-app-demo zipper_addr=tap.yomo.dev:9140
sfn received 57 bytes
sfn received 59 bytes
sfn received 59 bytes
sfn received 59 bytes
sfn received 58 bytes
sfn received 59 bytes
sfn received 58 bytes
sfn received 59 bytes
sfn received 58 bytes
^C

It works!

Note: yomo dev sfn.wasm is more convinient for development, it will connect to tap.yomo.dev:9140 automatically. It's a shortcut of yomo run -z tap.yomo.dev:9140 -n yomo-app-demo.

There are many other examples that can help reduce the learning curve:

• 0-basic: Write Stream Function in pure golang.
• 1-pipeline: Unix Pipeline over Cloud.
• 2-iopipe: Unix Pipeline over Cloud.
• 3-multi-sfn: Write programs that do one thing and do it well. Write programs to work together. -- Doug Mcllroy
• 4-cascading-zipper: Flexible adjustment of sfn deployment and run locations.
• 5-backflow
• 6-mesh: Demonstrate how to put your serverless closer to end-user.
• 7-wasm: Implement Stream Function by WebAssembly in c, go, rust and even zig.
• 8-deno: Demonstrate how to write Stream Function with TypeScript and deno.
• 9-cli: Implement Stream Function in Rx way.

Read more about YoMo at yomo.run/docs.

🧩 Interop

Metaverse Workplace (Virtual Office) with YoMo

• Frontend
• Backend

Sources

• Connect EMQ X Broker to YoMo
• Connect MQTT to YoMo

Stream Functions

• Write a Stream Function with WebAssembly by WasmEdge

Output Connectors

• Connect to FaunaDB to store post-processed result the serverless way
• Connect to InfluxDB to store post-processed result
• Connect to TDEngine to store post-processed result

🗺 Location Insensitive Deployment

📚 Documentation

• YoMo-Source: docs.yomo.run/source
• YoMo-Stream-Function: docs.yomo.run/stream-function
• YoMo-Zipper: docs.yomo.run/zipper
• Faster than real-time codec: Y3

YoMo ❤️ Vercel, our documentation website is

🎯 Focuses on computings out of data center

• IoT/IIoT/AIoT
• Latency-sensitive applications.
• Networking situation with packet loss or high latency.
• Handling continuous high frequency generated data with stream-processing.
• Building Complex systems with Streaming-Serverless architecture.

🌟 Why YoMo

• Based on QUIC (Quick UDP Internet Connection) protocol for data transmission, which uses the User Datagram Protocol (UDP) as its basis instead of the Transmission Control Protocol (TCP); significantly improves the stability and throughput of data transmission. Especially for cellular networks like 5G.
• A self-developed y3-codec optimizes decoding performance. For more information, visit its own repository on GitHub.
• Based on stream computing, which improves speed and accuracy when dealing with data handling and analysis; simplifies the complexity of stream-oriented programming.
• Secure-by-default from transport protocol.

🦸 Contributing

First off, thank you for considering making contributions. It's people like you that make YoMo better. There are many ways in which you can participate in the project, for example:

• File a bug report. Be sure to include information like what version of YoMo you are using, what your operating system is, and steps to recreate the bug.
• Suggest a new feature.
• Read our contributing guidelines to learn about what types of contributions we are looking for.
• We have also adopted a code of conduct that we expect project participants to adhere to.

🤹🏻‍♀️ Feedback

Any questions or good ideas, please feel free to come to our Discussion. Any feedback would be greatly appreciated!

🏄‍♂️ Best Practice in Production

Discussion #314 Tips: YoMo/QUIC Server Performance Tuning

License

Apache License 2.0

Documentation

Index

• Variables
• func RunZipper(ctx context.Context, configPath string) error
• type ClientOption
• type SfnOption
• type Source
  • func NewSource(name, zipperAddr string, opts ...SourceOption) Source
• type SourceOption
• type StreamFunction
  • func NewStreamFunction(name, zipperAddr string, opts ...SfnOption) StreamFunction
• type Zipper
  • func NewZipper(name string, meshConfig map[string]config.Downstream, options ...ZipperOption) (Zipper, error)
• type ZipperOption

Variables

var (
	// WithObserveDataTags sets the list of data tags for the Source.
	WithObserveDataTags = func(tags ...frame.Tag) SourceOption { return SourceOption(core.WithObserveDataTags(tags...)) }

	// WithCredential sets the credential method for the Source.
	WithCredential = func(payload string) SourceOption { return SourceOption(core.WithCredential(payload)) }

	// WithSourceTLSConfig sets tls config for the Source.
	WithSourceTLSConfig = func(tc *tls.Config) SourceOption { return SourceOption(core.WithClientTLSConfig(tc)) }

	// WithSourceQuicConfig sets quic config for the Source.
	WithSourceQuicConfig = func(qc *quic.Config) SourceOption { return SourceOption(core.WithClientQuicConfig(qc)) }

	// WithLogger sets logger for the Source.
	WithLogger = func(l *slog.Logger) SourceOption { return SourceOption(core.WithLogger(l)) }

	// WithTracerProvider sets tracer provider for the Source.
	WithTracerProvider = func(tp trace.TracerProvider) SourceOption { return SourceOption(core.WithTracerProvider(tp)) }
)

SourceOption Options.

var (
	// WithSfnCredential sets the credential method for the Sfn.
	WithSfnCredential = func(payload string) SfnOption { return SfnOption(core.WithCredential(payload)) }

	// WithSfnTLSConfig sets tls config for the Sfn.
	WithSfnTLSConfig = func(tc *tls.Config) SfnOption { return SfnOption(core.WithClientTLSConfig(tc)) }

	// WithSfnQuicConfig sets quic config for the Sfn.
	WithSfnQuicConfig = func(qc *quic.Config) SfnOption { return SfnOption(core.WithClientQuicConfig(qc)) }

	// WithSfnLogger sets logger for the Sfn.
	WithSfnLogger = func(l *slog.Logger) SfnOption { return SfnOption(core.WithLogger(l)) }

	// WithSfnTracerProvider sets tracer provider for the Sfn.
	WithSfnTracerProvider = func(tp trace.TracerProvider) SfnOption { return SfnOption(core.WithTracerProvider(tp)) }
)

Sfn Options.

var (
	// WithAuth sets the zipper authentication method.
	WithAuth = func(name string, args ...string) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithAuth(name, args...))
		}
	}

	// WithZipperTLSConfig sets the TLS configuration for the zipper.
	WithZipperTLSConfig = func(tc *tls.Config) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithServerTLSConfig(tc))
		}
	}

	// WithZipperQuicConfig sets the QUIC configuration for the zipper.
	WithZipperQuicConfig = func(qc *quic.Config) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithServerQuicConfig(qc))
		}
	}

	// WithZipperLogger sets logger for the zipper.
	WithZipperLogger = func(l *slog.Logger) ZipperOption {
		return func(zo *zipperOptions) {
			zo.serverOption = append(zo.serverOption, core.WithServerLogger(l))
		}
	}

	// WithUpstreamOption provides upstream zipper options for Zipper.
	WithUpstreamOption = func(opts ...ClientOption) ZipperOption {
		return func(o *zipperOptions) {
			o.clientOption = opts
		}
	}

	// WithZipperTracerProvider sets tracer provider for the zipper.
	WithZipperTracerProvider = func(tp trace.TracerProvider) ZipperOption {
		return func(o *zipperOptions) {
			o.serverOption = append(o.serverOption, core.WithServerTracerProvider(tp))
		}
	}

	// WithConnMiddleware sets conn middleware for the zipper.
	WithZipperConnMiddleware = func(mw ...core.ConnMiddleware) ZipperOption {
		return func(o *zipperOptions) {
			o.serverOption = append(o.serverOption, core.WithConnMiddleware(mw...))
		}
	}

	// WithFrameMiddleware sets frame middleware for the zipper.
	WithZipperFrameMiddleware = func(mw ...core.FrameMiddleware) ZipperOption {
		return func(o *zipperOptions) {
			o.serverOption = append(o.serverOption, core.WithFrameMiddleware(mw...))
		}
	}
)

Functions

func RunZipper

func RunZipper(ctx context.Context, configPath string) error

RunZipper run a zipper from a config file.

Types

type ClientOption

type ClientOption = core.ClientOption

ClientOption is option for the upstream Zipper.

type SfnOption

type SfnOption core.ClientOption

SfnOption is option for the SFN.

type Source

type Source interface {
	// Close will close the connection to YoMo-Zipper.
	Close() error
	// Connect to YoMo-Zipper.
	Connect() error
	// Write the data to directed downstream.
	Write(tag uint32, data []byte) error
	// SetErrorHandler set the error handler function when server error occurs
	SetErrorHandler(fn func(err error))
	// [Experimental] SetReceiveHandler set the observe handler function
	SetReceiveHandler(fn func(tag uint32, data []byte))
}

Source is responsible for sending data to yomo.

func NewSource

func NewSource(name, zipperAddr string, opts ...SourceOption) Source

NewSource create a yomo-source

type SourceOption

type SourceOption core.ClientOption

SourceOption is option for the Source.

type StreamFunction

type StreamFunction interface {
	// SetObserveDataTags set the data tag list that will be observed
	SetObserveDataTags(tag ...uint32)
	// Init will initialize the stream function
	Init(fn func() error) error
	// SetHandler set the handler function, which accept the raw bytes data and return the tag & response
	SetHandler(fn core.AsyncHandler) error
	// SetErrorHandler set the error handler function when server error occurs
	SetErrorHandler(fn func(err error))
	// SetPipeHandler set the pipe handler function
	SetPipeHandler(fn core.PipeHandler) error
	// Connect create a connection to the zipper
	Connect() error
	// Close will close the connection
	Close() error
	// Wait waits sfn to finish.
	Wait()
}

StreamFunction defines serverless streaming functions.

func NewStreamFunction

func NewStreamFunction(name, zipperAddr string, opts ...SfnOption) StreamFunction

NewStreamFunction create a stream function.

type Zipper

type Zipper interface {
	// Logger returns the logger of zipper.
	Logger() *slog.Logger

	// ListenAndServe start zipper as server.
	ListenAndServe(context.Context, string) error

	// Close will close the zipper.
	Close() error
}

Zipper is the orchestrator of yomo. There are two types of zipper: one is Upstream Zipper, which is used to connect to multiple downstream zippers, another one is Downstream Zipper (will call it as Zipper directly), which is used to connected by `Upstream Zipper`, `Source` and `Stream Function`.

func NewZipper

func NewZipper(name string, meshConfig map[string]config.Downstream, options ...ZipperOption) (Zipper, error)

NewZipper returns a zipper.

type ZipperOption

type ZipperOption func(*zipperOptions)

ZipperOption is option for the Zipper.