yomo

README

YoMo

YoMo is an open-source Streaming Serverless Framework for building Low-latency Edge Computing applications. Built atop QUIC Transport Protocol and Functional Reactive Programming interface, it makes real-time data processing reliable, secure, and easy.

Official Website: 🦖https://yomo.run

💚 We care about: The Demand For Real-Time Digital User Experiences

It’s no secret that today’s users want instant gratification, every productivity application is more powerful when it's collaborative. But, currently, when we talk about distribution, it represents distribution in data center. API is far away from their users from all over the world.

If an application can be deployed anywhere close to their end users, solve the problem, this is Geo-distributed System Architecture:

🌶 Features

	Features
⚡️	Low-latency Guaranteed by implementing atop QUIC QUIC
🔐	Security TLS v1.3 on every data packet by design
📱	5G/WiFi-6 Reliable networking in Cellular/Wireless
🌎	Geo-Distributed Edge Mesh Edge-Mesh Native architecture makes your services close to end users
📸	Event-First Architecture leverages serverless service to be event driven and elastic
🦖	Streaming Serverless Write only a few lines of code to build applications and microservices
🚀	Y3 a faster than real-time codec
📨	Reactive stream processing based on Rx

🚀 Getting Started

Prerequisite

Install Go

1. Install CLI

curl -fsSL https://get.yomo.run | sh

Verify if the CLI was installed successfully

$ yomo version

YoMo CLI Version: v1.1.1
Runtime Version: v1.8.1

2. Create your stream function

$ yomo init yomo-app-demo

⌛  Initializing the Stream Function...
✅  Congratulations! You have initialized the stream function successfully.
ℹ️   You can enjoy the YoMo Stream Function via the command: 
ℹ️   	DEV: 	yomo dev -n Noise yomo-app-demo/app.go
ℹ️   	PROD: 	First run source application, eg: go run example/source/main.go
		Second: yomo run -n yomo-app-demo yomo-app-demo/app.go

$ cd yomo-app-demo

CLI will automatically create the app.go:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"time"

	"github.com/yomorun/yomo/rx"
)

// NoiseData represents the structure of data
type NoiseData struct {
	Noise float32 `json:"noise"` // Noise value
	Time  int64   `json:"time"`  // Timestamp (ms)
	From  string  `json:"from"`  // Source IP
}

var echo = func(_ context.Context, i interface{}) (interface{}, error) {
	value := i.(*NoiseData)
	value.Noise = value.Noise / 10
	rightNow := time.Now().UnixNano() / int64(time.Millisecond)
	fmt.Println(fmt.Sprintf("[%s] %d > value: %f ⚡️=%dms", value.From, value.Time, value.Noise, rightNow-value.Time))
	return value.Noise, nil
}

// Handler will handle data in Rx way
func Handler(rxstream rx.Stream) rx.Stream {
	stream := rxstream.
		Unmarshal(json.Unmarshal, func() interface{} { return &NoiseData{} }).
		Debounce(50).
		Map(echo).
		StdOut()

	return stream
}

func DataTags() []byte {
	return []byte{0x33}
}

3. Build and run

1. Run yomo dev from the terminal. you will see the following message:

$ yomo dev

ℹ️  YoMo Stream Function file: app.go
⌛  Create YoMo Stream Function instance...
⌛  YoMo Stream Function building...
✅  Success! YoMo Stream Function build.
ℹ️   YoMo Stream Function is running...
2021/11/16 10:02:43 [core:client]  has connected to yomo-app-demo (dev.yomo.run:9140)
[localhost] 1637028164050 > value: 6.575044 ⚡️=9ms
[StdOut]:  6.5750437
[localhost] 1637028164151 > value: 10.076103 ⚡️=5ms
[StdOut]:  10.076103
[localhost] 1637028164251 > value: 15.560066 ⚡️=8ms
[StdOut]:  15.560066
[localhost] 1637028164352 > value: 15.330824 ⚡️=2ms
[StdOut]:  15.330824
[localhost] 1637028164453 > value: 10.859857 ⚡️=7ms
[StdOut]:  10.859857

Congratulations! You have done your first YoMo Stream Function.

🧩 Interop

Metaverse Workplace (Virtual Office) with YoMo

• Frontend
• Backend

Sources

• Connect EMQ X Broker to YoMo
• Connect MQTT to YoMo

Stream Functions

• Write a Stream Function with WebAssembly by WasmEdge

Output Connectors

• Connect to FaunaDB to store post-processed result the serverless way
• Connect to InfluxDB to store post-processed result
• Connect to TDEngine to store post-processed result

🗺 Location Insensitive Deployment

📚 Documentation

• YoMo-Source: docs.yomo.run/source
• YoMo-Stream-Function: docs.yomo.run/stream-function
• YoMo-Zipper: docs.yomo.run/zipper
• Stream Processing in Rx way: Rx
• Faster than real-time codec: Y3

YoMo ❤️ Vercel, our documentation website is

🎯 Focuses on computings out of data center

• IoT/IIoT/AIoT
• Latency-sensitive applications.
• Networking situation with packet loss or high latency.
• Handling continuous high frequency generated data with stream-processing.
• Building Complex systems with Streaming-Serverless architecture.

🌟 Why YoMo

• Based on QUIC (Quick UDP Internet Connection) protocol for data transmission, which uses the User Datagram Protocol (UDP) as its basis instead of the Transmission Control Protocol (TCP); significantly improves the stability and throughput of data transmission. Especially for cellular networks like 5G.
• A self-developed y3-codec optimizes decoding performance. For more information, visit its own repository on GitHub.
• Based on stream computing, which improves speed and accuracy when dealing with data handling and analysis; simplifies the complexity of stream-oriented programming.
• Secure-by-default from transport protocol.

🦸 Contributing

First off, thank you for considering making contributions. It's people like you that make YoMo better. There are many ways in which you can participate in the project, for example:

• File a bug report. Be sure to include information like what version of YoMo you are using, what your operating system is, and steps to recreate the bug.
• Suggest a new feature.
• Read our contributing guidelines to learn about what types of contributions we are looking for.
• We have also adopted a code of conduct that we expect project participants to adhere to.

🤹🏻‍♀️ Feedback

Any questions or good ideas, please feel free to come to our Discussion. Any feedback would be greatly appreciated!

🏄‍♂️ Best Practice in Production

Discussion #314 Tips: YoMo/QUIC Server Performance Tuning

License

Apache License 2.0

Documentation

Index

• Constants
• type Option
  • func WithAuth(name string, args ...string) Option
  • func WithClientOptions(opts ...core.ClientOption) Option
  • func WithCredential(payload string) Option
  • func WithLogger(logger *slog.Logger) Option
  • func WithMeshConfigURL(url string) Option
  • func WithObserveDataTags(tags ...frame.Tag) Option
  • func WithQuicConfig(qc *quic.Config) Option
  • func WithServerOptions(opts ...core.ServerOption) Option
  • func WithTLSConfig(tc *tls.Config) Option
  • func WithZipperAddr(addr string) Option
• type Options
  • func NewOptions(opts ...Option) *Options
• type Source
  • func NewSource(name string, opts ...Option) Source
• type StreamFunction
  • func NewStreamFunction(name string, opts ...Option) StreamFunction
• type Zipper
  • func NewDownstreamZipper(name string, opts ...Option) Zipper
  • func NewZipper(conf string) (Zipper, error)
  • func NewZipperWithOptions(name string, opts ...Option) Zipper

Constants

const (
	// DefaultZipperAddr is the default address of downstream zipper.
	DefaultZipperAddr = "localhost:9000"
)

Types

type Option

type Option func(o *Options)

Option is a function that applies a YoMo-Client option.

func WithAuth

func WithAuth(name string, args ...string) Option

WithAuth sets the server authentication method (used by server)

func WithClientOptions

func WithClientOptions(opts ...core.ClientOption) Option

WithClientOptions returns a new options with opts.

func WithCredential

func WithCredential(payload string) Option

WithCredential sets the client credential method (used by client)

func WithLogger

func WithLogger(logger *slog.Logger) Option

WithLogger sets the client logger

func WithMeshConfigURL

func WithMeshConfigURL(url string) Option

WithMeshConfigURL sets the initial edge-mesh config URL for the YoMo-Zipper.

func WithObserveDataTags

func WithObserveDataTags(tags ...frame.Tag) Option

WithObserveDataTags sets client data tag list.

func WithQuicConfig

func WithQuicConfig(qc *quic.Config) Option

WithQuicConfig sets the QUIC configuration for the client.

func WithServerOptions

func WithServerOptions(opts ...core.ServerOption) Option

WithServerOptions returns a new options with opts.

func WithTLSConfig

func WithTLSConfig(tc *tls.Config) Option

WithTLSConfig sets the TLS configuration for the client.

func WithZipperAddr

func WithZipperAddr(addr string) Option

WithZipperAddr return a new options with ZipperAddr set to addr.

type Options

type Options struct {
	ZipperAddr    string // target Zipper endpoint address
	MeshConfigURL string // meshConfigURL is the URL of edge-mesh config
	ServerOptions []core.ServerOption
	ClientOptions []core.ClientOption
	QuicConfig    *quic.Config
	TLSConfig     *tls.Config
}

Options are the options for YoMo

func NewOptions

func NewOptions(opts ...Option) *Options

NewOptions creates a new options for YoMo-Client.

type Source

type Source interface {
	// Close will close the connection to YoMo-Zipper.
	Close() error
	// Connect to YoMo-Zipper.
	Connect() error
	// SetDataTag will set the tag of data when invoking Write().
	SetDataTag(tag frame.Tag)
	// Write the data to directed downstream.
	Write(data []byte) (n int, err error)
	// WriteWithTag will write data with specified tag, default transactionID is epoch time.
	WriteWithTag(tag frame.Tag, data []byte) error
	// SetErrorHandler set the error handler function when server error occurs
	SetErrorHandler(fn func(err error))
	// [Experimental] SetReceiveHandler set the observe handler function
	SetReceiveHandler(fn func(tag frame.Tag, data []byte))
	// Write the data to all downstream
	Broadcast(data []byte) error
}

Source is responsible for sending data to yomo.

func NewSource

func NewSource(name string, opts ...Option) Source

NewSource create a yomo-source

type StreamFunction

type StreamFunction interface {
	// SetObserveDataTags set the data tag list that will be observed
	// Deprecated: use yomo.WithObserveDataTags instead
	SetObserveDataTags(tag ...frame.Tag)
	// SetHandler set the handler function, which accept the raw bytes data and return the tag & response
	SetHandler(fn core.AsyncHandler) error
	// SetErrorHandler set the error handler function when server error occurs
	SetErrorHandler(fn func(err error))
	// SetPipeHandler set the pipe handler function
	SetPipeHandler(fn core.PipeHandler) error
	// Connect create a connection to the zipper
	Connect() error
	// Close will close the connection
	Close() error
	// Send a data to zipper.
	Write(tag frame.Tag, carriage []byte) error
}

StreamFunction defines serverless streaming functions.

func NewStreamFunction

func NewStreamFunction(name string, opts ...Option) StreamFunction

NewStreamFunction create a stream function.

type Zipper

type Zipper interface {
	// ConfigWorkflow will register workflows from config files to zipper.
	ConfigWorkflow(conf string) error

	// ConfigMesh will register edge-mesh config URL
	ConfigMesh(url string) error

	// ListenAndServe start zipper as server.
	ListenAndServe() error

	// AddDownstreamZipper will add downstream zipper.
	AddDownstreamZipper(downstream Zipper) error

	// Addr returns the listen address of zipper.
	Addr() string

	// Stats return insight data
	Stats() int

	// Close will close the zipper.
	Close() error

	// InitOptions initialize options
	InitOptions(opts ...Option)
}

Zipper is the orchestrator of yomo. There are two types of zipper: one is Upstream Zipper, which is used to connect to multiple downstream zippers, another one is Downstream Zipper (will call it as Zipper directly), which is used to connected by `Upstream Zipper`, `Source` and `Stream Function`

func NewDownstreamZipper

func NewDownstreamZipper(name string, opts ...Option) Zipper

NewDownstreamZipper create a zipper descriptor for downstream zipper.

func NewZipper

func NewZipper(conf string) (Zipper, error)

NewZipper create a zipper instance from config files.

func NewZipperWithOptions

func NewZipperWithOptions(name string, opts ...Option) Zipper

NewZipperWithOptions create a zipper instance.