glow

README

🌟Computational Framework

The glow is an idiomatic general purpose computational framework.

Installation

Simply add the following import to your code, and then go [build|run|test] will automatically fetch the necessary dependencies:

import "github.com/lnashier/glow"

Examples

Examples

Node Function

Node Function is the basic unit in the glow that processes the data.

Basic Function

Basic Node Function (BasicFunc) allows a Node to operate in "push-pull" mode. The Network pushes data to BasicFunc, and it waits for the function to return with output data, which is then forwarded to connected Node(s).

func(ctx context.Context, data any) (any, error)

Emit Function

Emit Node Function (EmitFunc) allows a Node to operate in "push-push" mode. The Network pushes data to EmitFunc, and the function emits zero or more data points back to the Network through the supplied callback emit function. Eventually, it returns control back to the Network.

func(ctx context.Context, data any, emit func(any)) error

Node

A Node is an abstraction over Node Function that forms connections among Node Functions, enabling the flow of data through the glow Network.

Isolated Node

A Node with no links is considered an isolated-node.

Seed Node

A Node with only egress links is considered a seed-node.

Transit Node

A Node with both egress and ingress links is considered a transit-node.

Terminal Node

A Node with only ingress links is considered a terminal-node.

Link

A Link represents a connection between two Nodes, facilitating data flow from one Node to another.

Paused Link

A Paused Link temporarily stops the flow of data between Nodes without removing the Link itself from the Network.

Removed Link

A Removed Link permanently disconnects two Nodes, ceasing all data flow through that Link. The Network may be purged to physically remove such links.

Mode

Broadcaster Mode

In Broadcaster Mode, a Node broadcasts all incoming data to all its outgoing links, ensuring that all downstream Nodes receive the same data.

Distributor Mode

In Distributor Mode, a Node distributes incoming data among its outgoing links, balancing the data load across multiple downstream Nodes.

Session

A Session represents a single instance of data processing within the Network. It tracks the state and progress of data as it moves through the Nodes and Links.

Integrity Checks

Avoid Cycles

This check ensures that the Network remains a Directed Acyclic Graph (DAG), preventing any circular dependencies or infinite loops.

Ignore Isolated Nodes

This option allows the Network to continue operating even when there are isolated Nodes, which have no incoming or outgoing links.

Documentation

Index

• Variables
• func DOT(n *Network) ([]byte, error)
• type Link
  • func (l *Link) Tally() int
  • func (l *Link) Uptime() time.Duration
• type LinkOpt
  • func Size(k int) LinkOpt
• type Network
  • func New(opt ...NetworkOpt) *Network
  • func (n *Network) AddLink(from, to string, opt ...LinkOpt) error
  • func (n *Network) AddNode(opt ...NodeOpt) (string, error)
  • func (n *Network) Egress(key string) []*Link
  • func (n *Network) Ingress(key string) []*Link
  • func (n *Network) Keys() []string
  • func (n *Network) Link(from, to string) (*Link, error)
  • func (n *Network) Links() []*Link
  • func (n *Network) Node(k string) (*Node, error)
  • func (n *Network) Nodes() []*Node
  • func (n *Network) PauseLink(from, to string) error
  • func (n *Network) Purge() error
  • func (n *Network) RemoveLink(from, to string) error
  • func (n *Network) RemoveNode(k string) error
  • func (n *Network) ResumeLink(from, to string) error
  • func (n *Network) Seeds() []string
  • func (n *Network) Start(ctx context.Context) error
  • func (n *Network) Stop() error
  • func (n *Network) Terminals() []string
  • func (n *Network) Uptime() time.Duration
• type NetworkOpt
  • func IgnoreIsolatedNodes() NetworkOpt
  • func PreventCycles() NetworkOpt
  • func StopGracetime(t time.Duration) NetworkOpt
  • func Verbose() NetworkOpt
• type Node
  • func (n *Node) Key() string
  • func (n *Node) Uptime() time.Duration
• type NodeOpt
  • func BasicFunc(f func(ctx context.Context, data any) (any, error)) NodeOpt
  • func Distributor() NodeOpt
  • func EmitFunc(f func(ctx context.Context, data any, emit func(any)) error) NodeOpt
  • func Key(k string) NodeOpt
  • func KeyFunc(f func() string) NodeOpt

Variables

var (
	ErrEmptyNetwork       = errors.New("network is empty")
	ErrNetworkNeedPurging = errors.New("network needs purging")

	ErrNodeNotFound        = errors.New("node not found")
	ErrBadNodeKey          = errors.New("bad node key")
	ErrNodeAlreadyExists   = errors.New("node already exists")
	ErrNodeIsConnected     = errors.New("node is connected")
	ErrSeedingDone         = errors.New("seeding is done")
	ErrNodeGoingAway       = errors.New("node is going away")
	ErrIsolatedNodeFound   = errors.New("isolated node found")
	ErrNodeFunctionMissing = errors.New("node function missing")
	ErrTooManyNodeFunction = errors.New("too many node functions")

	ErrLinkNotFound      = errors.New("link not found")
	ErrLinkAlreadyExists = errors.New("link already exists")
	ErrCyclesNotAllowed  = errors.New("cycles not allowed")
	ErrLinkAlreadyPaused = errors.New("link already paused")
)

Functions

func DOT

func DOT(n *Network) ([]byte, error)

DOT describes the Network.

Types

type Link

type Link struct {
	// contains filtered or unexported fields
}

Link captures the connection between two nodes. Data flows from x to y over the Link. A Link can have 3 states:

• Closed - The Link enters the closed state when the Link.From Node goes away. The Link remains in this state until a new session (Network.Start) or system restart.
• Paused - The Link enters the paused state when Network.PauseLink is called. The Link remains in this state until resumed (Network.ResumeLink) or system restart.
• Removed - The Link enters the removed state when Network.RemoveLink is called. The Link remains in this state until added back (Network.AddLink) or system restart.

func (*Link) Tally

func (l *Link) Tally() int

Tally returns the total count of data transmitted over the link thus far.

func (*Link) Uptime

func (l *Link) Uptime() time.Duration

type LinkOpt

type LinkOpt func(*Link)

func Size

func Size(k int) LinkOpt

Size sets bandwidth for the Link.

type Network

type Network struct {
	// contains filtered or unexported fields
}

Network represents nodes and their links.

func New

func New(opt ...NetworkOpt) *Network

New creates a new Network.

func (*Network) AddLink

func (n *Network) AddLink(from, to string, opt ...LinkOpt) error

AddLink connects from-node to to-node. Once Link is made, nodes are said to be communicating over the Link in the direction from -> to. See:

• RemoveLink
• PauseLink
• ResumeLink

func (*Network) AddNode

func (n *Network) AddNode(opt ...NodeOpt) (string, error)

AddNode adds a new Node in the Network. Node key is retrieved from the provided KeyFunc function if not given.

func (*Network) Egress

func (n *Network) Egress(key string) []*Link

Egress returns all the egress links for the Node.

func (*Network) Ingress

func (n *Network) Ingress(key string) []*Link

Ingress returns all the ingress links for the Node.

func (*Network) Keys

func (n *Network) Keys() []string

Keys returns all the nodes as their unique keys in the Network. Network.Node should be called to get actual Node.

func (*Network) Link

func (n *Network) Link(from, to string) (*Link, error)

Link returns connection between from and to nodes if any.

func (*Network) Links

func (n *Network) Links() []*Link

Links returns all the links in the Network.

func (*Network) Node

func (n *Network) Node(k string) (*Node, error)

Node returns the node identified by the provided key.

func (*Network) Nodes

func (n *Network) Nodes() []*Node

Nodes returns all the Node(s) in the Network.

func (*Network) PauseLink

func (n *Network) PauseLink(from, to string) error

PauseLink pauses communication from Node and to Node. See:

• AddLink
• ResumeLink
• RemoveLink

func (*Network) Purge

func (n *Network) Purge() error

Purge cleans up the Network by removing isolated Node(s) and removed Link(s).

func (*Network) RemoveLink

func (n *Network) RemoveLink(from, to string) error

RemoveLink disconnects "from" Node and "to" Node. See:

• AddLink
• PauseLink
• ResumeLink

func (*Network) RemoveNode

func (n *Network) RemoveNode(k string) error

RemoveNode removes a node with provided key. A node can't be removed if it is linked to any other node in the Network.

func (*Network) ResumeLink

func (n *Network) ResumeLink(from, to string) error

ResumeLink resumes communication from node and to node. See:

• PauseLink

func (*Network) Seeds

func (n *Network) Seeds() []string

Seeds returns all the nodes that have only egress Link(s). Network.Node should be called to get actual Node.

func (*Network) Start

func (n *Network) Start(ctx context.Context) error

Start runs the Network.

func (*Network) Stop

func (n *Network) Stop() error

Stop signals the Network to cease all communications. If stop grace period is set, communications will terminate after that period.

func (*Network) Terminals

func (n *Network) Terminals() []string

Terminals returns all the nodes that have only ingress Link(s). Network.Node should be called to get actual Node.

func (*Network) Uptime

func (n *Network) Uptime() time.Duration

type NetworkOpt

type NetworkOpt func(*Network)

func IgnoreIsolatedNodes

func IgnoreIsolatedNodes() NetworkOpt

IgnoreIsolatedNodes allows the Network to run even when there are isolated nodes.

func PreventCycles

func PreventCycles() NetworkOpt

PreventCycles ensures the Network remains a Directed Acyclic Graph (DAG).

func StopGracetime

func StopGracetime(t time.Duration) NetworkOpt

StopGracetime sets the grace period for which the Network waits for processes to finish before stopping.

func Verbose

func Verbose() NetworkOpt

Verbose enables Network to send logs to stdout.

type Node

type Node struct {
	// contains filtered or unexported fields
}

Node represents a node within the Network.

Node types:

• With no Links, Node is considered an isolated-node.
• With only egress Links, Node is considered a seed-node.
• With only ingress Links, Node is considered a terminal-node.
• With both egress and ingress Links, Node is considered a transit-node.

Node operating modes:

• By default, a Node operates in broadcaster mode unless the distributor flag is set. In broadcaster mode, Node broadcasts all incoming data to all outgoing links. When the distributor flag is enabled, a Node distributes incoming data among its outgoing links. Distributor mode is not functional for isolated and terminal nodes.
• By default, a Node operates in "push-pull" mode: the Network pushes data to BasicFunc, and it waits for BasicFunc to return with output data, which is then forwarded to connected Node(s). This behavior can be changed to "push-push" by setting the EmitFunc for the Node. When EmitFunc is set, the Network pushes data to EmitFunc, and it emits zero or more data points back to the Network through the supplied callback emit function.

func (*Node) Key

func (n *Node) Key() string

func (*Node) Uptime

func (n *Node) Uptime() time.Duration

type NodeOpt

type NodeOpt func(*Node)

func BasicFunc

func BasicFunc(f func(ctx context.Context, data any) (any, error)) NodeOpt

BasicFunc is responsible for processing incoming data on the Node. Output from the Node is forwarded to downstream connected Node(s).

func Distributor

func Distributor() NodeOpt

Distributor enables a Node to distribute incoming data among its outgoing links.

func EmitFunc

func EmitFunc(f func(ctx context.Context, data any, emit func(any)) error) NodeOpt

EmitFunc handles processing incoming data on the Node. It provides a callback where output data can be optionally emitted. Emitted data is forwarded to downstream connected Node(s).

func Key

func Key(k string) NodeOpt

Key sets the key for the Node.

func KeyFunc

func KeyFunc(f func() string) NodeOpt

KeyFunc sets function to generate unique keys for the Node.