pipeline

package
v0.0.0-...-e7db92f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 9, 2024 License: MPL-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package pipeline : execution modelling The pipeline package offers mappings from JointJS Diagrams into golang for the purposes of designing a simplified overview of what a kubernetes deployment might look like.

Index

Constants

View Source 
const TIMEOUT = 15

TIMEOUT : The maximum number of minutes a command can execute for if not forever

Variables

This section is empty.

Functions

func Sanitize 

func Sanitize(str string, sep string) string

Sanitize a given string, removing any non-alphanumeric characters and replacing them with sep.

Types

type Command 

type Command struct {

	// ID is the jointJS element ID
	ID string `json:"id"`

	// Parent is the jointJS ID of a parent container
	Parent string `json:"parent"`

	// The command container name
	Name string `json:"name"`

	// The executable binary name to trigger when events are recieved
	Command string `json:"command"`

	// Should the command automatically be started
	AutoStart bool `json:"autostart"`

	// Arguments to give to the command
	Args string `json:"args"`

	// The version of the container
	Version string `json:"version"`

	// The programming language the container is executing
	Language string `json:"element"`

	// Does this container have a script to execute
	Script bool `json:"script"`

	// The content of the script - will be base64 encoded for transmission safety
	ScriptContent string `json:"scriptcontent"`

	// Is this a custom container or is it a pre-fabricated container from an upstream source
	Custom bool `json:"custom"`

	// The user defined timeout in minutes - if >15 will be rounded down to 15
	Timeout int `json:"timeout"`

	// (Re)Use an existing container
	UseExisting bool `json:"existing"`

	// Expose this port as a service port
	ExposePort int `json:"exposeport"`

	// Is the service port a UDP port
	IsUDP bool `json:"isudp"`

	// The required CPU of the container (default 500m cpu)
	CPU string `json:"cpu"`

	// The required memory of the container (default 256Mi)
	Memory string `json:"memory"`

	// The time this command was started in unix nano
	StartTime int64 `json:"starttime"`

	// The time this command ended in unix nano
	EndTime int64 `json:"endtime"`

	// Environment variables to set directly on the command
	Environment []string `json:"environment"`

	// Details about any Git repository configured for the command
	GitRepo *GitRepo `json:"gitrepo"`

	// The image string to build the docker container from and load into kubernetes
	Image string

	// The computed tag to upload the container docker image as
	Tag string

	// Standard output buffer
	Stdout bytes.Buffer

	// Standard error buffer
	Stderr bytes.Buffer

	// The commands process ID whilst executing
	ProcessID int

	// A computed set of process arguments including any files
	ProcessArgs []string

	// A computed file separator
	FileSeperator string
}

Command : Define the structure of a command taken from JointJS and executed by Syphon

func NewCommand 

func NewCommand(cell map[string]interface{}) *Command

NewCommand : Create a new command instance

func (*Command) AddEnvVar 

func (command *Command) AddEnvVar(key, value string)

AddEnvVar : Add a value to the environment variables key, value will be converted to KEY=value

func (*Command) Execute 

func (command *Command) Execute(directory string, subdir string, filename string, event string, libraryDir string) int

Execute : Execute the current command inside the container

This is the main workhorse function for the application, taking all User configured and temporary variables, mapping them into the environment and process arguments then executing the command, terminating as necesssary after the configured timeout, and capturing the output for later storage and retrieval

func (*Command) ExecuteForever 

func (command *Command) ExecuteForever(cmd *exec.Cmd, done chan error) int

ExecuteForever : Executes the given command in a "forever" loop

Note: This command does not copy any logs - presuming they will grow exponentially large over time. Check STDERR/STDOUT on the pod for the logs garnered by it.

func (*Command) ExecuteWithTimeout 

func (command *Command) ExecuteWithTimeout(cmd *exec.Cmd, done chan error) int

ExecuteWithTimeout : Runs the given command with a timeout

By default, this timeout is set to 15 minutes and this value is used if 0 is provided. Longer timeouts can be set or set to -1 to run forever.

func (*Command) GenerateRandomString 

func (command *Command) GenerateRandomString() string

GenerateRandomString : Generates a random string of 12 characters for use in temporary filenames

func (*Command) GetContainer 

func (command *Command) GetContainer(asTag bool) string

GetContainer : Gets a container build string.

if asTag is True, returns a modified form of the container string for uploading to the primary source

type Container 

type Container struct {

	// The JointJS ID of this container element
	ID string

	// The name of the container element
	Name string

	// How many pods to build
	Scale int32

	// A list of all child IDs this container manages
	Children []string

	// The type of container
	SetType string

	// The containers state (Building, Ready, Destroying)
	State string

	// The number of pods last seen
	LastCount int

	// The pipeline this container belongs to
	Pipeline *Pipeline

	// Environment settings for the all pods under this set
	Environment []string
}

Container :_Overall structure of a set container

func NewContainer 

func NewContainer(pipeline *Pipeline, cell map[string]interface{}) *Container

NewContainer : Construct a new container instance

func (*Container) GetChildren 

func (container *Container) GetChildren() []*Command

GetChildren ; Get the set of child command containers of this container

type GitRepo 

type GitRepo struct {

	// The repository URL
	RepoURL string `json:"repo"`

	// The branch or commit to checkout on to
	Branch string `json:"branch"`

	// The Username to checkout with
	Username string `json:"username"`

	// The password or token to use for checkout
	Password string `json:"password"`

	// A script to act as the entrypoint inside the container
	Entrypoint string `json:"entrypoint"`
	// contains filtered or unexported fields
}

GitRepo : Handle git operations for the pipeline

func NewGitRepo 

func NewGitRepo() *GitRepo

NewGitRepo : Create a git repository object

func (*GitRepo) Checkout 

func (gitRepo *GitRepo) Checkout() error

Checkout : Checks out a given branch or revision

func (*GitRepo) Clone 

func (gitRepo *GitRepo) Clone(destination string, options map[string]string) error

Clone : Clone out a given repo

func (*GitRepo) Configure 

func (gitRepo *GitRepo) Configure(values map[string]interface{})

Set up the GitRepo from form input

func (*GitRepo) HasKey 

func (gitRepo *GitRepo) HasKey(where map[string]string, key string) bool

HasKey : Check if a given map has a particular key 

type Link struct {

	// JointJS ID of the current link
	ID string

	// Type of link being drawn
	Type string

	// JointJS Source element ID of the link
	Source string

	// JointJS Destination element ID of the link
	Target string
}

Link : Common structure between element link types 

func GetLink(cell map[string]interface{}) Link

GetLink : Unpack a link out of map[string]interface

type LinkInterface 

type LinkInterface interface {

	// GetType : Get the type of link
	GetType() string

	// GetLink : Get the common link details
	GetLink() Link
}

LinkInterface : Common methods to read data from a link type 

func NewLink(cell map[string]interface{}) LinkInterface

NewLink : Create a new Link object

type Matcher 

type Matcher struct {
	Source  string
	Pattern *regexp.Regexp
}

Matcher : A container for regex matches 

type PathLink struct {

	// Common Link details
	Link

	// The path this link will listen against
	// If link type is Path, this is mutually exclusive with Pattern
	Path string

	// Regex pattern to match when reading paths
	// ignored if link type is socket
	Pattern string

	// If link type is path sets up inotify watchers for the path
	Watch bool
}

PathLink : Socket / Path connections 

func NewPathLink(cell map[string]interface{}) *PathLink

NewPathLink : Convert a map[string]interface into a PathLink type 

func (path *PathLink) GetLink() Link

GetLink : Get common link details

func (*PathLink) GetType 

func (path *PathLink) GetType() string

GetType : Get the type of link

type Pipeline 

type Pipeline struct {

	// The name of the pipeline as specified by the user
	Name string

	// DNS name is a DNS formatted identifier of the pipeline
	DNSName string

	// The Fully qualified domain name of the pipeline
	Fqdn string

	// How the pipeline is represented in storeage
	BucketName string

	// A list of container group objects
	Containers map[string]*Container

	// Commands which become docker containers in their own right
	Commands map[string]*Command

	// Links between the commands
	Links map[string]*LinkInterface

	// A set of source objects
	Sources map[string]*Source

	// Tiyo config object
	Config *config.Config

	// Global environment settings
	Environment []string

	// Global pipeline credentials (encrypted)
	Credentials map[string]string
}

Pipeline : The principle interface for mapping JointJS to Kubernetes

func GetPipeline 

func GetPipeline(config *config.Config, name string) (*Pipeline, error)

GetPipeline : Load a pipeline by name from the bolt store and return a new pipeline

func (*Pipeline) AddEnv 

func (pipeline *Pipeline) AddEnv(command *Command)

AddEnv : Add the full pipeline environment to a command

func (*Pipeline) CommandFromContainerName 

func (pipeline *Pipeline) CommandFromContainerName(kubernetesGroup string, image string) *Command

CommandFromContainerName : Get a command id from its final image name

func (*Pipeline) ContainerFromCommandID 

func (pipeline *Pipeline) ContainerFromCommandID(commandID string) *Container

ContainerFromCommandID : Get a container from a command id

func (*Pipeline) ContainerFromServiceName 

func (pipeline *Pipeline) ContainerFromServiceName(serviceName string) *Container

ContainerFromServiceName : Get a container for a given kubernetes service

func (*Pipeline) GetCommand 

func (pipeline *Pipeline) GetCommand(id string) *Command

GetCommand : Gets the command at a given ID returns nil if not found

func (*Pipeline) GetConnection 

func (pipeline *Pipeline) GetConnection(source *Command, dest *Command) *LinkInterface

GetConnection : Get the link between two instances

func (*Pipeline) GetEndIds 

func (pipeline *Pipeline) GetEndIds() []string

GetEndIds : Get all command ids at the end of the pipeline

return []string 

func (pipeline *Pipeline) GetLink(id string) *LinkInterface

GetLink : Get the link at a given ID returns nil if not found

func (*Pipeline) GetLinksFrom 

func (pipeline *Pipeline) GetLinksFrom(command *Command) []*LinkInterface

GetLinksFrom : Get all links leading from a given command Return slice of type *LinkInterface

func (*Pipeline) GetLinksTo 

func (pipeline *Pipeline) GetLinksTo(command *Command) []*LinkInterface

GetLinksTo : Get all links feeding into a given command returns a slice of type *LinkInterface

func (*Pipeline) GetNext 

func (pipeline *Pipeline) GetNext(after *Command) []*Command

GetNext : Get the next command[s] following the present command return []*Command

func (*Pipeline) GetNextID 

func (pipeline *Pipeline) GetNextID(after *Command) []string

GetNextID : Get all IDs of commands following the present command

return []string slice of IDs

func (*Pipeline) GetParent 

func (pipeline *Pipeline) GetParent(command *Command) *Container

GetParent : Gets the parent (if any) of the current command element returns nil if not found

func (*Pipeline) GetPathSources 

func (pipeline *Pipeline) GetPathSources(source *Command) []string

GetPathSources : get a list of commands which feed the current command

func (*Pipeline) GetPrev 

func (pipeline *Pipeline) GetPrev(before *Command) []*Command

GetPrev : Get previous command[s] return []*Command

func (*Pipeline) GetPreviousID 

func (pipeline *Pipeline) GetPreviousID(before *Command) []string

GetPreviousID : Get the previous ID[s] return []string

func (*Pipeline) GetStart 

func (pipeline *Pipeline) GetStart() []*Command

GetStart : Gets all starting point commands

Starting commands are commands which either have no links leading into them, or their links have a source whose type is not a Command struct.

Starting commands will generally pick up directly off the event queue or require special case handling to initiate flow through the system

return []*Command

func (*Pipeline) GetStartIds 

func (pipeline *Pipeline) GetStartIds() []string

GetStartIds : Gets a list of all IDs which have no inputs from other Command elements return slice of type string

func (*Pipeline) IsConvergence 

func (pipeline *Pipeline) IsConvergence(command *Command) bool

IsConvergence : Is this path a convergence point

A convergence point is where multiple paths come together into a single command. Such points are often bottlenecks for data-flow, or offer services such as API or storage Under normal flow, a convergence point only runs a single instance and may cause the pipeline to wait for feed paths to complete before continuing.

func (*Pipeline) Unique 

func (pipeline *Pipeline) Unique(matchers []Matcher) []Matcher

Unique : Gets a unique list of Match items

func (*Pipeline) WatchItems 

func (pipeline *Pipeline) WatchItems() []Matcher

WatchItems : Gets a list of directories/files to watch for events. Does not create.

If path is empty, takes the name of the upstream command 

type PortLink struct {
	Link

	// Source port for connections
	SourcePort int

	// Destination port on the service
	DestPort int

	// Address to connect to
	Address string
}

PortLink : TCP / UDP connections between elements 

func NewPortLink(cell map[string]interface{}) *PortLink

NewPortLink : Create a new PortLink object 

func (port PortLink) GetLink() Link

GetLink : Get details about this link

func (*PortLink) GetType 

func (port *PortLink) GetType() string

GetType : Gets the type of PortLink (TCP/UDP)

type Source 

type Source struct {

	// The JointJS ID of this element
	ID string `json:"id"`

	// The name of this element in the pipeline
	Name string `json:"name"`

	// The source type of this element
	Type string `json:"sourcetype"`
}

Source : A source data structure

func NewSource 

func NewSource(cell map[string]interface{}) *Source

NewSource : create a new source object

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.