phly

package module
v0.0.0-...-e362888 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 7, 2023 License: BSD-3-Clause Imports: 23 Imported by: 0

README

phly

A tool for running pipeline operations.

WARNING: Currently in development, API will change.

DOUBLE WARNING: This is currently being rewritten, is totally torn up, and close to non-functional.

A pipeline processing framework. Phly has two main pieces:

  • The phly framework that manages registering and instantiating nodes, and loading and running pipelines.
  • The phly app, a lightweight application designed to make it easy to compile in additional nodes.

Building

  • Install Go 1.10.3 or later.
  • From the command line, enter directory phly\phly.
  • Type go get to get all dependencies.
  • Type go build to build the app.

Alternatively, the phly library can be compiled into other Go apps.

Use

The work so far has been on the framework. The actual application currently does nothing but scale images. To that end, running the app will load the data/scale_image.json pipeline, which loads an example image and scales it.

Examples (compiled for Windows):

  • phly.exe. Run the app, which currently defaults to running the data/scale_image.json pipeline.
  • phly.exe -nodes. Display all installed nodes.
  • phly.exe -markdown. Generate markdown for all installed nodes.
  • phly.exe -vars. Display all node-defined variables.

Nodes

  • Batch (phly/batch). Perform multiple actions in parallel.
  • Files (phly/files). Create file lists from file names and folders. Produce a single doc with a single page.
    • cfg value. A value directly entered into the cfg file. Use this if no cla or env are present.
    • cfg env. A value from the environment variables. Use this if no cla is available.
    • cfg cla. A value from the command line arguments.
    • cfg expand. (true or false). When true, folders are expanded to the file contents.
    • output out. The file list.
  • Pipeline (phly/pipeline). Run an internal pipeline.
  • Text (phly/text). Acquire text from the cfg values. If a cla is available use that. If no cla, use the env. If no env, use the value.
    • cfg value. A value directly entered into the cfg file. Use this if no cla or env are present.
    • cfg env. A value from the environment variables. Use this if no cla is available.
    • cfg cla. A value from the command line arguments.
    • output out. The text output.

Documentation

Index

Constants

View Source 
const (
	BadRequestErrCode = 1000 + iota
	IllegalErrCode
	MissingErrCode
	ParseErrCode
)
View Source 
const (
	WhatPins = "pins"
	WhatStop = "stop"
)

Variables

View Source 
var (
	BadRequestErr = errors.New("Bad request")
)

Functions

func ErrorsEqual 

func ErrorsEqual(a, b error) bool

ErrorsEqual() returns true if the errors are equivalent.

func MergeErrors 

func MergeErrors(err ...error) error

MergeErrors() answers the first non-nil error in the list.

func NewBadRequestError 

func NewBadRequestError(msg string) error

func NewIllegalError 

func NewIllegalError(msg string) error

func NewMissingError 

func NewMissingError(msg string) error

func NewParseError 

func NewParseError(err error) error

func Register 

func Register(fac NodeFactory) error

func RegisterVar 

func RegisterVar(name, descr string, optional_value interface{})

RegisterVar() registers a variable with the system, which becomes part of the help system. Multiple variables with the same name can be registered, since each package can have its own non-conflicting variables. You can supply an optional value and the var wil be including when replacing vars through the Environment, although only a single var with the same name can have a value.

func StringPinsEqual 

func StringPinsEqual(a, b Pins) bool

func StringPinsToJson 

func StringPinsToJson(pins Pins) string

func WalkItems 

func WalkItems(pins Pins, channel string, fn ItemFunc)

WalkItems iterates over each item on the channel

func WalkStringItems 

func WalkStringItems(pins Pins, channel string, fn StringItemFunc)

WalkStringItems iterates over each string item on the channel

Types

type BuildPinsCmd 

type BuildPinsCmd int
const (
	PbsChan BuildPinsCmd // The following command must be a string. It will be used to
	// start a new pin with the given channel name.
	PbsDoc // Start a new doc. Can be ommitted after starting a channel. Only necessary

)

type CfgDescr 

type CfgDescr struct {
	Name    string
	Purpose string
}

type Doc 

type Doc struct {
	Header
	MimeType string
	Items    []interface{}
}

Doc describes a single abstract document. It includes a user-defined header, an optional content type, and optional pages.

func NewStringDoc 

func NewStringDoc(n ...string) *Doc

Create a new doc on string items

func (*Doc) AllItem 

func (d *Doc) AllItem(index int) interface{}

func (*Doc) AllItems 

func (d *Doc) AllItems() []interface{}

func (*Doc) AppendItem 

func (d *Doc) AppendItem(item interface{})

func (*Doc) StringItem 

func (d *Doc) StringItem(index int) string

func (*Doc) StringItems 

func (d *Doc) StringItems() []string

type Docs 

type Docs struct {
	Docs []*Doc
}

Docs is a wrapper on a slice of Doc structs, used to add convenience management functions.

func NewDocs 

func NewDocs(docs ...*Doc) *Docs

func (Docs) AllItem 

func (d Docs) AllItem(index int) interface{}

func (Docs) AllItems 

func (d Docs) AllItems() []interface{}

func (Docs) StringItem 

func (d Docs) StringItem(index int) string

func (Docs) StringItems 

func (d Docs) StringItems() []string

type Environment 

type Environment interface {
	// FindFile() answers the full path to the phlyp by searching paths for name.
	FindFile(name string) string
	// FindReader() answers a reader for the given name.
	FindReader(name string) io.Reader

	// Utility for replacing strings with a collection of my vars and supplied pairs.
	ReplaceVars(s string, pairs ...interface{}) string
}

Environment provides access to the system environment. 

type Header struct {
	Values interface{}
}

func (*Header) GetInt 

func (h *Header) GetInt(path string) (int, bool)

func (*Header) GetString 

func (h *Header) GetString(path string) (string, bool)

func (*Header) SetHeader 

func (h *Header) SetHeader(values interface{})

func (*Header) SetInt 

func (h *Header) SetInt(path string, value int) error

func (*Header) SetString 

func (h *Header) SetString(path, value string) error

type InstantiateArgs 

type InstantiateArgs struct {
	Env Environment
}

InstantiateArgs provides information during the instantiation phase.

type ItemFunc 

type ItemFunc func(channel string, doc *Doc, index int, item interface{})

ItemFunc is a callback for a single item in a pin doc.

type Items 

type Items interface {
	// Get collections in various formats
	AllItems() []interface{}
	StringItems() []string

	// Get at index in various formats
	AllItem(index int) interface{}
	StringItem(index int) string
}

Items provides common behaviour on an abstract collection.

type Msg 

type Msg struct {
	What    string
	Payload interface{}
}

Msg is an abstract node message.

func MsgFromPins 

func MsgFromPins(pins Pins) Msg

func MsgFromStop 

func MsgFromStop(err error) Msg

type Node 

type Node interface {
	Describe() NodeDescr

	// Process processes input, sending any results to output.
	// stage provides minimal lifecycle information: whether this node is starting or not.
	// input is the collection of inputs to process.
	// output is used to send any output to the calling graph. It is also used to
	// request StopNode() be called if processing should end.
	// error is used to report any errors; not that it will immediately stop all
	// graph processing.
	Process(args ProcessArgs, stage NodeStage, input Pins, output NodeOutput) error

	// Request to stop the node.
	StopNode(args StoppedArgs) error
}

Node performs abstract document processing.

type NodeDescr 

type NodeDescr struct {
	Id          string
	Name        string
	Purpose     string
	Cfgs        []CfgDescr
	StartupPins []PinDescr
	InputPins   []PinDescr
	OutputPins  []PinDescr
}

NodeDescr describes a node.

func (*NodeDescr) ClaString 

func (n *NodeDescr) ClaString() string

func (*NodeDescr) FindInput 

func (n *NodeDescr) FindInput(name string) *PinDescr

func (*NodeDescr) FindOutput 

func (n *NodeDescr) FindOutput(name string) *PinDescr

func (*NodeDescr) MarkdownString 

func (n *NodeDescr) MarkdownString() string

type NodeFactory 

type NodeFactory interface {
	Describe() NodeDescr
	Instantiate(args InstantiateArgs, tree interface{}) (Node, error)
}

NodeFactory instantiates a node. All nodes should have their factories installed in the bootstrap.

type NodeOutput 

type NodeOutput interface {
	SendPins(Pins)
	SendMsg(Msg)
}

NodeOutput sends data from the node to its parent runner.

type NodeStage 

type NodeStage string
const (
	NodeStarting NodeStage = "starting"
	NodeRunning            = "running"
)

type PhlyError 

type PhlyError struct {
	// contains filtered or unexported fields
}

func (*PhlyError) Error 

func (e *PhlyError) Error() string

func (*PhlyError) ErrorCode 

func (e *PhlyError) ErrorCode() int

type PinBuilder 

type PinBuilder struct {
	// contains filtered or unexported fields
}

func (PinBuilder) Add 

func (b PinBuilder) Add(name string, item *Doc) PinBuilder

func (PinBuilder) Pins 

func (b PinBuilder) Pins() Pins

type PinDescr 

type PinDescr struct {
	Name    string
	Purpose string
}

type Pins 

type Pins interface {
	GetPin(name string) Docs
	WalkPins(fn PinsWalkFunc)
}

Pins describes a collection of documents attached to named pins. It is meant for clients that are consuming pins.

func BuildPins 

func BuildPins(cmds ...interface{}) (Pins, error)

BuildPins() builds a new Pins object according to the command stream. See the tokens for the rules. Any cmd that is not a token or made use of by the last token becomes a new item in the current doc. Example. The first command defaults to being the channel (so it must be a string). 

BuildPins("input", "item1", "item2")

Example. You can use the channel command to create a new channel at any point. 

BuildPins(PbsChannel, "input", "item1", "item2")

Example. You can use the doc command to create a new doc at any point. 

BuildPins(PbsChannel, "input", "doc1_item1", PbsDoc, "doc2_item1")

func MustBuildPins 

func MustBuildPins(cmds ...interface{}) Pins

MustBuildPins() is identical to BuildPins but it will panic on an error. Intended only for testing

func RunApp 

func RunApp() (Pins, error)

type PinsWalkFunc 

type PinsWalkFunc func(name string, docs Docs)

PinsWalkFunc iterates over each pin in a Pins collection.

type Pipeline 

type Pipeline interface {
	Run(args StartArgs, input Pins) error
	Start(args StartArgs, input Pins) error
	Stop() error
	Wait() error
}

func LoadPipeline 

func LoadPipeline(name string) (Pipeline, error)

func ReadPipeline 

func ReadPipeline(r io.Reader) (Pipeline, error)

type ProcessArgs 

type ProcessArgs struct {
	// contains filtered or unexported fields
}

ProcessArgs provides arguments to the node during processing.

func (*ProcessArgs) ClaValue 

func (r *ProcessArgs) ClaValue(name string) string

ClaValue() answers the command line argument value for the given name.

func (*ProcessArgs) Env 

func (r *ProcessArgs) Env() Environment

func (*ProcessArgs) Filename 

func (r *ProcessArgs) Filename(rel string) string

Filename() answers an absolute filename for the supplied filename. Absolute filenames are returned as-is. Relative filenames are made relative to the cfg that generated the pipeline.

type SortNodeFactory 

type SortNodeFactory []NodeFactory

func (SortNodeFactory) Len 

func (s SortNodeFactory) Len() int

func (SortNodeFactory) Less 

func (s SortNodeFactory) Less(i, j int) bool

func (SortNodeFactory) Swap 

func (s SortNodeFactory) Swap(i, j int)

type SortVars 

type SortVars []varDescr

func (SortVars) Len 

func (s SortVars) Len() int

func (SortVars) Less 

func (s SortVars) Less(i, j int) bool

func (SortVars) Swap 

func (s SortVars) Swap(i, j int)

type StartArgs 

type StartArgs struct {
	Cla map[string]string // Command line arguments
	// contains filtered or unexported fields
}

StartArgs provides arguments when starting the pipeline.

type StopPayload 

type StopPayload struct {
	Err error
}

type StoppedArgs 

type StoppedArgs struct {
}

StoppedArgs provides information as nodes are stopping.

type StringItemFunc 

type StringItemFunc func(channel string, doc *Doc, index int, item string)

StringItemFunc is a callback for a single string item in a pin doc.

Source Files

View all Source files

Directories

Path Synopsis
tools/chatter

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.