kog

package module
v2.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 13, 2024 License: Apache-2.0 Imports: 44 Imported by: 0

README

Kog - process workflow engine (PWE) executor

A cross-platform CLI to execute a PWE job.

Execute Jobs

Kog supports executing a job either via docker directly or as a kubernetes job

  • --executor docker default.
    • Set in DOCKER_HOST environment variable to a valid docker host, for example tcp://localhost:2375
  • --executor k8s
    • This uses the KUBECONFIG env var and the current context to determine which cluster and namespace to use
Job Files

There are few job files checked in under data folder. The simplest workflow to execute is data/test/success.yaml, which executes echo command in alpine container.

kog run -f https://gitlab.com/f5-pwe/kog/raw/master/data/test/success.yaml
kog run -f /local/path/to/workflow.json
Example

Every Job has one or more workflows. In this example we have run and remediate. This is very useful if your workflow provisions physical hardware for exclusive use and you want that hardware left in the current state for investigation.

Every workflow step has four exit modes, success, failure, abort, and timeout Each exit mode can have its own step name to jump to

---
name: abort test due to env missing
description: long description of this job
current_workflow: run
current_step: start
# stop_on_failure as true is generally only used while debugging a workflow
# as it terminates the workflow on the step and does not follow the state machine
stop_on_failure: false
workflows:
    run:
        start:
            action: auto-success
            success: step2
            failure: notify
            abort: notify
            timeout: notify
            duration: 30m
        step2:
            action: auto-abort
            success: finalize
            failure: notify
            abort: notify
            timeout: notify
            duration: 30m
        notify:
            action: auto-success
            success: finalize
            failure: finalize
            abort: finalize
            timeout: finalize
            duration: 30m
        finalize:
            action: auto-remediatesuccess
            duration: 30m

    remediate:
        start:
            action: auto-success
            duration: 30m

actions:
    auto-success:
        image: quay.io/f5-pwe/auto-result
        version: 1.0.1
    auto-abort:
        image: quay.io/f5-pwe/auto-result
        version: 1.0.1
        env:
            RESULT: abort
            RESULT_CODE: '15'
            MESSAGE: you broke it
        # The requirements are verified at runtime
        requirements:
            env:
                - TEST
            context_paths:
                - build.product.version
        # This should contain any context structure created as output
        # by this action on success. The values should all be nil, this
        # is just so external tools can validate that the chain of actions will
        # contain the correct data.
        output:
            context:
                step_name: nil

context:
    build:
        product:
            name: Testing

The --file|-f flag works for both remote and local files encoded in either json or yaml

Workflow Action Container Specification
  1. Containers MUST have an entrypoint that reads the KOG_CONTEXT env var as a base64 encoded json serialized Context
  2. Containers MUST print to stdout a prefixed line that contains the json serialized ActionResult
    • eg KOG:{"result":"success","rc":0,"message":"auto success","context":{"action_name":"auto-success","correlation_id":"btzb6nwmekwj","run_id":"btzb6nwmengl","step_name":"start","task_name":"success test"}
    • the context from the ActionResult is passed into the next step in the workflow
  3. IF you support the docker executor type your Docker file MUST have an ENV KOG_ENV_CONTEXT=true
    • Example Dockerfile
    • This is just to tell the docker executor to not use stdin to pass the context, this is how it used to be done and I did not want to break backwards compatibility
Context

Every kog job and step has a Context that it is executed within. This context is passed to each workflow step, which can then use this context to configure details about how this step is configured. Each step returns a context that is used in the next step, so it is common to put results from this step into the context before returning it so the next step can use those details

  • --context /path/to/context.json

The --context flag works for both remote and local files encoded in either json or yaml

Environment Variables

There are three was to pass env vars to a workflow step

  1. --env VARIABLE=something
  2. --env-file /path/to/data.env this works for both remote and local files in the dotenv file format
  3. --import-current-env you SHOULD always use this with --ignore-env to exclude env vars you do NOT want passed to your workflow step

On docker the env variables are all passed in via --env and is thus inherently insecure, do NOT pass passwords, tokens, etc this way unless you are 100% running on a local docker daemon and are 100% sure the box is secure

On kubernetes all the env variables are saves as a kubernetes Secret then loaded into the job via the normal kubernetes methods. This is only remotely more secure then the docker method, so still do not pass passwords or tokens this way.

Volume Mounts

The docker executor supports mounting volumes into each step action, the symantics are identical to the docker --volume flag

Kubernetes does not yet support volume mounting. If you are interested in this feature feel free to do a MR

Logging

By default, Kog logs to console in logfmt format. It is possible to switch format to JSON by settings --log-formatter flag to json. The log settings will also be propagated to all spawned containers using passed context.

Remote Logging

To enable log aggregation, it is useful to set --remote-logger flag to syslog. You can also specify the remote logger address --remote-logger-addr 10.0.0.15:514 which is useful if you are passing your logs to logstash or some other log aggregation tool Enabling remote logger does not disable local logs, but adds a handler to ship logs to a remote location.

Status Notification

To enable step result notifications, set --notifier flag to either console, or https://url.example.com/kog/notifications This will enable sending a notification on every step start to keep track of a job progress. You can specify multiple notifiers for each run

HTTP Notifiers

passing a url to the notifier flag will enable kog to make an http POST request to that URL with the json serialized pwe.Job object as the body

Return Codes

By default the kog run command will use the job status to map to return codes, use --use-job-rc=false to always return 0 (except for issues with the kog command itself)

Upgrading and Switching to specific versions

As of v1.4.0 kog supports the upgrade and switch-to commands

  • kog upgrade to upgrade to the latest version
  • kog switch-to --version #.#.# to switch to a specific version
  • kog switch-to --list to list the latest 10 versions and show which version you are on.
$ kog switch-to --list
> 1.5.0
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0

Dev Environment Setup

  1. Install Go language on your system. You can use built-in package manager system and look for go or golang. For more advanced instruction, see official documentation.
  2. Create the folder for the project, this is important since Go expects packages to be under
mkdir -p ~/projects/f5-pwe
cd ~/projects//f5-pwe
  1. Clone this project
git clone https://gitlab.com/f5-pwe/kog
  1. Init the dependencies
cd kog
make init
  1. Compile and install kog into current directory
make build
  1. Verify you can run kog
./bin/kog-$GOOS-$GOARCH run -f https://gitlab.com/f5-pwe/kog/raw/master/data/test/success.yaml

Documentation

Overview

Copyright 2018 F5 Networks

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const LoggerContextKey ctxKey = iota

Variables

This section is empty.

Functions

func ContextLogger added in v2.0.2 

func ContextLogger(ctx context.Context, name string, fields ...zap.Field) (context.Context, *zap.Logger)

func DecodeResults 

func DecodeResults(result []byte) (actions.ActionResult, error)

func DecodeStepResults 

func DecodeStepResults(ctx variables.Context) (results []actions.ActionResult, err error)

func ExecuteStep 

func ExecuteStep(ctx context.Context, executor Executor, step *workflow.Step, job *pwe.Job, jobCTX variables.Context,
	notifierHooks *NotifyHooks, extraEnv map[string]string, volumes []string) actions.ActionResult

func FileOrURLReader 

func FileOrURLReader(ctx context.Context, taskFilename string) (reader io.Reader, err error)

func NewStepResult 

func NewStepResult(result actions.ActionStatus, message string) actions.ActionResult

func NextStep 

func NextStep(step *workflow.Step, code actions.ActionStatus) string

func ParseContext 

func ParseContext(data []byte) (ctx variables.Context, err error)

func ParseJob 

func ParseJob(data []byte) (job *pwe.Job, err error)

Types

type Executor 

type Executor interface {
	Prepare(ctx context.Context, job *pwe.Job) error
	CreateJob(ctx context.Context, inputData, correlationId, jobName, stepName, runId string, action *actions.Action, labels map[string]string, volumes []string) error
	GetJob() Job
	AttachIO(ctx context.Context) error
	SendInput(inputData string) error
	ProcessOutput(ctx context.Context, resultText *string)
	Wait(ctx context.Context, timeout time.Duration) (int, error)
	Cleanup(ctx context.Context) error
}

func NewDockerExecutor 

func NewDockerExecutor(ctx context.Context) (e Executor, err error)

func Newk8sExecutor 

func Newk8sExecutor(ctx context.Context) (e Executor, err error)

type Job 

type Job interface {
	ID() string
	Input(input io.WriteCloser) io.WriteCloser
	Output(output io.ReadCloser) io.ReadCloser
}

type JobResult 

type JobResult struct {
	ID            string        `json:"correlation_id" yaml:"correlation_id"`
	Name          string        `json:"task_name" yaml:"task_name"`
	Status        pwe.JobStatus `json:"status" yaml:"status"`
	Message       string        `json:"message" yaml:"message"`
	NotifyStatus  string        `json:"notify_status" yaml:"notify_status"`
	NotifyMessage string        `json:"notify_message" yaml:"notify_message"`
}

func ExecuteJob 

func ExecuteJob(ctx context.Context, executor Executor, job *pwe.Job, notifierHooks *NotifyHooks, extraEnv map[string]string, volumes []string) (retval JobResult)

type NotifyHook 

type NotifyHook interface {
	FireNotification(job *pwe.Job) error
}

func NewConsoleHook 

func NewConsoleHook() NotifyHook

func NewHTTPNotifyHook 

func NewHTTPNotifyHook(ctx context.Context, baseURL string) NotifyHook

type NotifyHooks 

type NotifyHooks []NotifyHook

func (*NotifyHooks) AddNotification 

func (h *NotifyHooks) AddNotification(hook NotifyHook)

func (NotifyHooks) FireNotification 

func (h NotifyHooks) FireNotification(job *pwe.Job) error

Source Files

View all Source files

Directories

Path Synopsis
cmd
kog

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.