atkmod

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2023 License: Apache-2.0 Imports: 14 Imported by: 2

README

atkmod: A go library API for module manifests

Go build

Note: this proof of concept is not dead, but as of Sept 21, 2022 is shelved while some other work. That being said, the module is in use by the itz CLI, which is also a proof of concept. This module includes some code for dealing with the Podman/Docker command line and output that is unit tested, and it was better to re-use this than copy and paste it or include it in the CLI outside this module.

The purpose of this project was originally to demonstrate a proof of concept for defining a "module file" that would be basically a descriptor used in a module's repository to provide a mechanism for dealing with the module using a Podman-based, container plugin architecture.

In other words, this was intended to prove an approach to answer the questions:

  • What if I didn't need to care how I got the parameters, or how I even deployed the module?
  • What if I left that up to plugins (in the form of containers that emit documented output)?

Overview

To accomplish these goals, this project uses a container-based, plugin-style architecture in which a manifest file defines a deployment lifecycle (see "Lifecycle Overview" for details) for the given module. This was inspired by build solutions, such as Drone CI to a great extent, especially the container-based execution steps.

This project includes an "executor" that simply stages of the lifecycle, using state to collect error conditions. For each plugin, the container mounts a local folder as a volume in the container and then executes the entrypoint to act upon the files in the volume. For example, during the deploy stage of the lifecycle, the itz-plugins/terrform-base plugin will use Terraform to apply the main.tf file and print the command's standard out to the plugin's standard out to be captured and used by whatever consumes this library.

Other plugins should print information to standard out for their stage in the lifecycle, such as the list-variables and get-state. See more in the documentation for the lifecycles.

What this is not

Notice, however, that other than the required input and output variables for the module that other dependency information is left out. That is by design. For the sake of separation of concerns, a module's dependencies and how a particular module should be installed and dealt with are two separate concerns. It's the opinionated stance of this library that it is only concerned with managing a module through its lifecycle via plugins. Dependency management--including downloading and resolving dependencies--are the responsibility of some other component.

Lifeycle Overview

The lifecycle is a set of three stages for the deployment: pre_deploy, deploy, and post_deploy. These stages are intended to be called in order listed, serially, with each stage completing successfully before continuing on to the next stage. That means that, by default, an error in pre_deploy stage means that the executor will not run the deploy and post_deploy stages.

Additionally, there are three hooks meta information about the lifecycle: list, validate, and get_state. These are called various times during the execution of the lifecycle should output information to standard output in a specific format and should also accept information via standard input in a specific format. You can read more on the expected formats in the documentation for the lifecycle hook. Because the hooks should get information about the module and the environment, they are covered first in this documentation.

Hook: get_state

The get_state hook is called by the executor to get the current state of the module. For each module, that could mean something slightly different. For example, for a module that configures basic networking that might mean returning the IP addresses of the existing network. Therefore, aside from the health element, the state returned as a response can be in any form returned in properly-formatted JSON within the data element. As example is show here:

{
  "health": {
    "status": "DEPLOYED",
    "lifecycle": {
      "pre_deploy": "SUCCESS",
      "deploy": "SUCCESS",
      "post_deploy": "SUCCESS"
    }
  },
  "data": {}
}

This state is passed along to the lifecycle plugins as STDIN. They can use it or ignore it--that's the flexibility of these plugins. The data element here should contain various state data important to the module. For example, if the module is a base VPC network in AWS, this state data may include the VPC and subnet IDs, IP addresses, etc.

Since health is reserved, a status of "DEPLOYED" should mean the module has been successfully deployed. In the lifecycle elements, each name maps exactly to the name of the stage in the lifecycle. This is intended to provide the capacity for retry or rollback logic, where a status of "SUCCESS" for the deploy stage of the lifecycle will be skipped by default. If the status were something else, the deploy stage may be retried, depending on the implementation of the plugin.

Hook: list

The responsibility of the list hook is to provide information about the module to the executor; most importantly is the list of input variables expected by the executor.

Implementations of this hook can vary from reading input files, such as .tfvars or tfinput files or source files.

Note: Output information should be included in the get_state hook.

Listed here is an example of the output from list:

{
  "specversion": "1.0",
  "type": "com.ibm.techzone.itz.tf_hook_list.response",
  "source": "https://github.ibm.com/skol/itz-deployer-plugins/tf-hook-list",
  "subject": "fyre-vm",
  "id": "7208f364-86af-4d18-8fcd-c1f5cd06cdb4",
  "time": "2023-02-13T17:17:48.570677",
  "datacontenttype": "application/json",
  "data": {
    "variables": [
      {
        "name": "TF_VAR_cloud_provider",
        "default": "fyre"
      },
      {
        "name": "TF_VAR_cloud_type",
        "default": "private"
      },
      {
        "name": "TF_VAR_fyre_api_key",
        "default": ""
      },
      {
        "name": "TF_VAR_fyre_root_password",
        "default": ""
      },
      {
        "name": "TF_VAR_fyre_username",
        "default": ""
      }
    ]
  }
}
Hook: validate

The validate hook provides a means to validate state of the module before executing any of the lifecycle stages. The validate hook should take the variables input as STDIN and return a status like the following JSON:

{
  "status": "OK",
  "messages": []
}

Shown here is an example of an error status that provides a meaningful error message to the caller:

{
  "status": "ERROR",
  "messages": [
    "Variable 'TF_VAR_cluster_api' is invalid."
  ]
}

The validate hook should be designed to validate single variables as well as the entire variable set or more and should be idempotent. This is so callers can call validate a single variable, such as in the case of an interactive prompt validating the input of a question before proceeding to the next, or validation of several variables at once in the case of a source or .env file that contains many input variables.

Stage: pre_deploy

The pre_deploy stage is used to initialize the workspace (working volume) to the state that it should be in prior to proceeding to the deploy stage. Plugin implementations for this stage could download any dependencies, run a command such as tf plan

Like the rest of the plugins, errors should result in a non-zero exit status from the container execution as well as some error messages written to STDOUT. See "Handling errors" for more information about the error message envelope.

Stage: deploy

The deploy stage is where the actual deployment of the module in the workspace is performed by the execution engine. In Terraform terms, this is where the Terraform plugin executes the tf apply command or the Ansible plugin executes the ansible-playbook or the CloudFormation plugin executes the aws cloudformation create-stack command or oc apply -f for an OpenShift application by default.

Regardless of the implementation, the plugin should actually deploy the module, whatever that means for the given module.

Stage: post_deploy

The post_deploy stage is where a plugin can perform cleanup, validation, writing state, etc., of the module.

The module manifest file

Examples of the module manifest file are best viewed in the test/examples directory, because those are the files that are run against unit tests and therefore verified against actual code. But, for convenience, an example is shown here:

# The apiVersion of the file. Supported values for the apiVersion are currently
# only v1alpha1. Any other value will cause an error when the file is being 
# loaded. itzcli is the namespace.
apiVersion: itzcli/v1alpha1
# "InstallManifest" is used for the type of file that is included in modules to 
# tell ITZ CLI how to install the module.
kind: InstallManifest

# Meta information about this project.
metadata:
  # The namespace for the module. This can be any value right now.
  namespace: IBMTechnologyZone
  # The name of the module. This really should match the name that is displayed
  # to users in software catalogs, etc.
  name: MyModule
  # Any arbitrary labels for the module. Reserved for future use.
  labels:
    "label1": value1

spec:

  # Hooks are not part of the lifecycle of the module but are called at various
  # points during the lifecycle to validate state and lifecycle completeness.
  hooks:
    # Uses the container specified by "image" to get a list of the parameters
    # for the project. This is either a custom container or command specified
    # by the maintainer, or could be a "plugin" that is supported by the
    # ITZ CLI.
    list:
      image: something/parameter-lister:latest
      env:
        - name: MY_PROJECT_NAME
          value: my-base-project
      volumeMounts:
        - mountPath: /workspace
          name: ${HOME}/.itz/cache

    # Similar to list (above), but uses the container to validate the values
    # for the parameters.
    validate:
      image: something/parameter-validator:latest

    # Gets the current state of the project and returns a structure documented at
    get_state:
      image: something/get-stater:latest

  lifecycle:

    # Uses the container specified by image to run any pre-deployment tasks for
    # the project. This could be, for example, generating files in the project
    # based on metadata before actually starting the deployment step.
    pre_deploy:
      image: something/pre-deployer:latest

    # Uses the container specified by image to run the deployment
    deploy:
      image: something/deployer:latest

    # Uses the container specified by image to run post-deployment steps, such
    # as clean-ups, notifications, etc.
    post_deploy:
      image: something/post-deployer:latest

The included Podman/Docker API

In order to read the img tag in the module manifest and do something with it, capturing the output, errors, etc. in an elegant fashion, I implemented a command builder using the Builder Patter and also incorporated the notion of contexts, which is similar to a Pipeline in that several commands can be strung together (for example, to execute the entire lifecycle of a module) and be contextually aware.

An example of using the PodmanCliCommandBuilder is shown here:

builder := atk.NewPodmanCliCommandBuilder(nil)

actual, err := builder.WithVolume("/home/myuser/workdir").
WithImage("localhost/myimage").
WithEnvvar("MYVAR", "thisismyvalue").
Build()

assert.Nil(t, err)
assert.Equal(t, "/usr/local/bin/podman run --rm -v /home/myuser/workdir:/workspace -e MYVAR=thisismyvalue localhost/myimage", actual)

More examples of using the builder can be found in podmanclibuilder_test.go.

Developing your own plugin

There are few basic rules for the plugins:

  1. Make sure to check the plugin documentation to understand the required STDIN and/or STDOUT of the plugin.
  2. Make sure to use proper UNIX exit codes--use zero for success and non-zero for failure or error conditions. For your own trouble-shooting, consider making your non-zero exit codes mean something--that is, a code of 135 means something different than a 140.
  3. Use STDOUT and STDERR properly. STDOUT is reserved for JSON output sent to executors, while STDERR is should be used for process debugging or logging messages that are either displayed to a console or printed to a log file.

Fortunately, there (will be) a container that you can call in your CI/CD pipeline to validate

Reference implementations

Reference implementations are in progress.

FAQ

Why not just use... [TravisCI, Drone, Jenkins, kubectl, Airflow, WorkflowXYX...]

I looked, and looked pretty hard, and even evaluated some of the command line runners such as that of AWS CodeBuild. Afterall, this is primarily a Day Zero installer--Day One operations should be handled by GitOps, DevOps, or DevSecOps pipelines. For a while, even, I used a Jenkins container and tried to basically implement this within Jenkins.

However, it turned out that building a very lightweight, purpose-built executor that deferred execution to container-based plugins was the best solution for both speed of development, lowest impact on modules, backwards compatibility, and future-proofing. For example, any current Terraform project such as that currently deployed in TechZone should--for the most part--simply be able to use the supported plugins and therefore require no additional code other than the itz-manifest.yaml file.

Why not just use settle on one specific tech (eg., Terraform) and use its built-in goodness?

This didn't seem like a realistic goal, long-term, and likely would drive some sub-optimal behavior. For example, Ansible is a great choice for deploying and managing configuration and infrastructure, so providing the ability to use Terraform or CloudFormation or Bicep for some infrastructure while providing Ansible or Helm or make for others is inline with both hybrid cloud and heterogeneous ecosystem approaches.

Documentation

Index

Constants

View Source 
const (
	ListHookResponseEvent           ModuleEventType = "com.ibm.techzone.cli.hook.list.response"
	ValidateHookResponseEvent       ModuleEventType = "com.ibm.techzone.cli.hook.validate.response"
	ValidateHookRequestEvent        ModuleEventType = "com.ibm.techzone.cli.hook.validate.request"
	GetStateHookResponseEvent       ModuleEventType = "com.ibm.techzone.cli.hook.get_state.response"
	GetStateHookRequestEvent        ModuleEventType = "com.ibm.techzone.cli.hook.get_state.request"
	PreDeployLifecycleRequestEvent  ModuleEventType = "com.ibm.techzone.cli.lifecycle.pre_deploy.request"
	DeployLifecycleRequestEvent     ModuleEventType = "com.ibm.techzone.cli.lifecycle.deploy.request"
	PostDeployLifecycleRequestEvent ModuleEventType = "com.ibm.techzone.cli.lifecycle.post_deploy.request"
	LoggerContextKey                AtkContextKey   = "atk.logger"
	StdOutContextKey                AtkContextKey   = "atk.stdout"
	StdErrContextKey                AtkContextKey   = "atk.stderr"
	BaseDirectory                   AtkContextKey   = "atk.basedir"
	ListHook                        Hook            = "list"
	ValidateHook                    Hook            = "validate"
	GetStateHook                    Hook            = "get_state"
)

Variables

View Source 
var DefaultOrder = []State{
	Invalid,
	Initializing,
	Configured,
	Validated,
	PreDeploying,
	PreDeployed,
	Deploying,
	Deployed,
	PostDeploying,
	PostDeployed,
	Done,
}

Functions

func DoneHandler 

func DoneHandler(ctx *RunContext, notifier Notifier) error

func Iif 

func Iif(value string, orValue string) string

func LoadEvent added in v0.1.1 

func LoadEvent(eventS string) (*cloudevents.Event, error)

func NoopHandler 

func NoopHandler(ctx *RunContext, notifier Notifier) error

NoopHandler is an implementation of the Null Object pattern. It does nothing except to insure we don't return a nil.

func NoopHookCmd 

func NoopHookCmd(ctx *RunContext) error

func WriteEvent added in v0.1.1 

func WriteEvent(event *cloudevents.Event, out io.Writer) error

Types

type ApiVersion 

type ApiVersion struct {
	Namespace string
	Version   string
}

func ParseApiVersion 

func ParseApiVersion(val string) (*ApiVersion, error)

ParseApiVersion parses the version of the apiVersion

func (ApiVersion) String 

func (a ApiVersion) String() string

type AtkContextKey 

type AtkContextKey string

type CliModuleRunner 

type CliModuleRunner struct {
	PodmanCliCommandBuilder
}

func (*CliModuleRunner) Run 

func (r *CliModuleRunner) Run(ctx *RunContext) error

Run runs the container that has been defined in the builder setup.

func (*CliModuleRunner) RunImage 

func (r *CliModuleRunner) RunImage(ctx *RunContext, info ImageInfo) error

RunImage runs the container that is defined in the provided ImageInfo

type CliParts 

type CliParts struct {
	Path             string
	Cmd              string
	Image            string
	Flags            []string
	Workdir          string
	VolumeMaps       []string
	DefaultVolumeOpt string
	Ports            map[string]string
	UidMaps          []string
	Envvars          []EnvVarInfo
	// TODO: Add command support that will be used instead of an entrypoint
	Commands []string
}

CliParts represents the parts of the entire podman command line.

type CmdItr 

type CmdItr interface {
	Next() (StateCmd, bool)
}

type DeployableModule 

type DeployableModule struct {
	// contains filtered or unexported fields
}

func NewDeployableModule 

func NewDeployableModule(runCtx *RunContext, module *ModuleInfo) *DeployableModule

func (*DeployableModule) AddCmd 

func (m *DeployableModule) AddCmd(status State, handler StateCmd) error

func (*DeployableModule) GetCmdFor 

func (m *DeployableModule) GetCmdFor(status State) StateCmd

func (*DeployableModule) GetHook 

func (m *DeployableModule) GetHook(name Hook) HookCmd

func (*DeployableModule) IsErrored 

func (m *DeployableModule) IsErrored() bool

func (*DeployableModule) Itr 

func (m *DeployableModule) Itr() (NextFunc, bool)

func (*DeployableModule) Notify 

func (m *DeployableModule) Notify(state State) error

func (*DeployableModule) NotifyErr 

func (m *DeployableModule) NotifyErr(state State, err error)

func (*DeployableModule) State 

func (m *DeployableModule) State() State

type EnvVarInfo 

type EnvVarInfo struct {
	Name  string `json:"name" yaml:"name"`
	Value string `json:"value" yaml:"value"`
}

func (*EnvVarInfo) String 

func (e *EnvVarInfo) String() string

type EventData added in v0.1.1 

type EventData struct {
	Variables []EventDataVarInfo `json:"variables,omitempty" yaml:"variables,omitempty"`
}

func LoadEventData added in v0.1.1 

func LoadEventData(event *cloudevents.Event) (*EventData, error)

type EventDataVarInfo added in v0.1.1 

type EventDataVarInfo struct {
	Name        string `json:"name" yaml:"name"`
	Value       string `json:"value,omitempty" yaml:"value,omitempty"`
	Default     string `json:"default,omitempty" yaml:"default,omitempty"`
	Description string `json:"description,omitempty" yaml:"description,omitempty"`
}

type Hook added in v0.1.1 

type Hook string

type HookCmd 

type HookCmd func(ctx *RunContext) error

type HookInfo 

type HookInfo struct {
	GetState ImageInfo `json:"get_state" yaml:"get_state"`
	List     ImageInfo `json:"list" yaml:"list"`
	Validate ImageInfo `json:"validate" yaml:"validate"`
}

type ImageInfo 

type ImageInfo struct {
	Image   string       `json:"image" yaml:"image"`
	Script  string       `json:"script" yaml:"script"`
	Command []string     `json:"command" yaml:"command"`
	Args    []string     `json:"args" yaml:"args"`
	EnvVars []EnvVarInfo `json:"env" yaml:"env"`
	Volumes []VolumeInfo `json:"volumeMounts" yaml:"volumeMounts"`
}

type LifecycleInfo 

type LifecycleInfo struct {
	PreDeploy  ImageInfo `json:"pre_deploy" yaml:"pre_deploy"`
	Deploy     ImageInfo `json:"deploy" yaml:"deploy"`
	PostDeploy ImageInfo `json:"post_deploy" yaml:"post_deploy"`
}

type ManifestFileLoader 

type ManifestFileLoader struct {
	// contains filtered or unexported fields
}

func NewAtkManifestFileLoader 

func NewAtkManifestFileLoader() *ManifestFileLoader

func (*ManifestFileLoader) Load 

func (l *ManifestFileLoader) Load(uri string) (*ModuleInfo, error)

type MetadataInfo 

type MetadataInfo struct {
	Name      string            `json:"name" yaml:"name"`
	Namespace string            `json:"namespace" yaml:"namespace"`
	Labels    map[string]string `json:"labels" yaml:"labels"`
}

type ModuleEventType added in v0.1.1 

type ModuleEventType string

type ModuleInfo 

type ModuleInfo struct {
	ApiVersion     string       `json:"apiVersion" yaml:"apiVersion"`
	Kind           string       `json:"kind" yaml:"kind"`
	Metadata       MetadataInfo `json:"metadata" yaml:"metadata"`
	Specifications SpecInfo     `json:"spec" yaml:"spec"`
}

func (*ModuleInfo) IsSupported 

func (m *ModuleInfo) IsSupported() bool

IsSupported returns true if the manifest file is supported.

func (*ModuleInfo) IsSupportedKind 

func (m *ModuleInfo) IsSupportedKind() bool

IsSupportedKind returns true if the kind is supported.

func (*ModuleInfo) IsSupportedVersion 

func (m *ModuleInfo) IsSupportedVersion() bool

IsSupportedVersion returns true if apiVersion is supported.

type ModuleLoader 

type ModuleLoader interface {
	Load(uri string) (ModuleInfo, error)
}

type NextFunc 

type NextFunc func() (StateCmd, bool)

type Notifier 

type Notifier interface {
	State() State
	Notify(State) error
	NotifyErr(State, error)
}

type PodmanCliCommandBuilder 

type PodmanCliCommandBuilder struct {
	// contains filtered or unexported fields
}

PodmanCliCommandBuilder allows you to build the podman command in a way that is already unit tested and verified so that you do not have to append your own strings or do variable interpolation.

func NewPodmanCliCommandBuilder 

func NewPodmanCliCommandBuilder(cli *CliParts) *PodmanCliCommandBuilder

NewPodmanCliCommandBuilder creates a new PodmanCliCommandBuilder with the given configuration. If there is no configuration provided (nil), or if certain values are not defined, then the constructor will provide reasonable defaults.

func (*PodmanCliCommandBuilder) Build 

func (b *PodmanCliCommandBuilder) Build() (string, error)

Build builds the command line for the container command

func (*PodmanCliCommandBuilder) BuildFrom 

func (b *PodmanCliCommandBuilder) BuildFrom(info ImageInfo) (string, error)

func (*PodmanCliCommandBuilder) WithEnvvar 

func (b *PodmanCliCommandBuilder) WithEnvvar(name string, value string) *PodmanCliCommandBuilder

WithEnvvar adds the given environment variable and value to the command. It is the same thing as adding -e ENVAR=value as a parameter to the container command.

func (*PodmanCliCommandBuilder) WithImage 

func (b *PodmanCliCommandBuilder) WithImage(imageName string) *PodmanCliCommandBuilder

WithImage specifies the container image used in the command.

func (*PodmanCliCommandBuilder) WithPath 

func (b *PodmanCliCommandBuilder) WithPath(path string) *PodmanCliCommandBuilder

WithPath allows you to override the default path of /usr/local/bin/podman for podman.

func (*PodmanCliCommandBuilder) WithPort 

func (b *PodmanCliCommandBuilder) WithPort(localport string, containerport string) *PodmanCliCommandBuilder

WithPort adds a port mapping to the command

func (*PodmanCliCommandBuilder) WithUserMap 

func (b *PodmanCliCommandBuilder) WithUserMap(localUser int, containerUser int, number int) *PodmanCliCommandBuilder

func (*PodmanCliCommandBuilder) WithVolume 

func (b *PodmanCliCommandBuilder) WithVolume(localdir string, containerdir string) *PodmanCliCommandBuilder

WithVolume adds a volume mapping to the command.

func (*PodmanCliCommandBuilder) WithVolumeOpt 

func (b *PodmanCliCommandBuilder) WithVolumeOpt(localdir string, containerdir string, option string) *PodmanCliCommandBuilder

WithVolume adds a volume mapping to the command.

func (*PodmanCliCommandBuilder) WithWorkspace 

func (b *PodmanCliCommandBuilder) WithWorkspace(localdir string) *PodmanCliCommandBuilder

WithWorkspace is a shortcut to adding a local path to the "workspace" on the container.

type RunContext 

type RunContext struct {
	Context     context.Context
	In          io.Reader
	Out         io.Writer
	Log         logger.Logger
	Err         io.Writer
	Errors      []error
	LastErrCode int
}

func (*RunContext) AddError 

func (c *RunContext) AddError(err error)

AddError adds an error to the context

func (*RunContext) IsErrored 

func (c *RunContext) IsErrored() bool

IsErrored returns true if there are errors in the context

func (*RunContext) Reset 

func (c *RunContext) Reset()

func (*RunContext) SetLastErrCode 

func (c *RunContext) SetLastErrCode(errCode int)

type SpecInfo 

type SpecInfo struct {
	Hooks     HookInfo      `json:"hooks" yaml:"hooks"`
	Lifecycle LifecycleInfo `json:"lifecycle" yaml:"lifecycle"`
}

type State 

type State string
const (
	None          State = "none"
	Invalid       State = "invalid"
	Initializing  State = "initializing"
	Configured    State = "configured"
	Validated     State = "validated"
	PreDeploying  State = "predeploying"
	PreDeployed   State = "predeployed"
	Deploying     State = "deploying"
	Deployed      State = "deployed"
	PostDeploying State = "postdeploying"
	PostDeployed  State = "postdeployed"
	Done                = PostDeployed
	Errored       State = "errored"
)

type StateCmd 

type StateCmd func(ctx *RunContext, notifier Notifier) error

StateCmd is an implementation of a Command pattern

type StateCmder 

type StateCmder interface {
	AddCmd(State, StateCmd) error
	GetCmdFor(State) StateCmd
}

type VolumeInfo 

type VolumeInfo struct {
	MountPath string `json:"mountPath" yaml:"mountPath"`
	Name      string `json:"name" yaml:"name"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.