exec

package
v0.64.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 24, 2023 License: Apache-2.0 Imports: 20 Imported by: 0

README

Execution service

The execution service is responsible for opening, managing terminal session, with the ability to send command and extract data. Service keeps tracking the session current directory commands, env variable setting, and will only issue SSH command if there is actual change.

Command execution

Command executor uses SSH service, to run all commands.

Each command is defined with the following:

  • When: optional run match criteria i.e. $stdout:/password/"
  • Command: command to be send to STDIN
  • Extract: optional data scraping rules
  • Errors: optional matching fragments resulting in error (blacklist)
  • Success: optional matching fragment defining success (whitelist)
  • TimeoutMs: optional command timeout

Each command has access to previous commands output with: - current workflow state - $stdout all run request commands output upto execution point - $cmd[ index ].stdout individual command output

Each command is sent to SSH session STDIN with optional response timeout and custom terminators to detect command result. Besides the custom terminators, shell prompt is also used to detect command ended. If stdout produces no output or no output is match, run command wait specified timeout (default 20sec).

    sshSession.Run(command, listener, timeoutMs, terminators...)

Optionally after each step, command execution code is check with echo $? if checkError flag is set (false by default).

@run.yaml

pipeline:
  testMe:
    action: exec:extract
    systemPaths:
      - /opt/sdk/go/bin
    commands:
      - command: go version
        extract:
          - Key: Version
            RegExpr: go(\d\.\d\d)
      - command: echo 'YOUR GO VERSION is $Version'

      - command: passwd tester
        terminators:
          - Old Password
        timeoutMs: 10000

      - command: changme
        terminators:
          - New Password
        success:
          - New Password
        timeoutMs: 10000

      - command: testerPass@1
        terminators:
          - Retype New Password
        timeoutMs: 10000

      - command: testerPass@1
        terminators:
          - Retype New Password

      - command: echo 'Done'

Usage

Credentials
Default credentials

By default SSH 'exec' service uses ssh://127.0.0.1:22 and ${env.USER} with private key auth. The private key has to exists in either locations:

  • ${env.HOME}/.secret/id_rsa
  • ${env.HOME}/.ssh/id_rsa
endly default_cred.yaml

@default_cred.yaml

pipeline:
  task1:
    action: exec:run
    commands:
      - hostname
      - echo 'welcome ${os.user} on $TrimSpace($cmd[0].stdout)'

You can also set default credentials with the following

pipeline:
  task1:
    action: exec:setTarget
    url: ssh://myCloudMatchine/
    credentials: myCloudCredentials

In that case you can skip defining target in all service using SSH exec service.

Custom credentials

When default method is not available you can generate encrypted credentials for your user useing the following instruction

To connect to host: myhost.com with myuser/mypassword

  1. Create credentials file with myuser credentials 
    endly -c=myuser-myhost
  2. Define workflow with target attribute 
    endly custom_cred.yaml

@custom_cred.yaml

pipeline:
  task1:
    action: exec:run
    target:
      URL: ssh://myhost:com/
      credentials: myuser-myhost
    commands:
      - hostname
      - echo 'welcome ${os.user} on $TrimSpace($cmd[0].stdout)'
Conditional command execution

Conditional execution uses the following syntax: COND ? WHEN_TRUE

endly cond.yaml

@cond.yaml


pipeline:
  myConTask:
    action: exec:run
    commands:
      - ls -al /tmp/myapp
      - ${cmd[0].stdout}:/No such file or directory/?  mkdir -p /tmp/myapp
      - ls -al /tmp/myapp
      - ${cmd[2].stdout}:/No such file or directory/? echo 'failed to create app folder'


  debugInfo:
    action: print
    message: $AsJSON($myConTask)


  nextStep:
    when: ${myConTask.Output}:!/failed/
    action: print
    message: Created app folder, moving to next step ...


endly cond_external_arg.yaml p=123

@cond_external_arg.yaml

pipeline:
  myConTask:
    action: exec:run
    commands:
      - $p = 123 ? echo 'p was $p'
      - echo 'done'
  myDebugInfo:
    action: print
    message: $myConTask.Output
Scraping data
endly scrape.yaml

@scrape.yaml

pipeline:
  extract:
    action: exec:run
    commands:
      - whoami
      - cat /etc/hosts
    extract:
      - key: aliases
        regExpr: (?sm)\s+127.0.0.1(.+)

  info:
    action: print
    message: "Extracted: ${extract.Data.aliases}"
Super user mode

To run command as super user set superUser to true

endly super.yaml

@super.yaml

init:
  target:
    URL: ssh://127.0.0.1/
    credentials: localhost

pipeline:
  myConTask:
    action: exec:run
    target: $target
    superUser: true
    commands:
      - whoami
      - mkdir /tmp/app2
      - chown ${os.user} /tmp/app2
      - ls -al /tmp/app2
Auto sudo mode

To run command as super user only when there is permission needed, set autoSudo flag

endly sudo.yaml

@auto_sudo.yaml

init:
  target:
    URL: ssh://127.0.0.1/
    credentials: localhost

pipeline:
  myConTask:
    action: exec:run
    target: $target
    autoSudo: true
    commands:
      - whoami
      - mkdir /opt/myapp
      - chown ${os.user} /opt/myapp
      - ls -al /opt/myapp
Controlling error
Commands error

By default any command error exit code is ignored, to enable command exit code check set checkError attribute.

endly check_errors.yaml

@check_errors.yaml

pipeline:
  build:
    action: exec:run
    checkError: true
    commands:
      - export GO111MODULE=on
      - unset GOPATH
      - cd $appPath
      - go build
Custom error detection

In some scenario, when a command returns success (0) code, you may still terminate command execution based on command output.

endly custom_error.yaml

@custom_error.yaml

pipeline:
  task1:
    action: exec:run
    errors:
      - myError
    commands:
      - echo 'starting .. '
      - echo ' myError'
      - echo 'done.'
Handling secrets

For security reason credentials should never be store in plain form, neither reveal on the terminal or any logs files.

Imaging that you have to build an app that uses private git repository.

  1. Encrypt private git repo credentials endly -c='myuser-git'
  2. Check created credentials ls -al ~/.secret/myuser-git.json
  3. Define workflow with secrets section. 
    pipeline:

  build:
    action: exec:run
    checkError: true
    terminators:
      - Password
      - Username
    secrets:
      gitSecrets: myuser-git
    commands:
      - export GIT_TERMINAL_PROMPT=1
      - export GO111MODULE=on
      - unset GOPATH
      - cd ${appPath}/
      - go build
      - '${cmd[3].stdout}:/Username/? $gitSecrets.Username'
      - '${output}:/Password/? $gitSecrets.Password'
Session variables:
  • ${os.user}
  • ${cmd[x].stdout}
  • $stdout
  • $secrets

Contract

Run the following command for exec service operation details:


endly -s=exec

endly -s=exec -a=run
endly -s=exec -a=extract
endly -s=exec -a=open
endly -s=exec -a=close
Service Id Action Description Request Response
exec open open SSH session on the target resource. OpenSessionRequest OpenSessionResponse
exec close close SSH session CloseSessionRequest CloseSessionResponse
exec run execute basic commands RunRequest RunResponse
exec extract execute commands with ability to extract data, define error or success state ExtractRequest RunResponse

Code level integration

SSH Session

In order to run any SSH command, service needs to open a session, it uses target.Credentials and secret service to connect the target host.

Opening session is an optional step, run or extract request will open session automatically.

By default session is open in non transient mode, which means once context.Close is called, session will be will be terminated. Otherwise caller is responsible for closing it.

    
        manager := endly.New()
        context := manager.NewContext(toolbox.NewContext())
        target := url.NewResource("ssh://127.0.0.1", "~/.secret/localhost.json")
        defer context.Close() // session closes as part of context.Close
        response, err := manager.Run(context, exec.NewOpenSessionRequest(target, []string{"/usr/local/bin"}, map[string]string{"M2_HOME":"/users/test/.m2/"},false, "/"))
        if err != nil {
            log.Fatal(err)
        }
        openResponse := response.(*exec.OpenSessionResponse)
        sessions :=context.TerminalSessions()
        assert.True(t,sessions.Has(openResponse.SessionID))
        log.Print(openResponse.SessionID)

Run vs Extract:

RunReuest provide a simple way to execute SSH command with conditional execution, it uses util.StdErrors as stdout errors. ExtractRequest has ability to fine tune SSH command execution with extraction data ability. Error matching in ExtractRequest does use any default value.

Run Commands

Command in RunRequest can represents one of the following:

  1. Simple command: i.e echo $HOME
  2. Conditional command: [criteria ?] command i.e. $stdout:/root/? echo 'hello root'",

    manager := endly.New()
    context := manager.NewContext(toolbox.NewContext())
    var target= url.NewResource("ssh://127.0.0.1", "localhost")
    var runRequest = exec.NewRunRequest(target, true, "whoami", "$stdout:/root/? echo 'hello root'")
    var runResponse = &exec.RunResponse{}
    err := endly.Run(context, runRequest, runResponse)

Run Command to Extract Data

    extractRequest := exec.NewExtractRequest(target,
		exec.DefaultOptions(),
		exec.NewExtractCommand(fmt.Sprintf("svn info"), "", nil, nil,
			endly.NewDataExtraction("origin", "^URL:[\\t\\s]+([^\\s]+)", false),
			endly.NewDataExtraction("revision", "Revision:\\s+([^\\s]+)", false)))
    manager := endly.New()
    context := manager.NewContext(toolbox.NewContext())
    var runResponse := &exec.RunResponse{}
    err := endly.Run(context, extractRequest, runResponse)
    if err != nil {
        log.Fatal(err)
    }

Exec SSH Unit Testing

This module provide SSH session recording ability to later replay it during unit testing without actual SSH involvement.

Recording SSH session

To record actual SSH session use exec.OpenRecorderContext helper method, the last parameters specify location where conversation is recorded, actual dump takes place when context is closed (defer context.Clode()). If you use sudo. any secret or credentials make sure that you rename it to *** before checking in any code so you can use var credentials, err = util.GetDummyCredential()

	manager := endly.New()
	target := url.NewResource("ssh://127.0.0.1", "~/.secret/localhost.json")
	context, err :=  exec.NewSSHRecodingContext(manager, target, "test/session/context")
	if err != nil {
		log.Fatal(err)
	}
	defer context.Close()

Replaying SSH session

In order to replay previously recoded SSH session use exec.GetReplayService helper method to create a test SSHService, use location of stored SSH conversation as parameter, then create context with exec.OpenTestContext

	manager := endly.New()
	var credentials, err = util.GetDummyCredential()
	if err != nil {
		log.Fatal(err)
	}
	target := url.NewResource("ssh://127.0.0.1", credentials)
	context, err := exec.NewSSHReplayContext(manager, target, "test/session/transient")
	if err != nil {
		log.Fatal(err)
	}
	response, err := manager.Run(context, exec.NewOpenSessionRequest(target, []string{"/usr/local/bin"}, map[string]string{"M2_HOME": "/users/test/.m2/"}, false, "/"))
    if err != nil {
        log.Fatal(err)
    }

Documentation

Index

Constants

View Source 
const ServiceID = "exec"

ServiceID represent system executor service id

View Source 
const SudoCredentialKey = "**sudo**"

SudoCredentialKey represent obsucated password sudo credentials key (target.Credentials)

Variables

View Source 
var CommandErrors = []string{util.CommandNotFound, util.NoSuchFileOrDirectory, util.ErrorIsNotRecoverable}

Functions

func GetReplayService 

func GetReplayService(basedir string) (ssh.Service, error)

GetReplayService return replay service

func GetServiceTarget added in v0.41.0 

func GetServiceTarget(target *url.Resource) *url.Resource

GetServiceTarget sets default target URL, credentials if emtpy

func New 

func New() endly.Service

New creates a new execution service

func NewSSHMultiRecordingContext 

func NewSSHMultiRecordingContext(manager endly.Manager, sessions map[string]*url.Resource) (*endly.Context, error)

NewSSHMultiRecordingContext open multi recorded session

func NewSSHMultiReplayContext 

func NewSSHMultiReplayContext(manager endly.Manager, sessions map[string]*url.Resource) (*endly.Context, error)

OpenMultiSessionTestContext opens test context with multi SSH replay/mocks session

func NewSSHRecodingContext 

func NewSSHRecodingContext(manager endly.Manager, target *url.Resource, sessionDir string) (*endly.Context, error)

NewSSHRecodingContext open recorder context (to capture SSH command)

func NewSSHReplayContext 

func NewSSHReplayContext(manager endly.Manager, target *url.Resource, basedir string) (*endly.Context, error)

NewSSHReplayContext opens test context with SSH commands to replay

func OperatingSystem 

func OperatingSystem(context *endly.Context, sessionName string) *model.OperatingSystem

Os returns operating system for provide session

func SessionID added in v0.40.0 

func SessionID(context *endly.Context, target *url.Resource) string

SessionID returns session I

func SetDefaultTarget added in v0.41.0 

func SetDefaultTarget(context *endly.Context, target *url.Resource)

func TerminalSession 

func TerminalSession(context *endly.Context, target *url.Resource) (*model.Session, error)

TerminalSession returns Session for passed in target resource.

func TerminalSessions 

func TerminalSessions(context *endly.Context) model.Sessions

TerminalSessions returns system sessions

Types

type CloseSessionRequest 

type CloseSessionRequest struct {
	SessionID string
}

CloseSessionRequest closes session

type CloseSessionResponse 

type CloseSessionResponse struct {
	SessionID string
}

CloseSessionResponse closes session response

type Command 

type Command string

Command represents a command expression: [when criteria ?] command

func (Command) String 

func (c Command) String() string

String returns command string

func (Command) WhenAndCommand 

func (c Command) WhenAndCommand() (string, string)

WhenAndCommand extract when criteria and command

type ExtractCommand 

type ExtractCommand struct {
	When    string         `description:"only run this command is criteria is matched i.e $stdout:/password/"`                                           //only run this execution is output from a previous command is matched
	Command string         `required:"true" description:"shell command to be executed"`                                                                  //command to be executed
	Extract model.Extracts `description:"stdout data extraction instruction"`                                                                            //Stdout data extraction instruction
	Errors  []string       `description:"fragments that will terminate execution with error if matched with standard output, in most cases leave empty"` //fragments that will terminate execution with error if matched with standard output
	Success []string       ``                                                                                                                            //if specified absence of all of the these fragment will terminate execution with error.
	/* 126-byte string literal not displayed */
	Terminators []string `description:"terminators"`
	TimeoutMs   int      `description:"timeoutMs stdout wait timeout "`
}

Extracts represents an execution instructions

func NewExtractCommand 

func NewExtractCommand(command, when string, success, errors []string, extractions ...*model.Extract) *ExtractCommand

NewExtractCommand creates a new extract command

func (*ExtractCommand) Init 

func (c *ExtractCommand) Init() error

type ExtractRequest 

type ExtractRequest struct {
	Target *url.Resource `required:"true" description:"host where command runs" ` //execution target - destination where to run a command.
	*Options
	Commands []*ExtractCommand `description:"command with data extraction instruction "` //extract command
}

ExtractRequest represents managed command request

func NewExtractRequest 

func NewExtractRequest(target *url.Resource, options *Options, commands ...*ExtractCommand) *ExtractRequest

NewExtractRequest returns a new command request

func NewExtractRequestFromURL 

func NewExtractRequestFromURL(URL string) (*ExtractRequest, error)

NewExtractRequestFromURL creates a new request from URL

func (*ExtractRequest) Clone 

func (r *ExtractRequest) Clone(target *url.Resource) *ExtractRequest

Clones clones requst with supplide target

func (*ExtractRequest) Init 

func (r *ExtractRequest) Init() error

Init initialises request

func (*ExtractRequest) Validate 

func (r *ExtractRequest) Validate() error

Validate validates managed command request

type Log 

type Log struct {
	Stdin  string
	Stdout string
	Error  string
}

Log represents an executed command with Stdin, Stdout or Error

func NewCommandLog 

func NewCommandLog(stdin, stdout string, err error) *Log

NewCommandLog creates a new command log

type OpenSessionRequest 

type OpenSessionRequest struct {
	Target        *url.Resource      //Session is created from target host (servername, port)
	Config        *ssh.SessionConfig //ssh configuration
	SystemPaths   []string           //system path that are applied to the ssh session
	Env           map[string]string
	Transient     bool        //if this flag is true, caller is responsible for closing session, othewise session is closed as context is closed
	Basedir       string      //capture all ssh service command in supplied dir (for unit test only)
	ReplayService ssh.Service //use Ssh ReplayService instead of actual SSH service (for unit test only)
}

OpenSessionRequest represents an open session request.

func NewOpenSessionRequest 

func NewOpenSessionRequest(target *url.Resource, systemPaths []string, env map[string]string, transient bool, basedir string) *OpenSessionRequest

NewOpenSessionRequest creates a new session if transient flag is true, caller is responsible for closing session, otherwise session is closed as context is closed

func (*OpenSessionRequest) Validate 

func (r *OpenSessionRequest) Validate() error

Validate checks if request is valid

type OpenSessionResponse 

type OpenSessionResponse struct {
	SessionID string
}

OpenSessionResponse represents a session id

type Options 

type Options struct {
	SystemPaths []string `description:"path that will be appended to the current SSH execution session the current and future commands"` //path that will be added to the system paths
	Terminators []string ``                                                                                                              //fragment that helps identify that command has been completed - the best is to leave it empty, which is the detected bash prompt
	/* 141-byte string literal not displayed */
	Errors    []string          `description:"fragments that will terminate execution with error if matched with standard output, in most cases leave empty"` //fragments that will terminate execution with error if matched with standard output
	TimeoutMs int               `description:"time after command was issued for waiting for command output if expect fragment were not matched"`              //time after command was issued for waiting for command output if expect fragment were not matched.
	Directory string            `description:"directory where this command should start - if does not exists there is no exception"`                          //directory where command should run
	Env       map[string]string `description:"environment variables to be set before command runs"`                                                           //environment variables to be set before command runs
	SuperUser bool              ``                                                                                                                            ///flag to run it as super user
	/* 156-byte string literal not displayed */
	Secrets    secret.Secrets `description:"secrets map see https://github.com/viant/toolbox/tree/master/secret"`
	CheckError bool           `description:"check after command execution if status is <> 0, then throws error"`
	AutoSudo   bool           `description:"when this flag is set, in case of permission denied error for non root user retry command with sudo"`
}

Options represents an execution options

func DefaultOptions 

func DefaultOptions() *Options

DefaultOptions creates a default execution options

func NewOptions 

func NewOptions(secrets, env map[string]string, terminators, path []string, superUser bool) *Options

type RunRequest 

type RunRequest struct {
	Target *url.Resource `required:"true" description:"host where command runs" ` //execution target - destination where to run a command.
	*Options
	Commands []Command      `required:"true" description:"command list" `      //list of commands to run
	Extract  model.Extracts `description:"stdout data extraction instruction"` //Stdout data extraction instruction
}

RunRequest represents a simple command

func NewRunRequest 

func NewRunRequest(target *url.Resource, superUser bool, commands ...string) *RunRequest

NewRunRequest creates a new request

func NewRunRequestFromURL 

func NewRunRequestFromURL(URL string) (*RunRequest, error)

NewExtractRequestFromURL creates a new request from URL

func (*RunRequest) AsExtractRequest 

func (r *RunRequest) AsExtractRequest() *ExtractRequest

AsExtractRequest returns ExtractRequest for this requests

func (*RunRequest) Init 

func (r *RunRequest) Init() error

Init initialises request

func (*RunRequest) Validate 

func (r *RunRequest) Validate() error

Validate validates managed command request

type RunResponse 

type RunResponse struct {
	Session string
	Cmd     []*Log
	Output  string
	Data    data.Map
	Error   string
}

RunResponse represents a command response with logged commands.

func NewRunResponse 

func NewRunResponse(session string) *RunResponse

NewRunResponse creates a new RunResponse

func (*RunResponse) Add 

func (i *RunResponse) Add(log *Log)

Add appends provided log into commands slice.

func (*RunResponse) Stdout 

func (i *RunResponse) Stdout(indexes ...int) string

Stdout returns stdout for provided index, or all concatenated otherwise

type SetTargetRequest added in v0.41.0 

type SetTargetRequest struct {
	*url.Resource
}

SetTargetRequest represents set default target request

type SetTargetResponse added in v0.41.0 

type SetTargetResponse struct{}

SetTargetRequest represents set default target response

type StdinEvent 

type StdinEvent struct {
	SessionID string
	Stdin     string
}

StdinEvent represents an execution event start

func NewSdtinEvent 

func NewSdtinEvent(sessionID string, stdin string) *StdinEvent

NewSdtinEvent crates a new execution start event value

func (*StdinEvent) Messages 

func (e *StdinEvent) Messages() []*msg.Message

Messages returns messages

type StdoutEvent 

type StdoutEvent struct {
	SessionID string
	Stdout    string
	Error     string
}

StdoutEvent represents an execution event end

func NewStdoutEvent 

func NewStdoutEvent(sessionID string, stdout string, err error) *StdoutEvent

NewStdoutEvent crates a new execution start event value

func (*StdoutEvent) Messages 

func (e *StdoutEvent) Messages() []*msg.Message

Messages returns messages

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.