deployer

README

Step Deployer

The Step Deployer is an AWS Step Function that can deploy step functions, so it can recursively deploy itself.

To create the necessary AWS resources you can use GeoEngineer which requires ruby and terraform:

bundle install
./scripts/geo apply resources/step_deployer.rb

We prefer to use AWS credentials exported via assume-role but any AWS access keys will do:

# Use AWS Creds or assume-role
./scripts/bootstrap_deployer

To update the deployer you can use:

git pull # pull down new code
./scripts/deploy_deployer # recursive deployer

To use the deployer:

step deploy -lambda <lambda name> \
            -step <step-fn-name>  \
            -states <state machine json>

This will default the AWS region and account to those in the environment variables, the project and config names to tags on the lambda, the lambda file to ./lambda.zip.

Implementation

The tasks of the deployer are:

1. Validate: Validate the sent release bundle
2. Lock: grab a lock in S3 so others cannot deploy at the same time
3. ValiadteResources: Validate the referenced resources exist and have the correct tags and paths
4. Deploy: Update the State Machine and Lambda, then release the Lock
5. ReleaseLockFailure: If something goes wrong, try release the lock and fail

The end states are:

1. Success: deployed correctly
2. FailureClean: something went wrong but it has recovered the previous good state
3. FailureDirty: something went wrong and it is not in a good state. The existing step function, Lambda and/or lock require manual cleanup

The limitations are:

1. State machine size must be less than 30Kb as it is sent as part of the step-function input.
2. Lambda size must be less than the RAM available to the step-deployer lambda as it validates the lambda SHA256 in memory

Security

Deployers are critical pieces of infrastructure as they may be used to compromise software they deploy. As such, we take security very seriously around the step-deployer and answer the following questions:

1. Authentication: Who can deploy?
2. Authorization: What can be deployed?
3. Replay and Man-in-the-middle (MITM): Can some unauthorized person edit or reuse a release to change what is deployed?
4. Audit: Who has done what, and when?

Authentication

The central authentication mechanisms are the AWS IAM permissions for step functions, lambda, and S3.

By limiting the lambda:UpdateFunctionCode, lambda:UpdateFunctionConfiguration, lambda:Invoke* and states:UpdateStateMachine permissions the step-deployer function becomes the only way to deploy. Once this is the case, limiting permissions to states:StartExecution of the step-deployer directly limits who can deploy.

Ensuring the step-deployer Lambdas role can only access a single single S3 bucket with:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject*", "s3:PutObject*",
    "s3:List*", "s3:DeleteObject*"
  ],
  "Resource": [
    "arn:aws:s3:::#{s3_bucket_name}/*",
    "arn:aws:s3:::#{s3_bucket_name}"
  ]
},
{
  "Effect": "Deny",
  "Action": ["s3:*"],
  "NotResource": [
    "arn:aws:s3:::#{s3_bucket_name}/*",
    "arn:aws:s3:::#{s3_bucket_name}"
  ]
},

Further restricts who can deploy to those that also can s3:PutObject to the bucket.

Who can execute the step function, and who can upload to S3 are the two permissions that guard deploys. Additionally, if you separate those two permissions, you gain extra security, e.g. by only allowing your CI/CD pipe to upload releases, and developers to execute the step function you can ensure only valid builds are ever deployed.

Authorization

We use tags and paths to restrict the resources that the step-deployer can deploy to.

The lambda function must have a ProjectName and ConfigName tag that match the release, and a DeployWith tag equal to "step-deployer".

Step functions don't support tags, so the path on their role must be must be equal to /step/<ProjectName>/<ConfigName>/.

Assets uploaded to S3 are in the path /<ProjectName>/<ConfigName> so limiting who can s3:PutObject to a path can be used to limit what project-configs they can deploy.

Replay and MITM

Each release the client generates a release release_id, a created_at date, and a SHA256 of the lambda file, and together also uploads the release to S3.

The step-deployer will reject any request where the created_at date is not recent, the lambdas SHA does not match the uploaded zip, or the release sent to the step function and S3 don't match. This means that if a user can invoke the step function, but not upload to S3 (or vice-versa) it is not possible to deploy old or malicious code.

Audit

Working out what happened when is very useful for debugging and security response. Step functions make it easy to see the history of all executions in the AWS console and via API. S3 can log all access to cloud-trail, so collecting from these two sources will show all information about a deploy.

Continuing Deployment

Some TODOs for the deployer are:

1. Automated rollback on a bad deploy
2. Assume-role sts into other accounts to deploy there, so only one step-deployer is needed for many accounts.
3. Health checking the Lambdas and Step Functions, if they have a health check.

Documentation

Overview

The deployer package contains the Step Deployer service that is a Step Function that Deploys Step Functions.

It also contains a client for messaging and bootstrapping the Step Deployer.

Index

• func CreateTaskFunctions(awsc aws.AwsClients) *handler.TaskHandlers
• func DeployHandler(awsc aws.AwsClients) interface{}
• func LockHandler(awsc aws.AwsClients) interface{}
• func ReleaseLockFailureHandler(awsc aws.AwsClients) interface{}
• func StateMachine() (*machine.StateMachine, error)
• func TaskHandlers() *handler.TaskHandlers
• func ValidateHandler(awsc aws.AwsClients) interface{}
• func ValidateResourcesHandler(awsc aws.AwsClients) interface{}
• type DeployLambdaError
  • func (e DeployLambdaError) Error() string
• type DeploySFNError
  • func (e DeploySFNError) Error() string
• type Release
  • func (release *Release) DeployLambda(lambdaClient aws.LambdaAPI, s3c aws.S3API) error
  • func (release *Release) DeployLambdaCode(lambdaClient aws.LambdaAPI, zip *[]byte) error
  • func (release *Release) DeployStepFunction(sfnClient aws.SFNAPI) error
  • func (release *Release) LambdaArn() *string
  • func (r *Release) LambdaProjectConfigDeployerTags(lambdac aws.LambdaAPI) (*string, *string, *string, error)
  • func (release *Release) LambdaZipPath() *string
  • func (release *Release) StepArn() *string
  • func (release *Release) UnmarshalJSON(data []byte) error
  • func (r *Release) Validate(s3c aws.S3API) error
  • func (r *Release) ValidateLambdaFunctionTags(lambdac aws.LambdaAPI) error
  • func (r *Release) ValidateLambdaSHA(s3c aws.S3API) error
  • func (r *Release) ValidateResources(lambdac aws.LambdaAPI, sfnc aws.SFNAPI) error
  • func (r *Release) ValidateStepFunctionPath(sfnc aws.SFNAPI) error
• type XRelease

Functions

func CreateTaskFunctions

func CreateTaskFunctions(awsc aws.AwsClients) *handler.TaskHandlers

CreateTaskFunctions returns

func DeployHandler

func DeployHandler(awsc aws.AwsClients) interface{}

func LockHandler

func LockHandler(awsc aws.AwsClients) interface{}

func ReleaseLockFailureHandler

func ReleaseLockFailureHandler(awsc aws.AwsClients) interface{}

func StateMachine

func StateMachine() (*machine.StateMachine, error)

StateMachine returns the StateMachine for the deployer

func TaskHandlers

func TaskHandlers() *handler.TaskHandlers

TaskHandlers returns

func ValidateHandler

func ValidateHandler(awsc aws.AwsClients) interface{}

func ValidateResourcesHandler

func ValidateResourcesHandler(awsc aws.AwsClients) interface{}

Types

type DeployLambdaError

type DeployLambdaError struct {
	// contains filtered or unexported fields
}

func (DeployLambdaError) Error

func (e DeployLambdaError) Error() string

type DeploySFNError

type DeploySFNError struct {
	// contains filtered or unexported fields
}

func (DeploySFNError) Error

func (e DeploySFNError) Error() string

type Release

type Release struct {
	bifrost.Release

	// Deploy Releases
	LambdaName   *string `json:"lambda_name,omitempty"`   // Lambda Name
	LambdaSHA256 *string `json:"lambda_sha256,omitempty"` // Lambda SHA256 Zip file
	StepFnName   *string `json:"step_fn_name,omitempty"`  // Step Function Name

	StateMachineJSON *string `json:"state_machine_json,omitempty"`
}

Release is the Data Structure passed between Client and Deployer

func (*Release) DeployLambda

func (release *Release) DeployLambda(lambdaClient aws.LambdaAPI, s3c aws.S3API) error

DeployLambda uploads new Code to the Lambda

func (*Release) DeployLambdaCode

func (release *Release) DeployLambdaCode(lambdaClient aws.LambdaAPI, zip *[]byte) error

DeployLambdaCode

func (*Release) DeployStepFunction

func (release *Release) DeployStepFunction(sfnClient aws.SFNAPI) error

DeployStepFunction updates the step function State Machine

func (*Release) LambdaArn

func (release *Release) LambdaArn() *string

func (*Release) LambdaProjectConfigDeployerTags

func (r *Release) LambdaProjectConfigDeployerTags(lambdac aws.LambdaAPI) (*string, *string, *string, error)

func (*Release) LambdaZipPath

func (release *Release) LambdaZipPath() *string

func (*Release) StepArn

func (release *Release) StepArn() *string

func (*Release) UnmarshalJSON

func (release *Release) UnmarshalJSON(data []byte) error

UnmarshalJSON should error if there is something unexpected

func (*Release) Validate

func (r *Release) Validate(s3c aws.S3API) error

func (*Release) ValidateLambdaFunctionTags

func (r *Release) ValidateLambdaFunctionTags(lambdac aws.LambdaAPI) error

func (*Release) ValidateLambdaSHA

func (r *Release) ValidateLambdaSHA(s3c aws.S3API) error

func (*Release) ValidateResources

func (r *Release) ValidateResources(lambdac aws.LambdaAPI, sfnc aws.SFNAPI) error

func (*Release) ValidateStepFunctionPath

func (r *Release) ValidateStepFunctionPath(sfnc aws.SFNAPI) error

type XRelease

type XRelease struct {
	Task *string // Do not include the Task because that can be implemented
	// contains filtered or unexported fields
}

But the problem is that there are exceptions that we have