cc

README

Cloud Control API Deployments

The rain cc deploy command provisions resources using the AWS Cloud Control API (CCAPI). It does not submit templates to CloudFormation, so there will be no managed stack to interact with after running this command. API calls are made directly from the client to CCAPI, and the state for the resources is stored by Rain in the same S3 bucket that is used for assets.

This is a highly experimental feature that exists mainly as a prototype for what a client-side provisioning engine might look like. Do not use this for production workloads. (Seriously)

Only resources that have been fully migrated to the CloudFormation registry can be provisioned with this command. It is important to note that resource provisioning is still done by the same back-end resource providers that CloudFormation uses. Those are available on GitHub under the aws-cloudformation organization, for example RDS. The cc deploy command makes client-side calls to CCAPI endpoints like CreateResource, but then CCAPI itself is the one invoking resource providers, which make SDK calls into specific services.

If you want to learn a bit more about the CloudFormation registry and the history of Cloud Control API, check out this blog post: The history and future roadmap of the AWS CloudFormation Registry

If you want to see if a CloudFormation resource is on the new registry model or not, check if the provisioning type is either Fully Mutable or Immutable by invoking the DescribeType API and inspecting the ProvisioningType response element.

Here is a CLI command that gets a description for the AWS::Lambda::Function resource, which is on the new registry model.

sh $ aws cloudformation describe-type --type RESOURCE \ --type-name AWS::Lambda::Function | grep ProvisioningType

   "ProvisioningType": "FULLY_MUTABLE", 

The difference between FULLY_MUTABLE and IMMUTABLE is the presence of the Update handler. FULLY_MUTABLE types include an update handler to process updates to the type during stack update operations. IMMUTABLE types do not include an update handler, so the type can’t be updated and must instead be replaced during stack update operations. Legacy resource types will be NON_PROVISIONABLE.

Why would I want to use this?

Again, for production workloads, you shouldn't. But the one big benefit is that you have access to the resource state, which is described in detail below. You could, in theory, modify the state to deal with unexpected deployment failures, or to remediate complex drift situations. Template deployment might be slightly faster, since you won't wait for the CloudFormation backend to push your stack through the workflow, but since CloudFormation uses the same resource providers, the difference will not be huge.

Another good reason is that you are curious about how CCAPI works, and you are interested in learning about all the really hard things a template provisioning engine has to do. Let us know if you want to dive in and contribute. The best way to learn is by doing!

State management

State cannot be managed based on the template alone, due to the fact that primary identifiers are not always required (and often disouraged). For example, the following template deploys an S3 bucket:

Resources: 
    MyBucket: 
        Type: AWS::S3::Bucket 

The physical name of the bucket was not specified, and since two different templates could both have an S3 bucket with the logical ID "MyBucket", there is no way to tell from looking at the template alone what bucket it corresponds to in your account.

It is obviously very important to store the state in a way that it cannot be lost or associated with the wrong AWS environment. It is also important to make sure two processes don't try to deploy the same template at the same time.

The state file for rain cc deploy is stored as a YAML file that has the same format as the source CloudFormation template. Extra sections are added to the file to associate state with the deployed resources.

State: 
  LastWriteTime: ...
  ResourceModels:
    MyResource:
      Identifier:
      Model:
        ...

Each deployment has its own state file in the rain artifacts bucket in the region where you are deploying. If a user tries a deployment while there is a locked state file, the command gives an error message with instructions on how to remediate the issue. Often times, this will result from a deployment that failed halfway through.

rain-artifacts-0123456789012-us-east-1/ 
    deployments/ 
        name1.yaml
        name2.yaml

Drift detection can be run on the state file to inspect the actual resource properties and compare them to the stored state. When you deploy a change to a template with this command, drift from the stored state will be pointed out and you will be able to resolve it before continuing with the update.

Usage

To use this command, supply the same arguments that you would supply to the deploy command:

$ rain cc deploy -x my-template.yaml my-deployment-name

(The -x argument stands for --experimental. This is a nag to make sure you understand this feature is still in active development!)

To remove resources deployed with cc deploy, use the cc rm command:

$ rain cc rm -x my-deployment-name

To view the state file for a deployment:

rain cc state -x my-deployment-name

To remediate drift on a deployment (also runs when you deploy)

rain cc drift -x my-deployment name

Unsupported features

Since this is a prototype, some features are not yet supported:

• Not all instrinsic functions have been implemented
• Tags are ignored
• Any resource not yet migrated to the new registry model
• Retention policies
• Probably more stuff that is totally necessary for production use

Documentation

Index

• Constants
• Variables
• func PackageTemplate(fn string, yes bool) cft.Template
• func Resolve(resource *Resource) (*yaml.Node, error)
• type DeploymentResults
  • func DeployTemplate(template cft.Template) (*DeploymentResults, error)
  • func (results *DeploymentResults) Summarize()
• type Resource
  • func NewResource(name string, resourceType string, state ResourceState, node *yaml.Node) *Resource
  • func (r Resource) String() string
• type ResourceState
• type StateResult

Constants

const AWS_PREFIX = "AWS::"

const FILE_PATH string = "FilePath"

Variables

var CCDeployCmd = &cobra.Command{
	Use:   "deploy <template> <name>",
	Short: "Deploy a local template directly using the Cloud Control API (Experimental!)",
	Long: `Creates or updates resources directly using Cloud Control API from the template file <template>.
You must pass the --experimental (-x) flag to use this command, to acknowledge that it is experimental and likely to be unstable!
`,
	Args:                  cobra.ExactArgs(2),
	DisableFlagsInUseLine: true,
	Run:                   deploy,
}

var CCDriftCmd = &cobra.Command{
	Use:   "drift <name>",
	Short: "Compare the state file to the live state of the resources",
	Long: `When deploying templates with the cc command, a state file is created and stored in the rain assets bucket. This command outputs a diff of that file and the actual state of the resources, according to Cloud Control API. You can then apply the changes by changing the live state, or by modifying the state file.
`,
	Args:                  cobra.ExactArgs(1),
	DisableFlagsInUseLine: true,
	Run:                   runDrift,
}

var CCRmCmd = &cobra.Command{
	Use:                   "rm <name>",
	Short:                 "Delete a deployment created by cc deploy (Experimental!)",
	Long:                  "Deletes the resources in the cc deploy deployment named <name> and waits for all CloudControl API calls to complete. This is an experimental feature that requires the -x flag to run.",
	Args:                  cobra.ExactArgs(1),
	Aliases:               []string{"ccremove", "ccdel", "ccdelete"},
	DisableFlagsInUseLine: true,
	Run: func(cmd *cobra.Command, args []string) {
		name := args[0]

		if !Experimental {
			panic("Please add the --experimental arg to use this feature")
		}

		spinner.Push("Fetching deployment status")
		key := getStateFileKey(name)
		var state cft.Template

		bucketName := s3.RainBucket(yes)

		obj, err := s3.GetObject(bucketName, key)
		if err != nil {
			panic(err)
		}

		state, err = parse.String(string(obj))
		if err != nil {
			panic(fmt.Errorf("unable to parse state file: %v", err))
		}

		_, stateMap, _ := s11n.GetMapValue(state.Node.Content[0], "State")
		if stateMap == nil {
			panic(fmt.Errorf("did not find State in state file"))
		}

		lock := ""
		for i, s := range stateMap.Content {
			if s.Kind == yaml.ScalarNode && s.Value == "Lock" {
				lock = stateMap.Content[i+1].Value
			}
		}

		spinner.Pop()

		if lock != "" {
			msg := "Unable to remove deployment, found a locked state file"
			panic(fmt.Errorf("%v:\ns3://%v/%v (%v)", msg, bucketName, key, lock))
		}

		if !yes {
			if !console.Confirm(false, "Are you sure you want to delete this deployment?") {

				panic(fmt.Errorf("Deployment removal cancelled: '%s'", name))
			}
		}

		spinner.StartTimer(fmt.Sprintf("Removing deployment %v", name))

		template := cft.Template{Node: node.Clone(state.Node)}
		rootMap := template.Node.Content[0]

		_, stateResourceModels, _ := s11n.GetMapValue(stateMap, "ResourceModels")
		if stateResourceModels == nil {
			panic("Expected to find State.ResourceModels in the state template")
		}
		identifiers := make(map[string]string, 0)
		for i, v := range stateResourceModels.Content {
			if i%2 == 0 {
				_, identifier, _ := s11n.GetMapValue(stateResourceModels.Content[i+1], "Identifier")
				if identifier != nil {
					identifiers[v.Value] = identifier.Value
				}
			}
		}
		config.Debugf("identifiers: %v", identifiers)

		_, resourceMap, _ := s11n.GetMapValue(rootMap, "Resources")
		for i, resource := range resourceMap.Content {
			if i%2 == 0 {
				if identifier, ok := identifiers[resource.Value]; !ok {
					panic(fmt.Errorf("unable to find identifier for %v", resource.Value))
				} else {
					r := resourceMap.Content[i+1]
					s := node.AddMap(r, "State")
					node.Add(s, "Action", "Delete")
					node.Add(s, "Identifier", identifier)
				}
			}
		}

		config.Debugf("About to delete deployment: %v", format.CftToYaml(template))

		results, err := DeployTemplate(template)
		if err != nil {
			panic(err)
		}

		spinner.StopTimer()

		results.Summarize()
		fmt.Printf("Deployment %v successfully removed\n", name)

		spinner.Push("Deleting state file")
		err = s3.DeleteObject(bucketName, key)
		if err != nil {

			panic(fmt.Errorf("Unable to delete state file %v/%v: %v", bucketName, key, err))
		}
		spinner.Pop()
	},
}

Cmd is the rm command's entrypoint

var CCStateCmd = &cobra.Command{
	Use:   "state <name>",
	Short: "Download the state file for a template deployed with cc deploy",
	Long: `When deploying templates with the cc command, a state file is created and stored in the rain assets bucket. This command outputs the contents of that file.
`,
	Args:                  cobra.ExactArgs(1),
	DisableFlagsInUseLine: true,
	Run:                   runState,
}

var Cmd = &cobra.Command{
	Use:   "cc <command>",
	Short: "Interact with templates using Cloud Control API instead of CloudFormation",
	Long: `You must pass the --experimental (-x) flag to use this command, to acknowledge that it is experimental and likely to be unstable!
`,
}

var Experimental bool

Functions

func PackageTemplate

func PackageTemplate(fn string, yes bool) cft.Template

PackageTemplate reads the template and performs any necessary packaging on it before deployment. The rain bucket will be created if it does not already exist.

func Resolve

func Resolve(resource *Resource) (*yaml.Node, error)

resolve resolves CloudFormation intrinsic functions

It relies on dependent resources already having been deployed, so that we can query values with CCAPI.

Supported:

Ref
Fn::GetAtt
Fn::Sub

Not Supported:

Fn::Base64
Fn::Cidr
Condition functions
Fn::FindInMap
Fn::ForEach
Fn::GetAZs
Fn::ImportValue
Fn::Join
Fn::Length
Fn::Select
Fn::Split
Fn::ToJsonString
Fn::Transform

Types

type DeploymentResults

type DeploymentResults struct {
	Succeeded bool
	State     cft.Template
	Resources map[string]*Resource
}

DeploymentResults captures everything that happened as a result of deployment

func DeployTemplate

func DeployTemplate(template cft.Template) (*DeploymentResults, error)

deployTemplate deploys the CloudFormation template using the Cloud Control API. A failed deployment will result in DeploymentResults.Succeeded = false. A non-nil error is returned when something unexpected caused a failure not related to actually deploying resources, like an invalid template.

func (*DeploymentResults) Summarize

func (results *DeploymentResults) Summarize()

Summarize prints out a summary of deployment results

type Resource

type Resource struct {
	Name       string
	Type       string
	Node       *yaml.Node
	State      ResourceState
	Message    string
	Identifier string
	Model      string
	Action     diff.ActionType
	PriorJson  string
	Start      time.Time
	End        time.Time
}

func NewResource

func NewResource(name string,
	resourceType string, state ResourceState, node *yaml.Node) *Resource

NewResource creates a new Resource and adds it to the map

func (Resource) String

func (r Resource) String() string

type ResourceState

type ResourceState int

const (
	Waiting ResourceState = iota
	Deploying
	Failed
	Deployed
	Canceled
)

type StateResult

type StateResult struct {
	StateFile cft.Template
	Lock      string
	IsUpdate  bool
}