atmos

command module

v1.11.2 Latest Latest Go to latest Published: Nov 4, 2022 License: Apache-2.0 Imports: 2 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/cloudposse/atmos

Links

Open Source Insights

README ¶

atmos

atmos - Universal Tool for DevOps and Cloud Automation.

This project is part of our comprehensive "SweetOps" approach towards DevOps.

It's 100% Open Source and licensed under the APACHE2.

Introduction

atmos is both a library and a command-line tool for provisioning, managing and orchestrating workflows across various toolchains. We use it extensively for automating cloud infrastructure and Kubernetes clusters.

atmos includes workflows for dealing with:

Provisioning Terraform components
Deploying helm charts to Kubernetes clusters using helmfile
Executing helm commands on Kubernetes clusters
Provisioning istio on Kubernetes clusters using istio operator and helmfile
Executing kubectl commands on Kubernetes clusters
Executing AWS SDK commands to orchestrate cloud infrastructure
Running AWS CDK constructs to define cloud resources
Executing commands for the serverless framework
Executing shell commands
Combining various commands into workflows to execute many commands sequentially in just one step
... and many more

In essence, it's a tool that orchestrates the other CLI tools in a consistent and self-explaining manner. It's a superset of all other tools and task runners (e.g. make, terragrunt, terraform, aws cli, gcloud, etc.) and is intended to be used to tie everything together, so you can provide a simple CLI interface for your organization.

Moreover, atmos is not only a command-line interface for managing clouds and clusters. It provides many useful patterns and best practices, such as:

Enforces Terraform and helmfile project structure (so everybody knows where things are)
Provides separation of configuration and code (so the same code could be easily deployed to different regions, environments and stages)
It can be extended to include new features, commands, and workflows
The commands have a clean, consistent and easy to understand syntax
The CLI code is modular and self-documenting

Install

OSX

From Homebrew directly:

brew install atmos

Linux

On Debian:

# Add the Cloud Posse package repository hosted by Cloudsmith
apt-get update && apt-get install -y apt-utils curl
curl -1sLf 'https://dl.cloudsmith.io/public/cloudposse/packages/cfg/setup/bash.deb.sh' | bash

# Install atmos
apt-get install atmos@="<ATMOS_VERSION>-*"

On Alpine:

# Install the Cloud Posse package repository hosted by Cloudsmith
curl -sSL https://apk.cloudposse.com/install.sh | bash

# Install atmos
apk add atmos@cloudposse~=<ATMOS_VERSION>

Go

Install the latest version

go install github.com/cloudposse/atmos

Get a specific version

NOTE: Since the version is passed in via -ldflags during build, when running go install without using -ldflags, the CLI will return 0.0.1 when running atmos version.

go install github.com/cloudposse/atmos@v1.3.9

Build from source

make build

or run this and replace $version with the version that should be returned with atmos version.

go build -o build/atmos -v -ldflags "-X 'github.com/cloudposse/atmos/cmd.Version=$version'"

Usage

There are a number of ways you can leverage this project:

Recommended Layout

Our recommended filesystem layout looks like this:

   │   # CLI configuration
   └── cli/
   │  
   │   # Centralized components configuration
   ├── stacks/
   │   │
   │   └── $stack.yaml
   │  
   │   # Components are broken down by tool
   ├── components/
   │   │
   │   ├── terraform/   # root modules in here
   │   │   ├── vpc/
   │   │   ├── eks/
   │   │   ├── rds/
   │   │   ├── iam/
   │   │   ├── dns/
   │   │   └── sso/
   │   │
   │   └── helmfile/  # helmfiles are organized by chart
   │       ├── cert-manager/helmfile.yaml
   │       └── external-dns/helmfile.yaml
   │  
   │   # Makefile for building the CLI
   ├── Makefile
   │  
   │   # Docker image for shipping the CLI and all dependencies
   └── Dockerfile (optional)

Example

The example folder contains a complete solution that shows how to:

Structure the Terraform and helmfile components
Configure the CLI
Add stack configurations for the Terraform and helmfile components (to provision them to different environments and stages)

CLI Configuration

Centralized Project Configuration

atmos provides separation of configuration and code, allowing you to provision the same code into different regions, environments and stages.

In our example, all the code (Terraform and helmfiles) is in the components folder.

The centralized stack configurations (variables for the Terraform and helmfile components) are in the stacks folder.

In the example, all stack configuration files are broken down by environments and stages and use the predefined format $environment-$stage.yaml.

Environments are abbreviations of AWS regions, e.g. ue2 stands for us-east-2, whereas uw2 would stand for us-west-2.

$environment-globals.yaml is where you define the global values for all stages for a particular environment. The global values get merged with the $environment-$stage.yaml configuration for a specific environment/stage by the CLI.

In the example, we defined a few config files:

ue2-dev.yaml - stack configuration (Terraform and helmfile variables) for the environment ue2 and stage dev
ue2-staging.yaml - stack configuration (Terraform and helmfile variables) for the environment ue2 and stage staging
ue2-prod.yaml - stack configuration (Terraform and helmfile variables) for the environment ue2 and stage prod
ue2-globals.yaml - global settings for the environment ue2 (e.g. region, environment)
globals.yaml - global settings for the entire solution

NOTE: The stack configuration structure and the file names described above are just an example of how to name and structure the config files. You can choose any file name for a stack. You can also include other configuration files (e.g. globals for the environment, and globals for the entire solution) into a stack config using the import directive.

Stack configuration files have a predefined format:

import:
  - ue2-globals

vars:
  stage: dev

terraform:
  vars: {}

helmfile:
  vars: {}

components:
  terraform:
    vpc:
      command: "/usr/bin/terraform-0.15"
      backend:
        s3:
          workspace_key_prefix: "vpc"
      vars:
        cidr_block: "10.102.0.0/18"
    eks:
      backend:
        s3:
          workspace_key_prefix: "eks"
      vars: {}

  helmfile:
    nginx-ingress:
      vars:
        installed: true

It has the following main sections:

import - (optional) a list of stack configuration files to import and combine with the current configuration file
vars - (optional) a map of variables for all components (Terraform and helmfile) in the stack
terraform - (optional) configuration (variables) for all Terraform components
helmfile - (optional) configuration (variables) for all helmfile components
components - (required) configuration for the Terraform and helmfile components

The components section consists of the following:

terraform - defines variables, the binary to execute, and the backend for each Terraform component. Terraform component names correspond to the Terraform components in the components folder
helmfile - defines variables and the binary to execute for each helmfile component. Helmfile component names correspond to the helmfile components in the helmfile folder

Run the Example

To run the example, execute the following commands in a terminal:

cd example
make all - it will build the Docker image, build the CLI tool inside the image, and then start the container

Provision Terraform Component

To provision a Terraform component using the atmos CLI, run the following commands in the container shell:

atmos terraform plan eks --stack=ue2-dev
atmos terraform apply eks --stack=ue2-dev

where:

eks is the Terraform component to provision (from the components/terraform folder)
--stack=ue2-dev is the stack to provision the component into

Short versions of the command-line arguments can be used:

atmos terraform plan eks -s ue2-dev
atmos terraform apply eks -s ue2-dev

To execute plan and apply in one step, use terraform deploy command:

atmos terraform deploy eks -s ue2-dev

Provision Helmfile Component

To provision a helmfile component using the atmos CLI, run the following commands in the container shell:

atmos helmfile diff nginx-ingress --stack=ue2-dev
atmos helmfile apply nginx-ingress --stack=ue2-dev

where:

nginx-ingress is the helmfile component to provision (from the components/helmfile folder)
--stack=ue2-dev is the stack to provision the component into

Short versions of the command-line arguments can be used:

atmos helmfile diff nginx-ingress -s ue2-dev
atmos helmfile apply nginx-ingress -s ue2-dev

To execute diff and apply in one step, use helmfile deploy command:

atmos helmfile deploy nginx-ingress -s ue2-dev

Workflows

Workflows are a way of combining multiple commands into one executable unit of work.

In the CLI, workflows can be defined using two different methods:

In the configuration file for a stack (see workflows in ue2-dev.yaml for an example)
In a separate file (see workflow1.yaml

In the first case, we define workflows in the configuration file for the stack (which we specify on the command line). To execute the workflows from workflows in ue2-dev.yaml, run the following commands:

atmos workflow terraform-plan-all-tenant1-ue2-dev -s tenant1-ue2-dev

Note that workflows defined in the stack config files can be executed only for the particular stack (environment and stage). It's not possible to provision resources for multiple stacks this way.

In the second case (defining workflows in a separate file), a single workflow can be created to provision resources into different stacks. The stacks for the workflow steps can be specified in the workflow config.

For example, to run terraform plan and helmfile diff on all terraform and helmfile components in the example, execute the following command:

atmos workflow plan-all -f workflows

where the command-line option -f (--file for long version) instructs the atmos CLI to look for the plan-all workflow in the file workflows.

As we can see, in multi-environment workflows, each workflow job specifies the stack it's operating on:

workflows:
terraform-plan-all-test-components:
  description: |
    Run 'terraform plan' on 'test/test-component' and all its derived components.
    The stack must be provided on the command line: atmos workflow terraform-plan-all-test-components -f workflow1 -s <stack>
  steps:
    - command: terraform plan test/test-component
    - command: terraform plan test/test-component-override
    - command: terraform plan test/test-component-override-2
    - command: terraform plan test/test-component-override-3

You can also define a workflow in a separate file without specifying the stack in the workflow's job config. In this case, the stack needs to be provided on the command line.

For example, to run the terraform-plan-all-tenant1-ue2-dev workflow from the workflows file for the tenant1-ue2-dev stack, execute the following command:

atmos workflow terraform-plan-all-tenant1-ue2-dev -f workflow1 -s tenant1-ue2-dev

Validation

atmos component validation allows:

Validate component config (vars, settings, backend, and other sections) using JSON Schema
Check if the component config (including relations between different component variables) is correct to allow or deny component provisioning using OPA/Rego policies

atmos validate component command supports --schema-path and --schema-type command line arguments. If the arguments are not provided, atmos will try to find and use the settings.validation section defined in the component's YAML config.

atmos validate component infra/vpc -s tenant1-ue2-prod --schema-path validate-infra-vpc-component.json --schema-type jsonschema

atmos validate component infra/vpc -s tenant1-ue2-prod --schema-path validate-infra-vpc-component.rego --schema-type opa

atmos validate component infra/vpc -s tenant1-ue2-prod

atmos validate component infra/vpc -s tenant1-ue2-dev

Configure component validation

In atmos.yaml, add the schemas config:

# Validation schemas (for validating atmos stacks and components)
schemas:
  # https://json-schema.org
  jsonschema:
    # Can also be set using `ATMOS_SCHEMAS_JSONSCHEMA_BASE_PATH` ENV var, or `--schemas-jsonschema-dir` command-line arguments
    # Supports both absolute and relative paths
    base_path: "stacks/schemas/jsonschema"
  # https://www.openpolicyagent.org
  opa:
    # Can also be set using `ATMOS_SCHEMAS_OPA_BASE_PATH` ENV var, or `--schemas-opa-dir` command-line arguments
    # Supports both absolute and relative paths
    base_path: "stacks/schemas/opa"

In the component YAML config, add the settings.validation section:

components:
  terraform:
    infra/vpc:
      settings:
        # Validation
        # Supports JSON Schema and OPA policies
        # All validation steps must succeed to allow the component to be provisioned
        validation:
          validate-infra-vpc-component-with-jsonschema:
            schema_type: jsonschema
            # 'schema_path' can be an absolute path or a path relative to 'schemas.jsonschema.base_path' defined in `atmos.yaml`
            schema_path: validate-infra-vpc-component.json
            description: Validate 'infra/vpc' component variables using JSON Schema
          check-infra-vpc-component-config-with-opa-policy:
            schema_type: opa
            # 'schema_path' can be an absolute path or a path relative to 'schemas.opa.base_path' defined in `atmos.yaml`
            schema_path: validate-infra-vpc-component.rego
            description: Check 'infra/vpc' component configuration using OPA policy

Add the following JSON Schema in the file stacks/schemas/jsonschema/validate-infra-vpc-component.json:

{
  "$id": "infra-vpc-component",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "infra/vpc component validation",
  "description": "JSON Schema for infra/vpc atmos component.",
  "type": "object",
  "properties": {
    "vars": {
      "type": "object",
      "properties": {
        "region": {
          "type": "string"
        },
        "cidr_block": {
          "type": "string",
          "pattern": "^([0-9]{1,3}\\.){3}[0-9]{1,3}(/([0-9]|[1-2][0-9]|3[0-2]))?$"
        },
        "map_public_ip_on_launch": {
          "type": "boolean"
        }
      },
      "additionalProperties": true,
      "required": [
        "region",
        "cidr_block",
        "map_public_ip_on_launch"
      ]
    }
  }
}

Add the following OPA policy in the file stacks/schemas/opa/validate-infra-vpc-component.rego:

# 'atmos' looks for the 'errors' (array of strings) output from all OPA policies
# If the 'errors' output contains one or more error messages, 'atmos' considers the policy failed

# 'package atmos' is required in all `atmos` OPA policies
package atmos

# In production, don't allow mapping public IPs on launch
errors[message] {
  input.vars.stage == "prod"
  input.vars.map_public_ip_on_launch == true
  message = "Mapping public IPs on launch is not allowed in 'prod'. Set 'map_public_ip_on_launch' variable to 'false'"
}

# In 'dev', only 2 Availability Zones are allowed
  errors[message] {
  input.vars.stage == "dev"
  count(input.vars.availability_zones) != 2
  message = "In 'dev', only 2 Availability Zones are allowed"
}

Run the following commands to validate the component in the stacks:

> atmos validate component infra/vpc -s tenant1-ue2-prod

Check 'infra/vpc' component configuration using OPA policy
Mapping public IPs on launch is not allowed in 'prod'. Set 'map_public_ip_on_launch' variable to 'false'

exit status 1

> atmos validate component infra/vpc -s tenant1-ue2-dev

Check 'infra/vpc' component configuration using OPA policy
In 'dev', only 2 Availability Zones are allowed

exit status 1

> atmos validate component infra/vpc -s tenant1-ue2-staging

Validate 'infra/vpc' component variables using JSON Schema
{
  "valid": false,
  "errors": [
    {
      "keywordLocation": "",
      "absoluteKeywordLocation": "file:///examples/complete/stacks/schemas/jsonschema/infra-vpc-component#",
      "instanceLocation": "",
      "error": "doesn't validate with file:///examples/complete/stacks/schemas/jsonschema/infra-vpc-component#"
    },
    {
      "keywordLocation": "/properties/vars/properties/cidr_block/pattern",
      "absoluteKeywordLocation": "file:///examples/complete/stacks/schemas/jsonschema/infra-vpc-component#/properties/vars/properties/cidr_block/pattern",
      "instanceLocation": "/vars/cidr_block",
      "error": "does not match pattern '^([0-9]{1,3}\\\\.){3}[0-9]{1,3}(/([0-9]|[1-2][0-9]|3[0-2]))?$'"
    }
  ]
}

exit status 1

Run the following commands to provision the component in the stacks:

atmos terraform apply infra/vpc -s tenant1-ue2-prod
atmos terraform apply infra/vpc -s tenant1-ue2-dev

Since the OPA validation policies don't pass, atmos does not allow provisioning the component in the stacks:

atmos-validate-infra-vpc-in-tenant1-ue2-prod

atmos-validate-infra-vpc-in-tenant1-ue2-dev

Help

Got a question? We got answers.

File a GitHub issue, send us an email or join our Slack Community.

DevOps Accelerator for Startups

We are a DevOps Accelerator. We'll help you build your cloud infrastructure from the ground up so you can own it. Then we'll show you how to operate it and stick around for as long as you need us.

Work directly with our team of DevOps experts via email, slack, and video conferencing.

We deliver 10x the value for a fraction of the cost of a full-time engineer. Our track record is not even funny. If you want things done right and you need it done FAST, then we're your best bet.

Reference Architecture. You'll get everything you need from the ground up built using 100% infrastructure as code.
Release Engineering. You'll have end-to-end CI/CD with unlimited staging environments.
Site Reliability Engineering. You'll have total visibility into your apps and microservices.
Security Baseline. You'll have built-in governance with accountability and audit logs for all changes.
GitOps. You'll be able to operate your infrastructure via Pull Requests.
Training. You'll receive hands-on training so your team can operate what we build.
Questions. You'll have a direct line of communication between our teams via a Shared Slack channel.
Troubleshooting. You'll get help to triage when things aren't working.
Code Reviews. You'll receive constructive feedback on Pull Requests.
Bug Fixes. We'll rapidly work with you to fix any bugs in our projects.

Slack Community

Join our Open Source Community on Slack. It's FREE for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totally sweet infrastructure.

Discourse Forums

Participate in our Discourse Forums. Here you'll find answers to commonly asked questions. Most questions will be related to the enormous number of projects we support on our GitHub. Come here to collaborate on answers, find solutions, and get ideas about the products and services we value. It only takes a minute to get started! Just sign in with SSO using your GitHub account.

Sign up for our newsletter that covers everything on our technology radar. Receive updates on what we're up to on GitHub as well as awesome new projects we discover.

Office Hours

Join us every Wednesday via Zoom for our weekly "Lunch & Learn" sessions. It's FREE for everyone!

Contributing

Bug Reports & Feature Requests

Please use the issue tracker to report any bugs or file feature requests.

Developing

If you are interested in being a contributor and want to get involved in developing this project or help out with our other projects, we would love to hear from you! Shoot us an email.

In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.

Fork the repo on GitHub
Clone the project to your own machine
Commit changes to your own branch
Push your work back up to your fork
Submit a Pull Request so that we can review your changes

NOTE: Be sure to merge the latest changes from "upstream" before making a pull request!

Copyright

License

See LICENSE for full details.

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.