cm-cicada

README

Management of cloud migration workflow

This is a sub-system on Cloud-Barista platform provides a features of create and management workflow for cloud migration.

Overview

• Create and management workflow through Airflow.
• Create workflow based on dag-factory.

Execution and development environment

• Tested operating systems (OSs):
  • Ubuntu 23.10, Ubuntu 22.04, Ubuntu 18.04
• Language:
  • Go: 1.21.5

Prerequisites

1. Run Airflow

• Create the directory to run Airflow

  mkdir airflow
  cd airflow
  

• Download Airflow Docker Compose file

  curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.7.2/docker-compose.yaml'
  

• Setting Airflow user's UID to be the same as the host's UID. If not set, Airflow will run with root permission.

  mkdir -p ./dags ./logs ./plugins ./config
  echo -e "AIRFLOW_UID=$(id -u)" > .env
  

• Initialize Airflow database and user.

  docker compose up airflow-init
  

• Run all of Airflow services.

  docker compose up -d
  

2. Install dag-factory

cm-cicada uses dag-factory to make DAG as YAML file.

dag-factory must be installed to Airflow containers.

Install dag-factory to these three containers.

• airflow-triggerer
• airflow-worker
• airflow-scheduler

docker exec -it airflow-airflow-triggerer-1 /bin/bash
pip install dag-factory
exit

docker exec -it airflow-airflow-worker-1 /bin/bash
pip install dag-factory
exit

docker exec -it airflow-airflow-scheduler-1 /bin/bash
pip install dag-factory
exit

How to run

1. Build the binary

   • Run on Linux.

     make
     

   • Run on Linux for build Windows binary or run on Windows where make command is available.

     make windows
     

2. Write the configuration file.

   • Configuration file name is 'cm-cicada.yaml'
   • The configuration file must be placed in one of the following directories.
     • .cm-cicada/conf directory under user's home directory
     • 'conf' directory where running the binary
     • 'conf' directory where placed in the path of 'CMCICADA_ROOT' environment variable
   • Configuration options
     • airflow-server
       • address : Specify Airflow server's address ({IP or Domain}:{Port})
       • use_tls : Must be true if Airflow server uses HTTPS.
       • skip_tls_verify : Skip TLS/SSL certificate verification. Must be set if 'use_tls' is true.
       • timeout : HTTP timeout value as seconds.
       • username : Airflow login username.
       • password : Airflow login password.
     • dag_directory_host : Specify DAG directory of the host. (Mounted DAG directory used by Airflow containers)
     • dag_directory_airflow : Specify DAG directory of Airflow container. (DAG directory inside the container.)
   • listen
     • port : Listen port of the API.
   • Configuration file example

     cm-cicada:
          airflow-server:
               address: 127.0.0.1:8080
               use_tls: false
               # skip_tls_verify: true
               timeout: 10
               username: "airflow"
               password: "airflow"
          dag_directory_host: "/home/ish/test/airflow/dags"
          dag_directory_airflow: "/opt/airflow/dags" # Use dag_directory_host for dag_directory_airflow, if this value is empty
          listen:
               port: 8083
     

3. Run with privileges

   sudo ./cm-grasshopper