redshiftdatasetannotator

README

redshift-data-set-annotator

Annotator for QuickSight datasets with Redshift as the data source

The redshift-data-set-annotator is a tool to add dataset annotations based on column names, column comments, etc. for QuickSight datasets based on a single redshift Relation (table, view, etc.).

Install

Homebrew (macOS and Linux)

$ brew install mashiike/tap/redshift-data-set-annotator

Binary packages

Releases

QuickStart

1st configure for provisoned cluster

$ redshift-data-set-annotator configure                                                                                          
default profile is serverless?: (yes/no) [no]: no
Enter cluster identifier: warehouse
Enter db user: admin 

and execute annotate

$ redshift-data-set-annotator annotate --data-set-id <data-set-id>j

Usage

Usage: redshift-data-set-annotator <command>

Flags:
  -h, --help                     Show context-sensitive help.
      --aws-account-id=STRING    QuickSight aws account id
  -r, --region=STRING            AWS region ($AWS_REGION)
      --log-level="info"         output log level ($LOG_LEVEL)

Commands:
  configure
    Create a configuration file of redshift-data-set-annotator

  annotate --data-set-id=STRING
    Annotate a QuickSight dataset with Redshift as the data source

  version
    Show version

Run "redshift-data-set-annotator <command> --help" for more information on a command.

Usage: redshift-data-set-annotator annotate --data-set-id=STRING

Annotate a QuickSight dataset with Redshift as the data source

Flags:
  -h, --help                        Show context-sensitive help.
      --aws-account-id=STRING       QuickSight aws account id
  -r, --region=STRING               AWS region ($AWS_REGION)
      --log-level="info"            output log level ($LOG_LEVEL)

      --data-set-id=STRING          task ID
      --dry-run                     if true, no update data set and display plan
      --force-rename                The default is to keep any renaming that has already taken place. Enabling this option forces a name overwrite.
      --force-update-description    The default is to keep any renaming that has already taken place. Enabling this option forces a description overwrite.

Configurations

Configuration file is HCL (HashiCorp Configuration Language) format. redshift-data-set-annotator init can generate a initial configuration file.

The most small configuration file is as follows:

redshift-data-set-annotator {
    required_version = ">=v0.2.0"
    sqs_queue_name   = "redshift-data-set-annotator"
    service          = "prod"
}

rule "any_alert" {
    alert {
        any = true
    }

    information = <<EOF
How do you respond to alerts?
Describe information about your alert response here.
(This area can use Go's template notation.)
EOF
}

For more advanced configuration, please see docs directory

Usage with AWS Lambda (serverless)

redshift-data-set-annotator works with AWS Lambda and Amazon SQS.

Lambda Function requires a webhook and a worker

sequenceDiagram
  autonumber
  Mackerel->>+http lambda function : POST /
  http lambda function ->>+Amazon SQS: SendMessage
  Amazon SQS-->- http lambda function: 200 Ok
  http lambda function-->- Mackerel: 200 Ok
  Amazon SQS ->>+ worker lambda function: trigger by AWS Lambda
  worker lambda function ->>+ Data Source: query
  Data Source -->- worker lambda function: query results
  worker lambda function  ->>+ Mackerel: Create Graph Annotation
  Mackerel-->- worker lambda function : 200 Ok
  worker lambda function ->>-  Amazon SQS: Success Delete message

Let's solidify the Lambda package with the following zip arcive (runtime provided.al2)

lambda.zip
├── bootstrap    # build binary
└── config.hcl   # configuration file

A related document is https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

for example.

deploy two lambda functions, redshift-data-set-annotator-http and redshift-data-set-annotator-worker in lambda directory
The example of lambda directory uses lambroll for deployment.

For more information on the infrastructure around lambda functions, please refer to example.tf.

LICENSE

MIT License

Copyright (c) 2022 IKEDA Masashi

Documentation

Index

• Variables
• func NewUpdateDataSetInput(dataSet *types.DataSet) (*quicksight.UpdateDataSetInput, error)
• func RunCLI(ctx context.Context, args []string) error
• type AnnotateOption
• type App
  • func New(ctx context.Context, awsAccountID string) (*App, error)
  • func (app *App) AWSAccountID() string
  • func (app *App) DescribeDataSrouce(ctx context.Context, dataSourceArn string) (*quicksight.DescribeDataSourceOutput, error)
  • func (app *App) Dispatch(ctx context.Context, command string, cli *CLI) error
  • func (app *App) GetColumnAnnotations(ctx context.Context, ds *types.DataSource, table types.RelationalTable) (ColumnAnnotations, error)
  • func (app *App) GetDSN(params types.RedshiftParameters) (string, error)
  • func (app *App) RunAnnotate(ctx context.Context, opt *AnnotateOption) error
  • func (app *App) RunConfigure(ctx context.Context, opt *ConfigureOption) error
• type CLI
• type ColumnAnnotation
• type ColumnAnnotations
• type Config
  • func (cfg Config) Get(host string) (*ProfileConfig, bool)
  • func (cfg Config) GetDefault() *ProfileConfig
  • func (cfg Config) String() string
• type ConfigureOption
• type ProfileConfig

Variables

var Version string

Functions

func NewUpdateDataSetInput

func NewUpdateDataSetInput(dataSet *types.DataSet) (*quicksight.UpdateDataSetInput, error)

func RunCLI

func RunCLI(ctx context.Context, args []string) error

Types

type AnnotateOption

type AnnotateOption struct {
	DataSetID              string `help:"task ID" required:""`
	DryRun                 bool   `help:"if true, no update data set and display plan"`
	ForceRename            bool   `help:"The default is to keep any renaming that has already taken place. Enabling this option forces a name overwrite."`
	ForceUpdateDescription bool   `help:"The default is to keep any renaming that has already taken place. Enabling this option forces a description overwrite."`
	Verbose                bool   `help:"Outputs the input information for the UpdateDataSet API"`
}

type App

type App struct {
	// contains filtered or unexported fields
}

func New

func New(ctx context.Context, awsAccountID string) (*App, error)

func (*App) AWSAccountID

func (app *App) AWSAccountID() string

func (*App) DescribeDataSrouce

func (app *App) DescribeDataSrouce(ctx context.Context, dataSourceArn string) (*quicksight.DescribeDataSourceOutput, error)

func (*App) Dispatch

func (app *App) Dispatch(ctx context.Context, command string, cli *CLI) error

func (*App) GetColumnAnnotations

func (app *App) GetColumnAnnotations(ctx context.Context, ds *types.DataSource, table types.RelationalTable) (ColumnAnnotations, error)

func (*App) GetDSN

func (app *App) GetDSN(params types.RedshiftParameters) (string, error)

func (*App) RunAnnotate

func (app *App) RunAnnotate(ctx context.Context, opt *AnnotateOption) error

func (*App) RunConfigure

func (app *App) RunConfigure(ctx context.Context, opt *ConfigureOption) error

type CLI

type CLI struct {
	AWSAccountID string `help:"QuickSight aws account id"`
	Region       string `help:"AWS region" short:"r" env:"AWS_REGION"`
	LogLevel     string `help:"output log level" env:"LOG_LEVEL" default:"info"`

	Configure *ConfigureOption `cmd:"" help:"Create a configuration file of redshift-data-set-annotator"`
	Annotate  *AnnotateOption  `cmd:"" help:"Annotate a QuickSight dataset with Redshift as the data source"`
	Version   struct{}         `cmd:"" help:"Show version"`
}

type ColumnAnnotation

type ColumnAnnotation struct {
	CoumnName   string  `db:"column_name"`
	Name        *string `db:"name"`
	Description *string `db:"description"`
}

type ColumnAnnotations

type ColumnAnnotations map[string]*ColumnAnnotation

type Config

type Config map[string]*ProfileConfig

func (Config) Get

func (cfg Config) Get(host string) (*ProfileConfig, bool)

func (Config) GetDefault

func (cfg Config) GetDefault() *ProfileConfig

func (Config) String

func (cfg Config) String() string

type ConfigureOption

type ConfigureOption struct {
	Host string `help:"redshift host address" default:""`
	Show bool   `help:"show current configuration" short:"s"`
}

type ProfileConfig

type ProfileConfig struct {
	ClusterIdentifier *string `json:"cluster_identifier,omitempty"`
	WorkgroupName     *string `json:"workgroup_name,omitempty"`
	DBUser            *string `json:"db_user,omitempty"`
}