cloudsnorkelcdkrdssanitizedsnapshots

README

CDK Construct for RDS Sanitized Snapshots

Periodically take snapshots of RDS databases, sanitize them, and share with selected accounts.

Use this to automate your development and/or QA database creation, instead of forcing them to use a database that was created last year and was kind of kept in shape by random acts of kindness. Developers and QA love real data and this lets you create non-production databases with sanitized production data. Use the sanitization step to delete passwords, remove credit card numbers, eliminate PII, etc.

See Constructs Hub for installation instructions and API in all supported languages.

Overview

This project supplies a CDK construct that sets up a step function and a timer to execute this function. The function will create a sanitized snapshot of a given database and share it with configured accounts. Those accounts can then create new databases from those snapshots.

The step function does the following to create the snapshot:

1. Get a snapshot of the given database by either:

   • Finding the latest snapshot for the given database
   • Creating and waiting for a new fresh snapshot

2. Re-encrypt snapshot if KMS key is supplied

3. Create a temporary database from the snapshot

4. Wait for the database to be ready

5. Reset the master password on the temporary database to a random password

6. Wait for the password to be set

7. Use a Fargate task to connect to the temporary database and run configured SQL statements to sanitize the data

8. Take a snapshot of the temporary database

9. Optionally share the snapshot with other accounts (if you have separate accounts for developers/QA)

10. Delete temporary database and snapshot

Usage

1. Confirm you're using CDK v2

2. Install the appropriate package

   1. Python

      pip install cloudsnorkel.cdk-rds-sanitized-snapshots
      

   2. TypeScript or JavaScript

      npm i @cloudsnorkel/cdk-rds-sanitized-snapshots
      

   3. Java

      <dependency>
      <groupId>com.cloudsnorkel</groupId>
      <artifactId>cdk.rds.sanitized-snapshots</artifactId>
      </dependency>
      

   4. Go

      go get github.com/CloudSnorkel/cdk-rds-sanitized-snapshots-go/cloudsnorkelcdkrdssanitizedsnapshots
      

   5. .NET

      dotnet add package CloudSnorkel.Cdk.Rds.SanitizedSnapshots
      

3. Use RdsSanitizedSnapshotter construct in your code (starting with default arguments is fine)

Code Sample

let vpc: ec2.Vpc;
let databaseInstance: rds.DatabaseInstance;

new RdsSanitizedSnapshotter(this, 'Snapshotter', {
  vpc: vpc,
  databaseInstance: databaseInstance,
  script: 'USE mydb; UPDATE users SET ssn = \'0000000000\'',
})

Encryption

The new snapshot will be encrypted with the same key used by the original database. If the original database wasn't encrypted, the snapshot won't be encrypted either. To add another step that changes the key, use the KMS key parameter.

See AWS documentation for instructions on giving other accounts access to the key.

Troubleshooting

• Check the status of the state machine for the step function. Click on the failed step and check out the input, output and exception.

Testing

npm run bundle && npm run integ:default:deploy

Documentation

Overview

CDK construct to periodically take snapshots of RDS databases, sanitize them, and share with selected accounts.

Index

• func NewRdsSanitizedSnapshotter_Override(r RdsSanitizedSnapshotter, scope constructs.Construct, id *string, ...)
• func RdsSanitizedSnapshotter_IsConstruct(x interface{}) *bool
• type IRdsSanitizedSnapshotter
• type RdsSanitizedSnapshotter
  • func NewRdsSanitizedSnapshotter(scope constructs.Construct, id *string, props IRdsSanitizedSnapshotter) RdsSanitizedSnapshotter

Functions

func NewRdsSanitizedSnapshotter_Override

func NewRdsSanitizedSnapshotter_Override(r RdsSanitizedSnapshotter, scope constructs.Construct, id *string, props IRdsSanitizedSnapshotter)

Experimental.

func RdsSanitizedSnapshotter_IsConstruct

func RdsSanitizedSnapshotter_IsConstruct(x interface{}) *bool

Checks if `x` is a construct.

Returns: true if `x` is an object created from a class which extends `Construct`. Deprecated: use `x instanceof Construct` instead.

Types

type IRdsSanitizedSnapshotter

type IRdsSanitizedSnapshotter interface {
	// Database cluster to snapshot and sanitize.
	//
	// Only one of `databaseCluster` and `databaseInstance` can be specified.
	// Experimental.
	DatabaseCluster() awsrds.IDatabaseCluster
	// Database instance to snapshot and sanitize.
	//
	// Only one of `databaseCluster` and `databaseInstance` can be specified.
	// Experimental.
	DatabaseInstance() awsrds.IDatabaseInstance
	// KMS key used to encrypt original database, if any.
	// Experimental.
	DatabaseKey() awskms.IKey
	// Name of database to connect to inside the RDS cluster or instance.
	//
	// This database will be used to execute the SQL script.
	// Default: 'postgres' for PostgreSQL and not set for MySQL.
	//
	// Experimental.
	DatabaseName() *string
	// VPC subnets to use for temporary databases.
	// Default: ec2.SubnetType.PRIVATE_ISOLATED
	//
	// Experimental.
	DbSubnets() *awsec2.SubnetSelection
	// Cluster where sanitization task will be executed.
	// Default: a new cluster running on given VPC.
	//
	// Experimental.
	FargateCluster() awsecs.ICluster
	// VPC subnets to use for sanitization task.
	// Default: ec2.SubnetType.PRIVATE_WITH_EGRESS
	//
	// Experimental.
	SanitizeSubnets() *awsec2.SubnetSelection
	// The schedule or rate (frequency) that determines when the sanitized snapshot runs automatically.
	// Experimental.
	Schedule() awsevents.Schedule
	// SQL script used to sanitize the database. It will be executed against the temporary database.
	//
	// You would usually want to start this with `USE mydatabase;`.
	// Experimental.
	Script() *string
	// List of accounts the sanitized snapshot should be shared with.
	// Experimental.
	ShareAccounts() *[]*string
	// Limit the number of snapshot history.
	//
	// Set this to delete old snapshots and only leave a certain number of snapshots.
	// Experimental.
	SnapshotHistoryLimit() *float64
	// Optional KMS key to encrypt target snapshot.
	// Experimental.
	SnapshotKey() awskms.IKey
	// Prefix for sanitized snapshot name.
	//
	// The current date and time will be added to it.
	// Default: cluster identifier (which might be too long).
	//
	// Experimental.
	SnapshotPrefix() *string
	// Prefix for all temporary snapshots and databases.
	//
	// The step function execution id will be added to it.
	// Default: 'sanitize'.
	//
	// Experimental.
	TempPrefix() *string
	// Use the latest available snapshot instead of taking a new one.
	//
	// This can be used to shorten the process at the cost of using a possibly older snapshot.
	//
	// This will use the latest snapshot whether it's an automatic system snapshot or a manual snapshot.
	// Default: false.
	//
	// Experimental.
	UseExistingSnapshot() *bool
	// VPC where temporary database and sanitizing task will be created.
	// Experimental.
	Vpc() awsec2.IVpc
}

Experimental.

type RdsSanitizedSnapshotter

type RdsSanitizedSnapshotter interface {
	constructs.Construct
	// The tree node.
	// Experimental.
	Node() constructs.Node
	// Experimental.
	Props() IRdsSanitizedSnapshotter
	// Step function in charge of the entire process including snapshotting, sanitizing, and cleanup.
	//
	// Trigger this step function to get a new snapshot.
	// Experimental.
	Snapshotter() awsstepfunctions.StateMachine
	// Experimental.
	SetSnapshotter(val awsstepfunctions.StateMachine)
	// Returns a string representation of this construct.
	// Experimental.
	ToString() *string
}

A process to create sanitized snapshots of RDS instance or cluster, optionally on a schedule.

The process is handled by a step function.

1. Snapshot the source database 2. Optionally re-encrypt the snapshot with a different key in case you want to share it with an account that doesn't have access to the original key 3. Create a temporary database 4. Run a Fargate task to connect to the temporary database and execute an arbitrary SQL script to sanitize it 5. Snapshot the sanitized database 6. Clean-up temporary snapshots and databases Experimental.

func NewRdsSanitizedSnapshotter

func NewRdsSanitizedSnapshotter(scope constructs.Construct, id *string, props IRdsSanitizedSnapshotter) RdsSanitizedSnapshotter

Experimental.