database-lab

module
v3.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 5, 2022 License: AGPL-3.0

README

Database Lab Engine (DLE)

:zap: Blazing-fast cloning of PostgreSQL databases :elephant:
Thin clones of PostgreSQL to build powerful development, test, QA, staging environments.
Available for any PostgreSQL, including AWS RDS, GCP CloudSQL, Heroku, Digital Ocean, and self-managed instances.

Latest release

CI pipeline status Go report

Contributor Covenant Community Slack Twitter Follow

Why DLE?

  • Build dev/QA/staging environments based on full-size production-like databases.
  • Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
  • Automatically test database changes in CI/CD pipelines to avoid incidents in production.

For example, cloning a 1 TiB PostgreSQL database takes ~10 seconds. Dozens of independent clones are up and running on a single machine, supporting lots of development and testing activities, not increasing costs for hardware.

How it works

Thin cloning is fast because it uses Copy-on-Write (CoW). DLE supports two technologies to enable CoW and thin cloning: ZFS (default) and LVM.

With ZFS, Database Lab Engine periodically creates a new snapshot of the data directory and maintains a set of snapshots, cleaning up the old and unused ones. When requesting a new clone, users can choose which snapshot to use.

Read more:

Where to start

Case studies

Features

  • Blazing-fast cloning of Postgres databases – a few seconds to create a new clone ready to accept connections and queries, regardless of the database size.
  • The theoretical maximum number of snapshots and clones is 264 (ZFS, default).
  • The theoretical maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default).
  • PostgreSQL major versions supported: 9.6–14.
  • Two technologies are supported to enable thin cloning (CoW): ZFS and LVM.
  • All components are packaged in Docker containers.
  • UI to make manual work more convenient.
  • API and CLI to automate the work with DLE snapshots and clones.
  • By default, PostgreSQL containers include many popular extensions (docs).
  • PostgreSQL containers can be customized (docs).
  • Source database can be located anywhere (self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, Timescale Cloud, and so on) and does NOT require any adjustments. There are NO requirements to install ZFS or Docker to the source (production) databases.
  • Initial data provisioning can be at both physical (pg_basebackup, backup / archiving tools such as WAL-G or pgBackRest), or logical (dump/restore directly from the source or from files stored at AWS S3) levels.
  • For the logical mode, partial data retrieval is supported (specific databases, specific tables).
  • For the physical mode, a continuously updated state is supported ("sync container"), making DLE a specialized version of standby Postgres.
  • For the logical mode, periodical full refresh is supported, automated, and controlled by DLE. It is possible to use multiple disks containing different versions of the database, so full refresh won't require downtime.
  • Fast Point in Time Recovery (PITR) to the points available in DLE snapshots.
  • Unused clones are automatically deleted.
  • "Deletion protection" flag can be used to block automatic or manual deletion of clones.
  • Snapshot retention policies supported in DLE configuration.
  • Persistent clones: clones survive DLE restarts (including full VM reboots).
  • The "reset" command can be used to switch to a different version of data.
  • DB Migration Checker component collects various artifacts useful for DB testing in CI (docs).
  • SSH port forwarding for API and Postgres connections.
  • Docker container config parameters can be specified in the DLE config.
  • Resource usage quotas for clones: CPU, RAM (container quotas, supported by Docker)
  • Postgres config parameters can be specified in the DLE config (separately for clones, the "sync" container, and the "promote" container).
  • Monitoring: auth-free /healthz API endpoint, extended /status (requires auth), Netdata module.

How to contribute

Give the project a star

The easiest way to contribute is to give the project a GitHub/GitLab star:

Add a star

Mention that you use DLE

Please post a tweet mentioning @Database_Lab or share the link to this repo in your favorite social network.

If you are actively using DLE at work, think about where you could mention it. The best way of mentioning it is using graphics with a link. Brand assets can be found in the ./assets folder. Feel free to put them in your documents, slide decks, application, and website interfaces to show that you use DLE.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

Fro darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

Propose an idea or report a bug

Check out our contributing guide for more details.

Participate in development

Check out our contributing guide for more details.

Reference guides

How-to guides

More you can found in the "How-to guides" section of the docs.

Miscellaneous

License

DLE source code is licensed under the OSI-approved open source license GNU Affero General Public License version 3 (AGPLv3).

Reach out to the Postgres.ai team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

Community & Support

Contributor Covenant

Directories

Path Synopsis
cmd
cli
cli/commands
Package commands provides general resources for CLI.
 Package commands provides general resources for CLI.
cli/commands/clone
Package clone provides clones management commands.
 Package clone provides clones management commands.
cli/commands/config
Package config provides commands for a CLI config management.
 Package config provides commands for a CLI config management.
cli/commands/global
Package global provides general commands for CLI usage.
 Package global provides general commands for CLI usage.
cli/commands/instance
Package instance provides instance management commands.
 Package instance provides instance management commands.
cli/commands/snapshot
Package snapshot provides snapshot management commands.
 Package snapshot provides snapshot management commands.
cli/templates
Package templates contains custom template output.
 Package templates contains custom template output.
database-lab
runci
internal
cloning
Package cloning provides a cloning service.
 Package cloning provides a cloning service.
embeddedui
Package embeddedui manages embedded UI container.
 Package embeddedui manages embedded UI container.
estimator
Package estimator provides tools to estimate query timing for a production environment.
 Package estimator provides tools to estimate query timing for a production environment.
observer
Package observer provides clone monitoring.
 Package observer provides clone monitoring.
platform
Package platform provides a Platform service.
 Package platform provides a Platform service.
portfwd
Package portfwd provides an SSH port forwarder.
 Package portfwd provides an SSH port forwarder.
provision
Package provision provides an interface to provision Database Lab clones.
 Package provision provides an interface to provision Database Lab clones.
provision/databases/postgres
Package postgres provides an interface to work Postgres application.
 Package postgres provides an interface to work Postgres application.
provision/databases/postgres/pgconfig
Package pgconfig provides tools for work with Postgres configuration.
 Package pgconfig provides tools for work with Postgres configuration.
provision/docker
Package docker provides an interface to work with Docker containers.
 Package docker provides an interface to work with Docker containers.
provision/pool
Package pool provides components to work with storage pools.
 Package pool provides components to work with storage pools.
provision/resources
Package resources defines models used for provisioning.
 Package resources defines models used for provisioning.
provision/runners
Package runners provides an interface to execute commands.
 Package runners provides an interface to execute commands.
provision/thinclones
Package thinclones provides an interface to work different thin-clone managers.
 Package thinclones provides an interface to work different thin-clone managers.
provision/thinclones/lvm
Package lvm provides an interface to work with LVM2.
 Package lvm provides an interface to work with LVM2.
provision/thinclones/zfs
Package zfs provides an interface to work with ZFS.
 Package zfs provides an interface to work with ZFS.
retrieval
Package retrieval provides data retrieval pipeline.
 Package retrieval provides data retrieval pipeline.
retrieval/components
Package components provides the key components of data retrieval.
 Package components provides the key components of data retrieval.
retrieval/config
Package config contains configuration options of the data retrieval.
 Package config contains configuration options of the data retrieval.
retrieval/dbmarker
Package dbmarker provides a tool for marking database data.
 Package dbmarker provides a tool for marking database data.
retrieval/engine
Package engine provides different engines.
 Package engine provides different engines.
retrieval/engine/postgres
Package postgres contains data retrieval jobs for a Postgres engine.
 Package postgres contains data retrieval jobs for a Postgres engine.
retrieval/engine/postgres/logical
Package logical provides jobs for logical initial operations.
 Package logical provides jobs for logical initial operations.
retrieval/engine/postgres/physical
Package physical provides jobs for physical initial operations.
 Package physical provides jobs for physical initial operations.
retrieval/engine/postgres/snapshot
Package snapshot provides components for preparing initial snapshots.
 Package snapshot provides components for preparing initial snapshots.
retrieval/engine/postgres/tools
Package tools provides helpers to initialize data.
 Package tools provides helpers to initialize data.
retrieval/engine/postgres/tools/cont
Package cont provides tools to manage service containers started by Database Lab Engine.
 Package cont provides tools to manage service containers started by Database Lab Engine.
retrieval/engine/postgres/tools/db
Package db provides database helpers.
 Package db provides database helpers.
retrieval/engine/postgres/tools/defaults
Package defaults contains default values.
 Package defaults contains default values.
retrieval/engine/postgres/tools/fs
Package fs provides tools for working with the filesystem.
 Package fs provides tools for working with the filesystem.
retrieval/engine/postgres/tools/health
Package health provides tools to set up container health check options.
 Package health provides tools to set up container health check options.
retrieval/engine/postgres/tools/pgtool
Package pgtool provides tools to run PostgreSQL-specific commands.
 Package pgtool provides tools to run PostgreSQL-specific commands.
retrieval/options
Package options provides helpers to process retrieval options.
 Package options provides helpers to process retrieval options.
runci
Package runci provides a tools to run and check migrations in CI.
 Package runci provides a tools to run and check migrations in CI.
runci/source
Package source provides a tools to use version control systems.
 Package source provides a tools to use version control systems.
srv
Package srv contains API routes and handlers.
 Package srv contains API routes and handlers.
srv/api
Package api contains helpers to work with HTTP requests and responses.
 Package api contains helpers to work with HTTP requests and responses.
srv/config
Package config contains configuration options of HTTP server.
 Package config contains configuration options of HTTP server.
srv/mw
Package mw contains middlewares.
 Package mw contains middlewares.
telemetry
Package telemetry contains tools to collect Database Lab Engine data.
 Package telemetry contains tools to collect Database Lab Engine data.
validator
Package validator provides a validation service.
 Package validator provides a validation service.
pkg
client/dblabapi
Package dblabapi provides a client for Database Lab HTTP API.
 Package dblabapi provides a client for Database Lab HTTP API.
client/dblabapi/types
Package types provides request structures for Database Lab HTTP API.
 Package types provides request structures for Database Lab HTTP API.
client/platform
Package platform provides the Platform API client.
 Package platform provides the Platform API client.
config
Package config provides access to the Database Lab configuration.
 Package config provides access to the Database Lab configuration.
config/global
Package global provides access to the global Database Lab Engine configuration.
 Package global provides access to the global Database Lab Engine configuration.
log
Package log formats and prints log messages.
 Package log formats and prints log messages.
models
Package models provides Database Lab struct.
 Package models provides Database Lab struct.
util
Package util provides utility functions.
 Package util provides utility functions.
util/engine
Package engine contains tools.
 Package engine contains tools.
util/networks
Package networks describes custom network elements.
 Package networks describes custom network elements.
util/pglog
Package pglog provides helpers for a Postgres logs processing.
 Package pglog provides helpers for a Postgres logs processing.
Package version provides the Database Lab version info.
 Package version provides the Database Lab version info.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.