lfsb

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 5, 2024 License: MIT Imports: 7 Imported by: 0

README ¶

📦 litefs-backup (lfsb)

A drop-in, self-hosted replacement for LiteFS Cloud, which was deprecated in July 2024.

Litefs-backup is a standalone backup process that streams backups of your LiteFS cluster to any s3-compatible storage provider, with 5 minute point-in-time restore granularity. It supports both static and dynamic LiteFS lease modes.

This project is sponsored by Tender - a private, productive personal finance tool.

This codebase is a simplified fork of LiteFS Cloud, originally authored and graciously donated by @benbjohnson and co at fly.io.

Limitations & differences with LiteFS Cloud

  • Lfsb does not do any authentication and assumes access across a private network.
  • Lfsb is API-compatible with LiteFS, but its control plane API slightly differs from LiteFS Cloud. Calls from packages like lfsc-go may not work or have expected results.
  • Lfsb expects to be run as a singleton service. Running multiple instances could cause corruption or inconsistencies.

Quickstart: switching from LiteFS Cloud

Lfsb is intended to be a mostly drop-in replacement for LiteFS Cloud. For system requirements, lfsb needs a single-node machine with a persistent disk and s3-compatible object storage.

Deployment on fly.io

Prerequisite: set up a fly.io account and install flyctl.

  1. Launch the app from this repository.
mkdir litefs-backup
cd litefs-backup
fly launch --from git@github.com:stephen/litefs-backup.git --build-only --generate-name --copy-config

When prompted to tweak settings, choose no.

  1. Add tigris object storage and sentry (optional)

If you want to use S3 or another S3-compatible provider, skip this step and use fly secrets set to set the necessary environment variables.

fly ext storage create
fly ext sentry create # optional

These commands should automatically configure the necessary environment variables.

  1. Deploy privately with a single instance
fly deploy --ha=false --no-public-ips --flycast
  1. Finish
fly logs

# Expected output:
# INFO running litefs-backup
# INFO server listening addr=:2200
# INFO waiting for signal or subprocess to exit
# INFO monitoring compaction level=1 interval=10s retention=1h0m0s
# INFO monitoring compaction level=2 interval=5m0s retention=720h0m0s
# INFO monitoring compaction level=3 interval=1h0m0s retention=720h0m0s

This configuration gives the same 5-minute granularity point-in-time restore as LiteFS Cloud.

Configure LiteFS to use lfsb

Use litefs from commit 2e78724973211ba426c224e89d99c256fd6722be or later. If you pull litefs from docker, this looks like:

COPY --from=flyio/litefs:sha-2e78724 /usr/local/bin/litefs /usr/local/bin/litefs

Configure your service running litefs with two environment variables:

  • Set LITEFS_CLOUD_ENDPOINT to the location of your newly deployed lfsb. On fly, this might look like http://someones-litefs-backup.internal:2200.
  • Set LITEFS_CLOUD_TOKEN to cluster [cluster name], e.g. cluster canary. Clusters do not need to be pre-registered.

Using the control plane

Install and configure the cli

Install the release for your system and place it in your $PATH.

curl "https://github.com/stephen/litefs-backup/releases/latest/download/litefs-backup_$(uname -s)_$(uname -m).tar.gz" 2>/dev/null | tar -xz run 2>/dev/null

You can optionally set LFSB_CLUSTER and LFSB_ENDPOINT to your expected cluster and endpoint url, e.g.

export LFSB_CLUSTER="prod" # or --cluster
export LFSB_ENDPOINT="http://someones-litefs-backup.internal:2200" # or --endpoint

If you are using fly.io, setup a wireguard vpn to connect into your private network:

fly wireguard create
Basic operation

List known databases:

lfsb list

List restore points for a database:

lfsb info data.db

Download the current snapshot of a database:

lfsb export data.db

Restore a database to a timestamp or txid:

# first, do a dry run to check that the restore point is available:
lfsb restore data.db --check --txid=[txid] # or --timestamp=[timestamp, e.g. 2024-06-08T12:22:39]

# now, actually run the restore
lfsb restore data.db --txid=[txid]

System overview

Lfsb is organized into clusters. A cluster can contain multiple sqlite databases. For instance, you might have a prod cluster with users.db and datapoints.db and another cluster beta with separate users.db and jobs.db.

Authentication

Lfsb does not support any authentication scheme. To keep API-compatibility with LiteFS, the Authorization header is still used to identify which cluster LiteFS is connecting to.

The authorization header is in the format cluster [cluster name], instead of the LiteFS Cloud format FlyV1 [token].

Storage subsystem

Lfsb stores data in two places:

  • a sqlite metadata db on disk
  • s3-compatible remote storage

Both are expected to be available and durable for the system to function.

Development

Setup

go install github.com/amonks/run/cmd/run@latest
run install

Test

run test
run test-integration

Migrations

dbmate [up, down, new]

Configuration

Configuration is done through environment variables.

LFSB_DATA_PATH (required)

The directory where lfsb will keep its local data store. This should be a persistent, durable location.

LFSB_BIND (optional)

The address to bind to. The default is :2200.

Note that the .envrc sets this to 127.0.0.1:2200 to help with macOS security prompting.

BUCKET_NAME (required)

The AWS S3 bucket name to use.

AWS_ENDPOINT_URL_S3 (optional)

The AWS S3 endpoint to use. The default is AWS S3.

This option is useful for using S3-compatible storage providers.

AWS_REGION (required)

The AWS S3 region to use.

AWS_ACCESS_KEY (required)

The AWS S3 access key to use.

AWS_SECRET_KEY (required)

The AWS S3 secret key to use.

SENTRY_DSN (optional)

The Sentry DSN to use. Sentry reporting will be disabled if unset.

Documentation ¶

Index ¶

Constants ¶

View Source 
const (
	EPOSMISMATCH       = "EPOSMISMATCH"
	ENOCLUSTER         = "ENOCLUSTER"
	ENOCOMPACTION      = "ENOCOMPACTION"
	EPARTIALCOMPACTION = "EPARTIALCOMPACTION"
)

Common error codes. Used for compile-time checks.

View Source 
const (
	// CompactionLevelMax is the highest compaction level allowed.
	CompactionLevelMax = 8

	// CompactionLevelSnapshot is a reserved level for full snapshots only.
	CompactionLevelSnapshot = 9
)
View Source 
const (
	MaxClusterLen  = 32
	MaxDatabaseLen = 256
)

Maximum identifier lengths.

View Source 
const LTXFilenameLen = 16 + 1 + 16 + 4 // min,dash,max,ext

LTXFilenameLen is the length of LTX base filenames.

Variables ¶

View Source 
var (
	ErrInvalidLTXFilename = Errorf(ErrorTypeValidation, "EBADPATH", "invalid ltx filename")

	ErrClusterRequired = Errorf(ErrorTypeValidation, "EBADCLUSTER", "cluster required")
	ErrClusterInvalid  = Errorf(ErrorTypeValidation, "EBADCLUSTER", "cluster invalid")

	ErrDatabaseRequired = Errorf(ErrorTypeValidation, "EBADDB", "database required")
	ErrDatabaseNotFound = Errorf(ErrorTypeNotFound, "ENODB", "database not found")

	ErrMinTXIDRequired = Errorf(ErrorTypeValidation, "EBADTXID", "minimum transaction id required")
	ErrMaxTXIDRequired = Errorf(ErrorTypeValidation, "EBADTXID", "maximum transaction id required")

	ErrCannotCompactToLevelZero = Errorf(ErrorTypeValidation, "EBADLEVEL", "cannot compact to level zero")
	ErrCompactionLevelTooHigh   = Errorf(ErrorTypeValidation, "EBADLEVEL", "compaction level too high")

	ErrTxNotAvailable = Errorf(ErrorTypeNotFound, "ENOTXID", "tx not available")
	// ErrPgnoOutOfBounds = Errorf(ErrorTypeNotFound, "EPGNOOOB", "page number out of bounds")
	ErrPageNotFound = Errorf(ErrorTypeNotFound, "ENOPAGE", "page not found")

	ErrTimestampNotAvailable = Errorf(ErrorTypeNotFound, "ENOTIMESTAMP", "timestamp not available")

	ErrPageSizeMismatch = Errorf(ErrorTypeUnprocessable, "EBADHEADER", "page size mismatch")
)

Functions ¶

func ErrorCode ¶ 

func ErrorCode(err error) string

ErrorCode returns the error code from an error. Returns blank if err is nil. Returns EINTERNAL if no lfsc.Error is found.

func FormatLTXFilename ¶ 

func FormatLTXFilename(min, max ltx.TXID) string

FormatLTXFilename returns a filename based on the min & max TXID.

func IsApplicationError ¶ 

func IsApplicationError(err error) bool

IsApplicationError returns true if err is an lfsc.Error or ltx.PositionMismatchError.

func ParseLTXFilename ¶ 

func ParseLTXFilename(name string) (min, max ltx.TXID, err error)

ParseLTXFilename parses a base filename into min & max TXID. Returns ErrInvalidLTXFilename if it is invalid.

func ValidateClusterName ¶ 

func ValidateClusterName(s string) error

ValidateClusterName returns nil if s is a valid cluster.

func ValidateDatabase ¶ 

func ValidateDatabase(s string) error

ValidateDatabase returns nil if s is a valid database name.

Types ¶

type Config ¶ 

type Config struct {
	// Path is the directory where litefs-backup should store its
	// local data.
	Path string

	// Address is the address to bind to.
	Address string

	// S3Bucket is the bucket to use for long term storage.
	S3Bucket string

	// S3Endpoint is the S3 api endpoint to use. Useful for using S3-compatible alternatives.
	S3Endpoint string

	// SentryDSN is the sentry DSN to use.
	// See: https://docs.sentry.io/concepts/key-terms/dsn-explainer/
	SentryDSN string
}

func ConfigFromEnv ¶ 

func ConfigFromEnv() (*Config, error)

type Error ¶ 

type Error struct {
	Type              ErrorType
	Code              string
	Message           string
	TXID              ltx.TXID
	PostApplyChecksum ltx.Pos
}

func Errorf ¶ 

func Errorf(typ ErrorType, code, format string, a ...any) *Error

Errorf is a helper function for returning Error values.

func (*Error) Error ¶ 

func (e *Error) Error() string

Error implements Error interace

func (*Error) Is ¶ 

func (e *Error) Is(err error) bool

type ErrorType ¶ 

type ErrorType string
const (
	ErrorTypeAuth          ErrorType = "auth"
	ErrorTypeConflict      ErrorType = "conflict"
	ErrorTypeNotFound      ErrorType = "notfound"
	ErrorTypeValidation    ErrorType = "validation"
	ErrorTypeUnprocessable ErrorType = "unprocessable"

	ErrorTypeUnknown = "unknown"
)

Source Files ¶

View all Source files

Directories ¶

Path Synopsis
cmd
lfsb
lfsb-server
lfsb/cmd
* From https://github.com/superfly/flyctl/blob/19481dfa110c8a6274500321b3214c5d2abbfc71/internal/render/render.go * Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * From https://github.com/superfly/flyctl/blob/19481dfa110c8a6274500321b3214c5d2abbfc71/internal/render/render.go * Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
lfsb/cmd/client
* Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
db
migrations
sqliteutil
testingutil
s3

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.