gitdb

README

GitDB

GitDB is an implementation of ScootDB (snapshot.DB) that stores local values in a git ODB.

Code Structure Overview

• db.go structure definition, top-level entry point, concurrency control
• backends.go ID definition and parsing; backend definition
• create.go create Snapshots (locally)
• checkout.go run git commands to checkout
• local_data.go Snapshots stored locally
• stream.go get Snapshots from an upstream git repo

Backends

GitDB uses different Backends to identify, upload and download Snapshots.

Exit Codes

GitDB defines a set of exit codes that are returned depending on various git related errors (see common/errors/exit_codes.go for definitive list)

Snapshot ID format

GitDB uses Snapshot IDs that encode the backend, kind and per-backend data, separated by '-':

• -(-)+

E.g.:

• local-gc-3cf4f8cf84976621c3ca9f19dff114842b4e5db7
  • backend: local
  • kind: GitCommitSnapshot *sha: 3cf4f8cf84976621c3ca9f19dff114842b4e5db7
• stream-gc-sm-3cf4f8cf84976621c3ca9f19dff114842b4e5db7
  • backend: stream
  • kind: GitCommitSnapshot
  • stream name: source_master *sha: 3cf4f8cf84976621c3ca9f19dff114842b4e5db7
• bs-fs-sm-530c6daad567a0765c10064e1c7fc4fa486a2638-d1f58ef31066244fc8590bfb6940b4060b1baab1
  • backend: Bundlestore
  • kind: FSSnapshot
  • stream name: source_master
  • bundle name: 530c6daad567a0765c10064e1c7fc4fa486a2638
  • sha: d1f58ef31066244fc8590bfb6940b4060b1baab1

Documentation

Index

• Constants
• func CreateBundlestoreSnapshot(sha string, kind SnapshotKind, bundleKey, streamName string) snapshot
• func Module() ice.Module
• type AutoUploadDest
• type BundlestoreConfig
• type DB
  • func MakeDBFromRepo(dataRepo *repo.Repository, updater RepoUpdater, tmp string, ...) *DB
  • func MakeDBNewRepo(initer RepoIniter, updater RepoUpdater, tmp string, stream *StreamConfig, ...) *DB
  • func (db *DB) Cancel() error
  • func (db *DB) Checkout(id snap.ID) (path string, err error)
  • func (db *DB) Close()
  • func (db *DB) ExportGitCommit(id snap.ID, exportRepo *repo.Repository) (string, error)
  • func (db *DB) IDForStreamCommitSHA(streamName string, sha string) snap.ID
  • func (db *DB) IngestDir(dir string) (snap.ID, error)
  • func (db *DB) IngestGitCommit(ingestRepo *repo.Repository, commitish string) (snap.ID, error)
  • func (db *DB) IngestGitWorkingDir(ingestRepo *repo.Repository) (snap.ID, error)
  • func (db *DB) ReadFileAll(id snap.ID, path string) ([]byte, error)
  • func (db *DB) ReleaseCheckout(path string) error
  • func (db *DB) StreamName() string
  • func (db *DB) Update() error
  • func (db *DB) UpdateInterval() time.Duration
  • func (db *DB) UploadFile(filePath string, ttl *store.TTLValue) (string, error)
• type NoopRepoUpdater
  • func (u *NoopRepoUpdater) Update(*repo.Repository) error
  • func (u *NoopRepoUpdater) UpdateInterval() time.Duration
• type RepoIniter
• type RepoUpdater
• type SnapshotKind
• type StreamConfig
• type TagsConfig
• type TmpRepoIniter
  • func (i *TmpRepoIniter) Init() (*repo.Repository, error)

Constants

const (
	KindFSSnapshot        SnapshotKind = "fs"
	KindGitCommitSnapshot SnapshotKind = "gc"

	AutoUploadNone AutoUploadDest = iota
	AutoUploadTags
	AutoUploadBundlestore
)

Functions

func CreateBundlestoreSnapshot

func CreateBundlestoreSnapshot(sha string, kind SnapshotKind, bundleKey, streamName string) snapshot

func Module

func Module() ice.Module

Module returns a module that supports typical GitDB usage

Types

type AutoUploadDest

type AutoUploadDest int

type BundlestoreConfig

type BundlestoreConfig struct {
	Store             store.Store
	AllowStreamUpdate bool
}

BundlestoreConfig defines how to talk to Bundlestore AllowStreamUpdate controls whether bundlestore download failures can fall back on stream updates

type DB

type DB struct {
	InitDoneCh chan error
	// contains filtered or unexported fields
}

DB stores its data in a Git Repo

func MakeDBFromRepo

func MakeDBFromRepo(
	dataRepo *repo.Repository,
	updater RepoUpdater,
	tmp string,
	stream *StreamConfig,
	tags *TagsConfig,
	bundles *BundlestoreConfig,
	autoUploadDest AutoUploadDest,
	stat stats.StatsReceiver) *DB

MakeDBFromRepo makes a gitdb.DB that uses dataRepo for data and tmp for temporary directories

func MakeDBNewRepo

func MakeDBNewRepo(
	initer RepoIniter,
	updater RepoUpdater,
	tmp string,
	stream *StreamConfig,
	tags *TagsConfig,
	bundles *BundlestoreConfig,
	autoUploadDest AutoUploadDest,
	stat stats.StatsReceiver) *DB

MakeDBNewRepo makes a gitDB that uses a new DB, populated by initer

func (*DB) Cancel

func (db *DB) Cancel() error

Unimplemented

func (*DB) Checkout

func (db *DB) Checkout(id snap.ID) (path string, err error)

Checkout puts the snapshot identified by id in the local filesystem, returning the path where it lives or an error.

func (*DB) Close

func (db *DB) Close()

Close stops the DB

func (*DB) ExportGitCommit

func (db *DB) ExportGitCommit(id snap.ID, exportRepo *repo.Repository) (string, error)

ExportGitCommit applies a snapshot to a repository, returning the sha of the exported commit or an error.

func (*DB) IDForStreamCommitSHA

func (db *DB) IDForStreamCommitSHA(streamName string, sha string) snap.ID

IDForStreamCommitSHA gets a SnapshotID from a string name and commit sha

func (*DB) IngestDir

func (db *DB) IngestDir(dir string) (snap.ID, error)

IngestDir ingests a directory directly.

func (*DB) IngestGitCommit

func (db *DB) IngestGitCommit(ingestRepo *repo.Repository, commitish string) (snap.ID, error)

IngestGitCommit ingests the commit identified by commitish from ingestRepo

func (*DB) IngestGitWorkingDir

func (db *DB) IngestGitWorkingDir(ingestRepo *repo.Repository) (snap.ID, error)

IngestGitWorkingDir ingests HEAD + working dir modifications from the ingestRepo.

func (*DB) ReadFileAll

func (db *DB) ReadFileAll(id snap.ID, path string) ([]byte, error)

ReadFileAll reads the contents of the file path in FSSnapshot ID, or errors

func (*DB) ReleaseCheckout

func (db *DB) ReleaseCheckout(path string) error

ReleaseCheckout releases a path from a previous Checkout. This allows Scoot to reuse the path. Scoot will not touch path after Checkout until ReleaseCheckout.

func (*DB) StreamName

func (db *DB) StreamName() string

func (*DB) Update

func (db *DB) Update() error

Update is used to trigger an underlying RepoUpdater to Update() the Repository. If db.updater is nil, has no effect.

func (*DB) UpdateInterval

func (db *DB) UpdateInterval() time.Duration

Returns RepoUpdater's interval if an updater exists, or a zero duration

func (*DB) UploadFile

func (db *DB) UploadFile(filePath string, ttl *store.TTLValue) (string, error)

Manual write of a file (like an existing bundle) to the underlying store Intended for HTTP-backed stores that implement bundlestore's TTL fields Base of the filePath will be used as bundle name Returns location of stored file

type NoopRepoUpdater

type NoopRepoUpdater struct{}

func (*NoopRepoUpdater) Update

func (u *NoopRepoUpdater) Update(*repo.Repository) error

func (*NoopRepoUpdater) UpdateInterval

func (u *NoopRepoUpdater) UpdateInterval() time.Duration

type RepoIniter

type RepoIniter interface {
	Init() (*repo.Repository, error)
}

Interface for defining initialization that results in a valid repo.Repository.

type RepoUpdater

type RepoUpdater interface {
	Update(r *repo.Repository) error
	UpdateInterval() time.Duration
}

Interface for defining update behavior for an underlying repo.Repository. Provides an update mechanism and an interval definition

type SnapshotKind

type SnapshotKind string

SnapshotKind describes the kind of a Snapshot: is it an FSSnapshot or a GitCommitSnapshot kind instead of type because type is a keyword

type StreamConfig

type StreamConfig struct {
	// Name (used in IDs (so it should be short)
	// e.g. sm for a Stream following Source (repo)'s Master (branch)
	//      name may include a ref string that'll override the refspec configured for 'sm'
	//      the full id might look like "stream-gc-sm:<branch>-<sha>" for a given branch and sha
	Name string

	// Remote to fetch from (e.g. https://github.com/twitter/scoot)
	Remote string

	// Name of ref to follow in data repo (e.g. refs/remotes/upstream/master)
	RefSpec string
}

A Stream is a sequence of GitCommitSnapshots that updates. The backend is a git refspec that can be fetched from a Git remote.

type TagsConfig

type TagsConfig struct {
	// Name (used in IDs (so it should be short)
	// e.g. sss for Source (repo)'s Scoot Snapshots
	Name string

	// Remote to fetch from and push to (e.g. https://github.com/twitter/scoot)
	Remote string

	// Prefix for tag names (e.g. reserved_scoot)
	Prefix string
}

TagsConfig specifies how GitDB should store Snapshots as tags in another git repo

type TmpRepoIniter

type TmpRepoIniter struct {
	// contains filtered or unexported fields
}

TmpRepoIniter creates a new Repo in a temp dir

func (*TmpRepoIniter) Init

func (i *TmpRepoIniter) Init() (*repo.Repository, error)

Init creates a new temp dir and a repo in it