base

package
v0.9.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 1, 2020 License: GPL-3.0 Imports: 32 Imported by: 1

Documentation

Overview

Package base defines business that operates on local data. it's main job is to composing APIs from the lower half of our tech stack, providing uniform functions for higher up packages, mainly p2p and lib. p2p and lib use base as the only way of operate on the local repo Here's some ascii art to clarify the stack: 

┌───────────────┐ ┌───────────────┐
│      cmd      │ │     api       │
└───────────────┘ └───────────────┘
┌─────────────────────────────────┐
│               lib               │
└─────────────────────────────────┘
┌───────────────────────┐
│          p2p          │
└───────────────────────┘
┌─────────────────────────────────┐
│              base               │  <-- you are here
└─────────────────────────────────┘
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│ repo │ │ dsfs │ │ qfs  │ │ ...  │
└──────┘ └──────┘ └──────┘ └──────┘

There are packages omitted from this diagram, but these are the vitals. base functions mainly work with repo.Repo instances, using repo interface methods and related packages to do their work. This is part of a larger pattern of having lib rely on lower level interfaces wherever possible to enhance configurability

Index

Constants

View Source 
const MaxNumDatasetRowsInPreview = 100

MaxNumDatasetRowsInPreview is the highest number of rows a dataset preview can contain

View Source 
const TimeoutDuration = 100 * time.Millisecond

TimeoutDuration is the duration allowed for a datasetLog lookup before it times out

Variables

View Source 
var DefaultTemplate = `` /* 249-byte string literal not displayed */

DefaultTemplate is the template that render will fall back to should no template be available

View Source 
var ErrDatasetLogTimeout = fmt.Errorf("datasetLog: timeout")

ErrDatasetLogTimeout is an error for when getting the datasetLog times out

View Source 
var ErrNameTaken = fmt.Errorf("name already in use")

ErrNameTaken is an error for when a name for a new dataset is already being used

View Source 
var (

	// OpenFileTimeoutDuration determines the maximium amount of time to wait for
	// a Filestore to open a file. Some filestores (like IPFS) fallback to a
	// network request when it can't find a file locally. Setting a short timeout
	// prevents waiting for a slow network response, at the expense of leaving
	// files unresolved.
	// TODO (b5) - allow -1 duration as a sentinel value for no timeout
	OpenFileTimeoutDuration = time.Millisecond * 250
)

Functions

func ApplyPath 

func ApplyPath(ds *dataset.Dataset, path string) (interface{}, error)

ApplyPath gets a dataset value by applying a case.Sensitve.dot.separated.path ApplyPath cannot select file fields

func CloseDataset added in v0.7.0 

func CloseDataset(ds *dataset.Dataset) (err error)

CloseDataset ensures all open dataset files are closed

func ConvertBodyFile added in v0.7.2 

func ConvertBodyFile(file qfs.File, in, out *dataset.Structure, limit, offset int, all bool) (data []byte, err error)

ConvertBodyFile takes an input file & structure, and converts a specified selection to the structure specified by out

func ConvertBodyFormat added in v0.6.1 

func ConvertBodyFormat(bodyFile qfs.File, fromSt, toSt *dataset.Structure) (qfs.File, error)

ConvertBodyFormat rewrites a body from a source format to a destination format. TODO (b5): Combine this with ConvertBodyFile, update callers.

func CreateDataset 

func CreateDataset(ctx context.Context, r repo.Repo, writeDest qfs.Filesystem, ds, dsPrev *dataset.Dataset, sw SaveSwitches) (ref reporef.DatasetRef, err error)

CreateDataset uses dsfs to add a dataset to a repo's store, updating the refstore

func CreatePreview added in v0.9.5 

func CreatePreview(ctx context.Context, r repo.Repo, ref dsref.Ref) (ds *dataset.Dataset, err error)

CreatePreview generates a preview for a dataset version

func DatasetNameExists added in v0.9.9 

func DatasetNameExists(r repo.Repo, dsName string) bool

DatasetNameExists determines whether the name exists in the repository TODO(dustmop): Add dscache support

func Drop added in v0.9.9 

func Drop(ds *dataset.Dataset, revStr string) error

Drop sets named components to nil from a revision string

func FetchDataset 

func FetchDataset(ctx context.Context, r repo.Repo, ref *reporef.DatasetRef, pin, load bool) (err error)

FetchDataset grabs a dataset from a remote source

func FinalizeNameAndStableIdentifers added in v0.9.9 

func FinalizeNameAndStableIdentifers(ctx context.Context, r repo.Repo, peername string, dsName string, ds *dataset.Dataset, newName bool) (dsref.Ref, error)

FinalizeNameAndStableIdentifers determines the final name for the dataset, by inferring one if necessary, and returns a ref with stable identifiers for the full dataset history, and for the most recent version.

func GenerateAvailableName added in v0.9.3 

func GenerateAvailableName(r repo.Repo, prefix string) string

GenerateAvailableName creates a name for the dataset that is not currently in use

func InLocalNamespace 

func InLocalNamespace(r repo.Repo, ref *reporef.DatasetRef) bool

InLocalNamespace checks if a dataset ref is local, assumes the reference is already resolved

func InferValues 

func InferValues(pro *profile.Profile, ds *dataset.Dataset) error

InferValues populates any missing fields that must exist to create a snapshot

func InlineJSONBody added in v0.7.2 

func InlineJSONBody(ds *dataset.Dataset) error

InlineJSONBody reads the contents dataset.BodyFile() into a json.RawMessage, assigning the result to dataset.Body

func ListDatasets 

func ListDatasets(ctx context.Context, r repo.Repo, term string, limit, offset int, RPC, publishedOnly, showVersions bool) (res []reporef.DatasetRef, err error)

ListDatasets lists datasets from a repo

func LoadRevs 

func LoadRevs(ctx context.Context, r repo.Repo, ref reporef.DatasetRef, revs []*dsref.Rev) (res *dataset.Dataset, err error)

LoadRevs grabs a component of a dataset that exists <n>th generation ancestor of the referenced version, where presence of a component in a previous snapshot constitutes ancestry

func MaybeAddDefaultViz added in v0.8.0 

func MaybeAddDefaultViz(ds *dataset.Dataset)

MaybeAddDefaultViz sets a dataset viz component and template if none exists TODO(dlong): This is deprecated and should be removed.

func MaybeInferName added in v0.9.3 

func MaybeInferName(ds *dataset.Dataset) string

MaybeInferName infer a name for the dataset if none is set

func ModifyRepoUsername added in v0.9.4 

func ModifyRepoUsername(ctx context.Context, r repo.Repo, book *logbook.Book, from, to string) error

ModifyRepoUsername performs all tasks necessary to switch a username (formerly: peername) for a local repo, and must be called when a username is changed TODO (b5) - make this transactional

func NewLocalDatasetLoader added in v0.9.9 

func NewLocalDatasetLoader(r repo.Repo) dsref.Loader

NewLocalDatasetLoader creates a dsfs.Loader that operates on a filestore

func OpenDataset added in v0.7.0 

func OpenDataset(ctx context.Context, fsys qfs.Filesystem, ds *dataset.Dataset) (err error)

OpenDataset prepares a dataset for use, checking each component for populated Path or Byte suffixed fields, consuming those fields to set File handlers that are ready for reading

func PinDataset 

func PinDataset(ctx context.Context, r repo.Repo, ref reporef.DatasetRef) error

PinDataset marks a dataset for retention in a store

func PrepareHeadDatasetVersion added in v0.9.9 

func PrepareHeadDatasetVersion(ctx context.Context, r repo.Repo, peername, name string) (curr, mutable *dataset.Dataset, currPath string, err error)

PrepareHeadDatasetVersion prepares to save by loading the head commit, opening the body file, and constructing a mutable version that has no transform or commit.

func RawDatasetRefs added in v0.9.3 

func RawDatasetRefs(ctx context.Context, r repo.Repo) (string, error)

RawDatasetRefs converts the dataset refs to a string

func ReadBody added in v0.9.1 

func ReadBody(ds *dataset.Dataset, format dataset.DataFormat, fcfg dataset.FormatConfig, limit, offset int, all bool) (data []byte, err error)

ReadBody grabs some or all of a dataset's body, writing an output in the desired format

func ReadDataset 

func ReadDataset(ctx context.Context, r repo.Repo, path string) (ds *dataset.Dataset, err error)

ReadDataset grabs a dataset from the store

Deprecated - use LoadDataset instead

func ReadEntries added in v0.7.1 

func ReadEntries(reader dsio.EntryReader) (interface{}, error)

ReadEntries reads entries and returns them as a native go array or map

func Recall added in v0.9.1 

func Recall(ctx context.Context, r repo.Repo, str string, ref reporef.DatasetRef) (*dataset.Dataset, error)

Recall loads revisions of a dataset from history

func RemoveEntireDataset added in v0.9.5 

func RemoveEntireDataset(ctx context.Context, r repo.Repo, ref dsref.Ref, history []DatasetLogItem) (didRemove string, removeErr error)

RemoveEntireDataset removes all of the information in the repository about a dataset. It will continue on even if some error occurs, returning a comma-separated list of what locations data was removed from, as well the last error that occured, if any. Note that in particular, FSI is not handled at all by this function. Callers should also call any relevent FSI operations.

func RemoveNVersionsFromStore added in v0.9.1 

func RemoveNVersionsFromStore(ctx context.Context, r repo.Repo, curr dsref.Ref, n int) (*dsref.VersionInfo, error)

RemoveNVersionsFromStore removes n versions of a dataset from the store starting with the most recent version when n == -1, remove all versions does not remove the dataset reference

func RenameDatasetRef added in v0.9.9 

func RenameDatasetRef(ctx context.Context, r repo.Repo, ref dsref.Ref, newName string) (*dsref.VersionInfo, error)

RenameDatasetRef changes a dataset's pretty name by modifying the ref in the repository refstore, and returns a versionInfo describing the resulting dataset reference.

func Render 

func Render(ctx context.Context, r repo.Repo, ref reporef.DatasetRef, tmplData []byte) ([]byte, error)

Render executes a template for a dataset, returning a slice of HTML Render uses go's html/template package to generate html documents from an input dataset. It's API has been adjusted to use lowerCamelCase instead of UpperCamelCase naming conventions

func RenderReadme added in v0.9.1 

func RenderReadme(ctx context.Context, file qfs.File) (string, error)

RenderReadme converts the markdown from the file into html.

func ReplaceRefIfMoreRecent added in v0.9.1 

func ReplaceRefIfMoreRecent(r repo.Repo, prev, curr *reporef.DatasetRef) error

ReplaceRefIfMoreRecent replaces the given ref in the ref store, if it is more recent then the ref currently in the refstore

func RewindDatasetRef added in v0.9.9 

func RewindDatasetRef(ctx context.Context, r repo.Repo, curr, next dsref.Ref) (*dsref.VersionInfo, error)

RewindDatasetRef changes a dsref in the repository refstore, by setting the path back to an older value, and returns a versionInfo describing the resulting dataset reference.

func SaveDataset added in v0.9.1 

func SaveDataset(ctx context.Context, r repo.Repo, writeDest qfs.Filesystem, initID, prevPath string, changes *dataset.Dataset, sw SaveSwitches) (ref reporef.DatasetRef, err error)

SaveDataset saves a version of the dataset for the given initID at the current path

func Select 

func Select(ctx context.Context, r repo.Repo, ref reporef.DatasetRef, path string) (interface{}, error)

Select loads a dataset value specified by case.Sensitve.dot.separated.paths

func SetPublishStatus 

func SetPublishStatus(r repo.Repo, ref *reporef.DatasetRef, published bool) error

SetPublishStatus updates the Published field of a dataset ref

func StoredHistoricalDatasets added in v0.9.9 

func StoredHistoricalDatasets(ctx context.Context, r repo.Repo, headPath string, offset, limit int, loadDatasets bool) (log []*dataset.Dataset, err error)

StoredHistoricalDatasets fetches the history of changes to a dataset by walking backwards through dataset commits. if loadDatasets is true, dataset information will be populated

func TransformApply added in v0.9.9 

func TransformApply(
	ctx context.Context,
	ds *dataset.Dataset,
	r repo.Repo,
	loader dsref.ParseResolveLoad,
	str ioes.IOStreams,
	scriptOut io.Writer,
	secrets map[string]string,
) error

TransformApply applies the transform script to order to modify the changing dataset

func UnpinDataset 

func UnpinDataset(ctx context.Context, r repo.Repo, ref reporef.DatasetRef) error

UnpinDataset unmarks a dataset for retention in a store

func Validate added in v0.9.1 

func Validate(ctx context.Context, r repo.Repo, body qfs.File, st *dataset.Structure) ([]jsonschema.KeyError, error)

Validate checks a dataset body for errors based on the structure's schema

func ValidateDataset 

func ValidateDataset(ds *dataset.Dataset) (err error)

ValidateDataset checks that a dataset is semantically valid

Types

type DatasetLogItem added in v0.9.1 

type DatasetLogItem = logbook.DatasetLogItem

DatasetLogItem aliases the type from logbook

func DatasetLog 

func DatasetLog(ctx context.Context, r repo.Repo, ref dsref.Ref, limit, offset int, loadDatasets bool) ([]DatasetLogItem, error)

DatasetLog fetches the change version history of a dataset

type SaveSwitches added in v0.9.8 

type SaveSwitches = dsfs.SaveSwitches

SaveSwitches is an alias for the switches that control how saves happen

Source Files

View all Source files

Directories

Path Synopsis
Package archive creates and consumes high-fidelity conversions of dataset documents for export & import
 Package archive creates and consumes high-fidelity conversions of dataset documents for export & import
Package dsfs glues datsets to cafs (content-addressed-file-system)
 Package dsfs glues datsets to cafs (content-addressed-file-system)
dstest
Package dstest defines an interface for reading test cases from static files leveraging directories of test dataset input files & expected output files
 Package dstest defines an interface for reading test cases from static files leveraging directories of test dataset input files & expected output files
Package fill assigns arbitrary values to struct fields using reflection.
 Package fill assigns arbitrary values to struct fields using reflection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.