core

package
v0.0.0-...-f50264a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 27, 2024 License: BSD-2-Clause Imports: 45 Imported by: 0

Documentation

Index

Constants

View Source 
const MaxErrors = 30

MaxErrors is the maximum number of errors the validator will collect before quitting and returning the error list. We don't quit at the first error because when developing a bagging process, multiple errors are common and we don't want to make depositors have to rebag constantly just to see the next error. We also try to be very specific with error messages so depositors know exactly what to fix.

Variables

View Source 
var AppSettingSettableFields = []string{"Value"}
View Source 
var ErrFileMissingFromBag = fmt.Errorf("file is missing from bag")

ErrFileMissingFromBag indicates that a file present in a payload manifest is not present in the bag's data directory.

View Source 
var ErrInvalidTagValue = errors.New("invalid tag value")
View Source 
var ErrTagNotFound = errors.New("tag not found")
View Source 
var RemoteRepositorySettableFields = []string{"APIToken", "LoginExtra", "Name", "Url", "UserID"}
View Source 
var StorageServiceSettableFields = []string{"AllowsDownload", "AllowsUpload", "Bucket", "Host", "Login", "LoginExtra", "Name", "Password", "Port", "Protocol"}
View Source 
var TagsSetBySystem = []string{
	"Bagging-Date",
	"Bagging-Software",
	"Payload-Oxum",
	"Bag-Size",
	"BagIt-Profile-Identifier",
}
View Source 
var TitleTags = []string{
	"Title",
	"Internal-Sender-Identifier",
	"External-Identifier",
	"Internal-Sender-Description",
	"External-Description",
	"Description",
}

TitleTags is a list of BagItProfile tags to check to try to find a meaningful title for this job. DART checks them in order and returns the first one that has a non-empty user-defined value.

Functions

func ArtifactDelete 

func ArtifactDelete(uuid string) error

func ArtifactSave 

func ArtifactSave(a *Artifact) error

func ArtifactsDeleteByJobID 

func ArtifactsDeleteByJobID(jobID string) error

ArtifactsDeleteByJobID deletes all artifacts associated with JobID.

func ClearArtifactsTable 

func ClearArtifactsTable() error

ClearArtifactsTable is for testing use only

func ClearDartTable 

func ClearDartTable() error

ClearDartTable is for testing use only

func DataFilePath 

func DataFilePath() string

func FindConflictingUUID 

func FindConflictingUUID(obj PersistentObject) string

FindConflictingUUID returns the UUID of the object having the same name and type as obj. The dart table has a unique constraint on obj_type + obj_name. That should prevent inserts that conflict with the constraint, but it doesn't. The modernc sqlite driver does not report an error on this conflict. It fails silently, So we check for the conflict on our own with this function.

If this returns a UUID, there's a conflict, and we can't do the insert. If it returns an empty string, we're fine, and the insert can proceed.

func GetAppSetting 

func GetAppSetting(name string) (string, error)

GetAppSetting returns the value of the AppSetting with the given name.

func GuessProfileType 

func GuessProfileType(obj map[string]interface{}) string

GuessProfileType tries to determine the type of a BagIt profile based on its structure.

func GuessProfileTypeFromJson 

func GuessProfileTypeFromJson(jsonBytes []byte) (string, error)

GuessProfileTypeFromJson tries to determine the type of a BagIt profile based on its structure.

func InitDBForFirstUse 

func InitDBForFirstUse()

func InitSchema 

func InitSchema() error

func LogFilePath 

func LogFilePath() string

func ObjCount 

func ObjCount(objType string) (int, error)

func ObjDelete 

func ObjDelete(obj PersistentObject) error

func ObjExists 

func ObjExists(objId string) (bool, error)

func ObjSave 

func ObjSave(obj PersistentObject) error

ObjSave validates an object and then saves it if it passes validation. If the object is invalid, this will return constants.ErrObjecValidation and you can get a map of specific validation errors from obj.Errors.

In rare cases, this may return constants.ErrUniqueConstraint, which means the database already contains a different object with the same UUID.

func ObjSaveWithoutValidation 

func ObjSaveWithoutValidation(obj PersistentObject) error

ObjSaveWithoutValidation saves an object without validating it first. This is useful when creating new Jobs through the UI, since the user must adjust settings on a number of screens to build up a fully valid Job.

func ParseManifest 

func ParseManifest(reader io.Reader) (map[string]string, error)

ParseManifest parses a checksum manifest, returning a slice of Checksums. Param reader should be an open reader. Returns a map in which file paths are keys and digests are values.

func ReadInput 

func ReadInput(reader io.Reader) []byte

ReadInput reads input from the specified reader (which, in practice, will usually be STDIN) and returns it with newlines preserved.

func RegisterRepoClient 

func RegisterRepoClient(name, id string, constructor func(*RemoteRepository) RemoteRepoClient)

RegisterRepoClient registers a remote repo client, so that DART knows it's available.

func RunJob 

func RunJob(job *Job, deleteOnSuccess, printOutput bool) int

func RunJobWithMessageChannel 

func RunJobWithMessageChannel(job *Job, deleteOnSuccess bool, messageChannel chan *EventMessage) int

RunJobWithMessageChannel runs a job and pumps progress details into the messageChannel. That channel passes messages back to the front end UI. This returns the job's exit code, where zero means success and non-zero is failure. See the exit codes defined in constants for more info.

func SFTPConnect 

func SFTPConnect(ss *StorageService) (*sftp.Client, error)

SFTPConnect returns an sftp connection or an error. If the StorageService specifies LoginExtra, this will treat that as the path to the SSH key; otherwise, it will use ss.Password as the password. In either case, it uses ss.Login as the user name.

func SFTPUpload 

func SFTPUpload(ss *StorageService, localPath string, uploadProgress *StreamProgress) (int64, error)

SFTPUpload uploads a file to the SFTP server described in the StorageService param. localPath is the path to the local file that you want to upload to the remote server. The uploadProgress param should be nil except when running a job from the UI. For jobs launched from the UI, the uploadProgress object will pass progress info back to the front end. Be sure to set uploadProgress.Total to the size of the file and make sure the MessageChannel is initialized.

func SSHConnect 

func SSHConnect(ss *StorageService) (*goph.Client, error)

SSHConnect returns an ssh connection or an error. If the StorageService specifies LoginExtra, this will treat that as the path to the SSH key; otherwise, it will use ss.Password as the password. In either case, it uses ss.Login as the user name.

func StdinHasData 

func StdinHasData() bool

StdinHasData returns true if STDIN is a pipe with data waiting to be read. This exits immediately if it can't access or stat STDIN.

Types

type APTrustClientV3 

type APTrustClientV3 struct {
	// contains filtered or unexported fields
}

APTrustClientV3 is a remote repository client that can talk to the APTrust Registry API.

func NewAPTrustClientV3 

func NewAPTrustClientV3(repo *RemoteRepository) *APTrustClientV3

NewAPTrustClientV3 returns a new instance of the APTrust remote repo client.

func (*APTrustClientV3) APIVersion 

func (client *APTrustClientV3) APIVersion() string

APIVersion returns the version number of the API that this client can talk to.

func (*APTrustClientV3) AvailableHTMLReports 

func (client *APTrustClientV3) AvailableHTMLReports() []util.NameValuePair

AvailableHTMLReports returns a list of available HTML report names. In the NameValuePair, Name is the name of the report and Value is a description.

func (*APTrustClientV3) Description 

func (client *APTrustClientV3) Description() string

Description returns a short description of this client.

func (*APTrustClientV3) ID 

func (client *APTrustClientV3) ID() string

ID returns this client's UUID.

func (*APTrustClientV3) Name 

func (client *APTrustClientV3) Name() string

Name returns the client name.

func (*APTrustClientV3) RunHTMLReport 

func (client *APTrustClientV3) RunHTMLReport(name string) (string, error)

RunHTMLReport runs the named report and returns HTML suitable for display on the DART dashboard. For a list of available report names, call AvailableHTMLReports().

func (*APTrustClientV3) TestConnection 

func (client *APTrustClientV3) TestConnection() error

TestConnection tests a connection to the remote repo. It returns true or false to describe whether the connection succeeded. Check the error if the connection did not succeed.

type AppSetting 

type AppSetting struct {
	ID            string            `json:"id"`
	Name          string            `json:"name"`
	Value         string            `json:"value"`
	Help          string            `json:"help"`
	Errors        map[string]string `json:"errors"`
	UserCanDelete bool              `json:"userCanDelete"`
}

AppSetting represents an application-wide setting that can be configured by the user. For example, the bagging directory into which DART writes new bags.

Field names for JSON serialization match the old DART 2 names, so we don't break legacy installations.

func CreateAppSettings 

func CreateAppSettings(howMany int) ([]*AppSetting, error)

func NewAppSetting 

func NewAppSetting(name, value string) *AppSetting

NewAppSetting creates a new AppSetting with the specified name and value. UserCanDelete will be true by default. If a setting is required for DART to function properly (such as the Bagging Directory setting), set UserCanDelete to false.

func (*AppSetting) GetErrors 

func (setting *AppSetting) GetErrors() map[string]string

func (*AppSetting) IsDeletable 

func (setting *AppSetting) IsDeletable() bool

func (*AppSetting) ObjID 

func (setting *AppSetting) ObjID() string

ObjID returns this setting's object id (uuid).

func (*AppSetting) ObjName 

func (setting *AppSetting) ObjName() string

ObjName returns this object's name, so names will be searchable and sortable in the DB.

func (*AppSetting) ObjType 

func (setting *AppSetting) ObjType() string

ObjType returns this object's type name.

func (*AppSetting) String 

func (setting *AppSetting) String() string

func (*AppSetting) ToForm 

func (setting *AppSetting) ToForm() *Form

ToForm returns a form so the user can edit this AppSetting. The form can be rendered by the app_setting/form.html template.

func (*AppSetting) Validate 

func (setting *AppSetting) Validate() bool

Validate validates this setting, returning true if it's valid, false if not. If false, this sets specific error messages in the Errors map, which are suitable for display on the form.

type Artifact 

type Artifact struct {
	ID        string
	JobID     string
	BagName   string
	ItemType  string // File or JobResult
	FileName  string // name of manifest or tag file
	FileType  string // manifest or tag file
	RawData   string // file content or work result json
	UpdatedAt time.Time
}

func ArtifactFind 

func ArtifactFind(uuid string) (*Artifact, error)

func ArtifactListByJobID 

func ArtifactListByJobID(jobID string) ([]*Artifact, error)

func ArtifactListByJobName 

func ArtifactListByJobName(bagName string) ([]*Artifact, error)

func NewArtifact 

func NewArtifact() *Artifact

NewArtifact creates a new empty Artifact with a unique id and timestamp.

func NewJobResultArtifact 

func NewJobResultArtifact(bagName string, jobResult *JobResult) *Artifact

NewJobResultArtifact creates a new Artifact to store a JobResult.

func NewManifestArtifact 

func NewManifestArtifact(bagName, jobID, manifestName, manifestContent string) *Artifact

NewManifestArtifact creates a new Artifact to store a bag's payload manifest.

func NewTagFileArtifact 

func NewTagFileArtifact(bagName, jobID, tagFileName, tagFileContent string) *Artifact

NewTagFileArtifact creates a new Artifact to store a bag's tag file.

type BagItProfile 

type BagItProfile struct {
	ID                   string            `json:"id"`
	AcceptBagItVersion   []string          `json:"acceptBagItVersion"`
	AcceptSerialization  []string          `json:"acceptSerialization"`
	AllowFetchTxt        bool              `json:"allowFetchTxt"`
	BagItProfileInfo     ProfileInfo       `json:"bagItProfileInfo"`
	BaseProfileID        string            `json:"baseProfileId"`
	Description          string            `json:"description"`
	Errors               map[string]string `json:"-"`
	IsBuiltIn            bool              `json:"isBuiltIn"`
	ManifestsAllowed     []string          `json:"manifestsAllowed"`
	ManifestsRequired    []string          `json:"manifestsRequired"`
	Name                 string            `json:"name"`
	Serialization        string            `json:"serialization"`
	TagFilesAllowed      []string          `json:"tagFilesAllowed"`
	TagFilesRequired     []string          `json:"tagFilesRequired"`
	TagManifestsAllowed  []string          `json:"tagManifestsAllowed"`
	TagManifestsRequired []string          `json:"tagManifestsRequired"`
	Tags                 []*TagDefinition  `json:"tags"`
	TarDirMustMatchName  bool              `json:"tarDirMustMatchName"`
}

BagItProfile represents a DART-type BagItProfile, as described at https://aptrust.github.io/dart/BagItProfile.html. This format differs slightly from the profiles at https://github.com/bagit-profiles/bagit-profiles-specification. The DART specification is richer and can describe requirements that the other profile format cannot. DART can convert between the two formats as described in https://aptrust.github.io/dart-docs/users/bagit/importing/ and https://aptrust.github.io/dart-docs/users/bagit/exporting/.

func BagItProfileClone 

func BagItProfileClone(p *BagItProfile) *BagItProfile

func BagItProfileFromJSON 

func BagItProfileFromJSON(jsonData string) (*BagItProfile, error)

BagItProfileFromJSON converts a JSON representation of a BagIt Profile to a Profile object.

func BagItProfileLoad 

func BagItProfileLoad(filename string) (*BagItProfile, error)

BagItProfileLoad loads a BagIt Profile from the specified file.

func ConvertProfile 

func ConvertProfile(jsonBytes []byte, sourceUrl string) (*BagItProfile, error)

ConvertProfile converts a BagIt profile from a known format to a DART BagIt profile. It does not try to save the profile, because some imported profiles may not have all required info, and that will cause an error. User should be able to convert the profile and then edit it to correct missing or invalid properties. Accepted profile formats include Dart, Standard, LOC Ordered, and LOC Unordered.

func CreateBagItProfiles 

func CreateBagItProfiles(howMany int) ([]*BagItProfile, error)

func NewBagItProfile 

func NewBagItProfile() *BagItProfile

func ProfileFromLOCOrdered 

func ProfileFromLOCOrdered(jsonBytes []byte, sourceUrl string) (*BagItProfile, error)

ProfileFromLOCOrdered converts the JSON representation of an ordered Library of Congress BagIt profile to a DART BagIt profile.

func ProfileFromLOCUnordered 

func ProfileFromLOCUnordered(jsonBytes []byte, sourceUrl string) (*BagItProfile, error)

ProfileFromLOCUnordered converts the JSON representation of an unordered Library of Congress BagIt profile to a DART BagIt profile.

func (*BagItProfile) EnsureMinimumRequirements 

func (p *BagItProfile) EnsureMinimumRequirements()

EnsureMinimumRequirements creates the minimum required attributes for this profile to pass validation, if those attributes are not already present. This function is required to flesh out some barebones profiles that user may import from elsewhere. For example, traditional Library of Congress profiles do not include any tag info for the bagit.txt file, nor do they include info about required or allowed manifest algorithms.

func (*BagItProfile) FindMatchingTags 

func (p *BagItProfile) FindMatchingTags(property, value string) ([]*TagDefinition, error)

func (*BagItProfile) FirstMatchingTag 

func (p *BagItProfile) FirstMatchingTag(property, value string) (*TagDefinition, error)

func (*BagItProfile) FlagUserAddedTagFiles 

func (p *BagItProfile) FlagUserAddedTagFiles()

FlagUserAddedTagFiles sets IsUserAddedFile to true on tag definitions where the user added the tag file to the profile. The user can do this on a per-job basis through the jobs/metadata UI page and through the BagIt Profile editor.

func (*BagItProfile) GetErrors 

func (p *BagItProfile) GetErrors() map[string]string

func (*BagItProfile) GetTagByFullyQualifiedName 

func (p *BagItProfile) GetTagByFullyQualifiedName(fullyQualifiedName string) *TagDefinition

func (*BagItProfile) GetTagDef 

func (p *BagItProfile) GetTagDef(tagFile, tagName string) *TagDefinition

GetTagDef returns the TagDefinition for the specified tag file and tag name. Note: BagIt spec section 2.2.2 says tag names are case-insensitive. https://tools.ietf.org/html/rfc8493#section-2.2.2

func (*BagItProfile) GetTagFileContents 

func (p *BagItProfile) GetTagFileContents(tagFileName string) (string, error)

GetTagFileContents returns the generated contents of the specified tag file.

func (*BagItProfile) HasTagFile 

func (p *BagItProfile) HasTagFile(name string) bool

func (*BagItProfile) IsDeletable 

func (p *BagItProfile) IsDeletable() bool

func (*BagItProfile) ObjID 

func (p *BagItProfile) ObjID() string

func (*BagItProfile) ObjName 

func (p *BagItProfile) ObjName() string

func (*BagItProfile) ObjType 

func (p *BagItProfile) ObjType() string

func (*BagItProfile) SetTagValue 

func (p *BagItProfile) SetTagValue(tagFile, tagName, value string)

SetTagValue sets the value of the specified tag in the specified file. It creates the tag if it doesn't already exist in the profile. This currently supports only one instance of each tag in each file.

func (*BagItProfile) String 

func (p *BagItProfile) String() string

func (*BagItProfile) TagFileNames 

func (p *BagItProfile) TagFileNames() []string

TagFileNames returns the names of the tag files for which we have actual tag definitions. The bag may require other tag files, but we can't produce them if we don't have tag defs. The list will include user-added tag files.

func (*BagItProfile) TagsInFile 

func (p *BagItProfile) TagsInFile(tagFileName string) []*TagDefinition

TagsInFile returns all of the tag definition objects in tagFileName, sorted by name. This is equivalent to calling FindMatchingTags("TagFile", tagFileName), except it returns results in sorted order.

func (*BagItProfile) ToForm 

func (p *BagItProfile) ToForm() *Form

func (*BagItProfile) ToJSON 

func (p *BagItProfile) ToJSON() (string, error)

ToJSON returns a JSON representation of this object.

func (*BagItProfile) ToStandardFormat 

func (p *BagItProfile) ToStandardFormat() *StandardProfile

ToStandardFormat converts this BagIt Profile to a version 1.3.0 BagIt Profile. See https://bagit-profiles.github.io/bagit-profiles-specification/ and https://github.com/bagit-profiles/bagit-profiles-specification

func (*BagItProfile) Validate 

func (p *BagItProfile) Validate() bool

Validate returns true if this profile is valid. This is not to be confused with bag validation. We're just making sure the profile itself is complete and makes sense.

type BagItProfileImport 

type BagItProfileImport struct {
	ImportSource string
	URL          string
	JsonData     []byte
	Errors       map[string]string
}

BagItProfileImport represents a URL or JSON blob to be imported as a DART BagIt profile.

func NewBagItProfileImport 

func NewBagItProfileImport(importSource, sourceUrl string, jsonData []byte) *BagItProfileImport

NewBagItProfileImport creates a new BagItProfileImport object. Param importSource should be either constants.ImportSourceUrl or constants.ImportSourceJson. If the source is a url, we'll fetch the JSON from the sourceUrl. Otherwise, we'll use jsonData. You only need to supply one or the other of the last two params.

func (*BagItProfileImport) Convert 

func (bpi *BagItProfileImport) Convert() (*BagItProfile, error)

Convert converts the JSON data provided by the user to a DART-style BagIt profile.

func (*BagItProfileImport) ToForm 

func (bpi *BagItProfileImport) ToForm() *Form

ToForm returns a form for creating a BagItProfile import job.

func (*BagItProfileImport) Validate 

func (bpi *BagItProfileImport) Validate() bool

Validate returns true if the object is valid, false if not. If this returns false, specific errors will be recorded in BagItProfileImport.Errors.

type BagReader 

type BagReader interface {
	// ScanMetadata scans and parses the tag files and manifests.
	// We do this first so we know which checksums to calculate when we
	// scan the payload later. Many BagIt profiles allow five or six
	// different manifest algorithms, but the bag may contain only one
	// or two from the allowed list. We won't know what they are until
	// we look.
	//
	// If you want to do a quick validation, you can scan the metadata
	// and then compare the Validator's oxum against the one in bag-info.txt.
	// If that's bad, you can skip the expensive payload scan.
	ScanMetadata() error

	// ScanPayload calculates checksums on the payload and tag files.
	ScanPayload() error

	// Close closes the underlying reader.
	Close()
}

BagReader describes the interface that bag readers must implement, whether reading from a file system, tar file, zip file, etc.

type BagWriter 

type BagWriter interface {
	// Open opens the writer.
	Open() error

	// AddFile adds a file to the bag. The returned map has digest
	// alg names for keys and digests for values. For example,
	// checksums["md5"] = "0987654321".
	AddFile(*util.ExtendedFileInfo, string) (map[string]string, error)

	// DigestAlgs returns a list of digest algoritms that the
	// writer calculates as it writes. E.g. ["md5", "sha256"].
	// These are defined in the contants package.
	DigestAlgs() []string

	// Close closes the underlying writer, flushing remaining data
	// as necessary.
	Close() error
}

BagWriter describes the interface that bag writers must implement, whether writing to a file system, tar file, zip file, etc.

type Bagger 

type Bagger struct {
	Profile           *BagItProfile
	OutputPath        string
	FilesToBag        []*util.ExtendedFileInfo
	Errors            map[string]string
	MessageChannel    chan *EventMessage
	PayloadFiles      *FileMap
	PayloadManifests  *FileMap
	TagFiles          *FileMap
	TagManifests      *FileMap
	ManifestArtifacts map[string]string
	TagFileArtifacts  map[string]string
	// contains filtered or unexported fields
}

func NewBagger 

func NewBagger(outputPath string, profile *BagItProfile, filesToBag []*util.ExtendedFileInfo) *Bagger

func (*Bagger) ArtifactsDir 

func (b *Bagger) ArtifactsDir() string

ArtifactsDir returns the name of the directory in which the bagger will leave artifacts, including manifests and tag files. These items will remain after bagging is complete.

Note that the bagger does not create or populate this directory. That happens in JobRunner.saveArtifactsToFileSystem, and it happens only when DART is running in command-line mode. Otherwise, we save artifacts to the DB.

func (*Bagger) GetTotalFilesBagged 

func (b *Bagger) GetTotalFilesBagged() int64

GetTotalFilesBagged returns the total number of files bagged, including payload files, tag files, and manifests. This will be zero if the bagger has not run yet, so call this after calling Run() to get an accurate number.

func (*Bagger) PayloadBytes 

func (b *Bagger) PayloadBytes() int64

func (*Bagger) PayloadFileCount 

func (b *Bagger) PayloadFileCount() int64

func (*Bagger) PayloadOxum 

func (b *Bagger) PayloadOxum() string

func (*Bagger) Run 

func (b *Bagger) Run() bool

Run builds the bag and returns the number of files bagged.

type CSVBatchParser 

type CSVBatchParser struct {
	PathToCSVFile string
	Workflow      *Workflow
}

func NewCSVBatchParser 

func NewCSVBatchParser(pathToCSVFile string, workflow *Workflow) *CSVBatchParser

NewCSVBatchParser creates a new parser to convert records in a CSV batch file to a slice of JobParams. Param pathToCSVFile is the path the CSV file you want to parse. Param workfow is the workflow that you want to apply to all items in that CSV file.

func (*CSVBatchParser) ParseAll 

func (p *CSVBatchParser) ParseAll(outputDir string) ([]*JobParams, error)

ParseAll converts the records in a CSV batch file to a slice of JobParams objects. Param outputDir will be passed through to each JobParams object, describing the directory in which to create bags. Unless you have some special reason, outputDir should be set to the value of the built-in app setting called "Bagging Directory". You can get that with a call to core.GetAppSetting("Bagging Directory").

type Checksum 

type Checksum struct {
	Source    string
	Algorithm string
	Digest    string
}

Checksum records information about a file's checksum.

func NewChecksum 

func NewChecksum(source, alg, digest string) *Checksum

func (*Checksum) SourceName 

func (cs *Checksum) SourceName() string

type Choice 

type Choice struct {
	Label    string
	Value    string
	Selected bool
}

Choice represents a choice on an HTML form. A list of choices can be rendered as an HTML select list or a checkbox group or radio group.

func MakeChoiceList 

func MakeChoiceList(options []string, selectedValue string) []Choice

func MakeChoiceListFromPairs 

func MakeChoiceListFromPairs(options []NameIDPair, selectedValue string) []Choice

func MakeMultiChoiceList 

func MakeMultiChoiceList(options []string, selectedValues []string) []Choice

func ObjChoiceList 

func ObjChoiceList(objType string, selectedIds []string) []Choice

ObjChoiceList returns a list of all items of type objType in the database. The return value is a slice of Choice objects in which the Label is the object name and the Value is the id (uuid) of each object. The slice is in alpha order by name. Choices whose values match a value in selectedIds will have their Selected attribute set to true.

func YesNoChoices 

func YesNoChoices(trueOrFalse bool) []Choice

type DartContext 

type DartContext struct {
	DB          *sql.DB
	Log         *logging.Logger
	Paths       *util.Paths
	RuntimeMode string
}
var Dart *DartContext

type EventMessage 

type EventMessage struct {
	// EventType is the type of event.
	EventType string `json:"eventType"`
	// Stage describes which stage of the job this
	// event pertains to (packaging, validation, upload).
	Stage string `json:"stage"`
	// Status describes the status of this stage
	// (running, success, failure).
	Status string `json:"status"`
	// Message is a human-friendly message to display to
	// the user.
	Message string `json:"message"`
	// Total is the total number of files to package or bytes
	// to upload. This is used to calculate percent complete for
	// progress bars on the front end.
	Total int64 `json:"total"`
	// Current is the current number of files packaged or bytes
	// uploaded. This is used to calculate percent complete for
	// progress bars on the front end.
	Current int64 `json:"current"`
	// Percent is the percent complete of the current packaging,
	// validation or upload operation. This is used to control
	// progress bars on the front end.
	Percent int `json:"percent"`
	// JobResult describes the final outcome of a job and all
	// its component operations (packaging, validation, upload).
	// This object will only be present in the finish event.
	// For all other events, it's null.
	JobResult *JobResult `json:"jobResult,omitempty"`
	// JobSummary contains information to help the front
	// end set up the job progress display. This object is present
	// only in the init event. For all other events, it's null.
	JobSummary *JobSummary `json:"jobSummary,omitempty"`
}

EventMessage contains info to send back to the front end UI about the progress of a job, so the user can see that things are moving along. We'll queue info events when we add files to a package, validate checksums, etc.

The JobResult property will be set only for the finish event. JobResult contains final information about the success or failure of the job.

func ErrorEvent 

func ErrorEvent(stage, message string) *EventMessage

ErrorEvent creates a new EventMessage with EventType error. This event has no JobResult.

func FinishEvent 

func FinishEvent(jobResult *JobResult) *EventMessage

Finish event creates an EventMessage with a JobResult describing how the job turned out. This sets the EventType to "info" if the job succeeded or to "error" if it failed.

func InfoEvent 

func InfoEvent(stage, message string) *EventMessage

InfoEvent creates a new EventMessage with EventType info. This event has no JobResult.

func InitEvent 

func InitEvent(jobSummary *JobSummary) *EventMessage

InitEvent creates a new initialization event message. This message contains info that the front end needs to set up the job progress display.

func StartEvent 

func StartEvent(stage, message string) *EventMessage

StartEvent creates a new EventMessage with EventType start. This event has no JobResult.

func WarningEvent 

func WarningEvent(stage, message string) *EventMessage

WarningEvent creates a new EventMessage with EventType warning. This event has no JobResult.

func (*EventMessage) ToJson 

func (e *EventMessage) ToJson() string

ToJson converts this EventMessage to JSON, so we can send it back to the UI for display.

type ExportOptions 

type ExportOptions struct {
	AppSettings            []NameIDPair            `json:"appSettings"`
	AppSettingFields       []string                `json:"appSettingFields"`
	BagItProfiles          []NameIDPair            `json:"bagItProfiles"`
	BagItProfileFields     map[string][]NameIDPair `json:"bagItProfileFields"`
	RemoteRepositories     []NameIDPair            `json:"remoteRepositories"`
	RemoteRepositoryFields []string                `json:"remoteRepositoryFields"`
	StorageServices        []NameIDPair            `json:"storageServices"`
	StorageServiceFields   []string                `json:"storageServiceFields"`
}

ExportOptions returns options to populate lists on the export settings page.

func NewExportOptions 

func NewExportOptions() *ExportOptions

NewExportOptions returns an object containing the data we need to populate lists on the export settings page.

func NewExportOptionsFromSettings 

func NewExportOptionsFromSettings(exportSettings *ExportSettings) *ExportOptions

NewExportOptionsFromSettings returns an object containing data for use on the export questions page. Questions only pertain to settings in the export settings, so we want to limit the data here to those settings only. This differs from NewExportOptions(), which returns info about all settings in the entire system.

type ExportQuestion 

type ExportQuestion struct {
	// ID is the question's unique uuid.
	ID string `json:"id"`
	// Prompt is the text of the question.
	Prompt string `json:"prompt"`
	// ObjType is the type of object to which we should
	// copy this question's response.
	ObjType string `json:"objType"`
	// ObjID is the id of the object to which we should
	// copy this question's response.
	ObjID string `json:"objId"`
	// Field is the name of the field on the object to
	// which we should copy this question's response.
	// For BagIt profiles, Field will contain the UUID
	// of the tag to which we should copy the response.
	Field  string            `json:"field"`
	Errors map[string]string `json:"-"`
}

func NewExportQuestion 

func NewExportQuestion() *ExportQuestion

NewExportQuestion returns a new ExportQuestion with a unique ID.

func (*ExportQuestion) CopyResponseToField 

func (q *ExportQuestion) CopyResponseToField(response string) error

CopyResponseToField copies the user's response to field Field on the object specified by ExportQuestion.ObjID. If the object is a BagIt profile, this copies the response the tag whose UUID matches the value in ExportQuestion.Field.

func (*ExportQuestion) GetExistingValue 

func (q *ExportQuestion) GetExistingValue() string

GetExistingValue returns the value currently stored in the object property to which this question refers. It will return a string, even if the value is of some other type. We use this to pre-populate answers in ExportQuestions.

func (*ExportQuestion) ToForm 

func (q *ExportQuestion) ToForm() *Form

type ExportSettings 

type ExportSettings struct {
	ID                 string              `json:"id"`
	Name               string              `json:"name"`
	AppSettings        []*AppSetting       `json:"appSettings"`
	BagItProfiles      []*BagItProfile     `json:"bagItProfiles"`
	Questions          []*ExportQuestion   `json:"questions"`
	RemoteRepositories []*RemoteRepository `json:"remoteRepositories"`
	StorageServices    []*StorageService   `json:"storageServices"`
	Errors             map[string]string   `json:"-"`
}

ExportSettings are settings that users can export as JSON to share with other users.

func NewExportSettings 

func NewExportSettings() *ExportSettings

NewExportSettings returns a new ExportSettings object.

func (*ExportSettings) ContainsPlaintextAPIToken 

func (settings *ExportSettings) ContainsPlaintextAPIToken() bool

ContainsPlaintextAPIToken returns true if any RemoteRepository in these settings contains a plaintext API token.

func (*ExportSettings) ContainsPlaintextPassword 

func (settings *ExportSettings) ContainsPlaintextPassword() bool

ContainsPlaintextPassword returns true if any StorageService in these settings contains a plaintext password.

func (*ExportSettings) GetErrors 

func (settings *ExportSettings) GetErrors() map[string]string

GetErrors returns a map of validation errors for this object.

func (*ExportSettings) IsDeletable 

func (settings *ExportSettings) IsDeletable() bool

IsDeletable returns a boolean indicating whether users are allowed to delete this object from the database.

func (*ExportSettings) ObjID 

func (settings *ExportSettings) ObjID() string

ObjID returns this object's id (uuid).

func (*ExportSettings) ObjName 

func (settings *ExportSettings) ObjName() string

ObjName returns this object's name.

func (*ExportSettings) ObjType 

func (settings *ExportSettings) ObjType() string

ObjType returns this object's type.

func (*ExportSettings) ObjectIds 

func (settings *ExportSettings) ObjectIds(objType string) ([]string, error)

ObjectIds returns the ids (uuids) for all the objects of the specified type in this group of export settings.

func (*ExportSettings) SetValueFromResponse 

func (settings *ExportSettings) SetValueFromResponse(question *ExportQuestion, response string) error

func (*ExportSettings) String 

func (settings *ExportSettings) String() string

String returns a string describing this object's type and name.

func (*ExportSettings) ToForm 

func (settings *ExportSettings) ToForm() *Form

ToForm returns a Form object that can represent this ExportSettings object. Note that the form does not include Questions. Because Questions are more complex than simple ID-Name object, the templates must handle them differently.

func (*ExportSettings) Validate 

func (settings *ExportSettings) Validate() bool

Validate always returns true for ExportSettings because we want to allow users to export any settings they choose.

type Field 

type Field struct {
	ID             string
	Name           string
	Label          string
	Value          string
	Values         []string
	Required       bool
	Help           string
	Hide           bool
	Error          string
	Choices        []Choice
	CssClasses     []string
	Attrs          map[string]string
	FormGroupClass string // set to "form-group-hidden" if SystemMustSet or tag has default value - https://github.com/APTrust/dart/blob/213e0a9acde2c73fcc8430ca436263b806228f8f/ui/forms/job_tags_form.js#L80
}

Field represents a field in an HTML form.

func NewField 

func NewField(id, name, label, value string, required bool) *Field

NewField creates a new field for a form.

func NewMultiValueField 

func NewMultiValueField(id, name, label string, values []string, required bool) *Field

func (*Field) AddChoice 

func (field *Field) AddChoice(label, value string)

AddChoice adds a choice to a field, and marks it as selected if it matches the field's value.

type FileMap 

type FileMap struct {
	Type  string
	Files map[string]*FileRecord
}

FileMap contains a map of FileRecord objects and some methods to help validate those records.

func NewFileMap 

func NewFileMap(fileType string) *FileMap

NewFileMap returns a pointer to a new FileMap object.

TODO: Deprecate this. New version should always use channel.

func (*FileMap) FileCount 

func (fm *FileMap) FileCount() int64

func (*FileMap) Oxum 

func (fm *FileMap) Oxum() string

func (*FileMap) TotalBytes 

func (fm *FileMap) TotalBytes() int64

func (*FileMap) ValidateChecksums 

func (fm *FileMap) ValidateChecksums(algs []string) map[string]string

ValidateChecksums validates all checksums for all files in this FileMap. Param algs is a list of algorithms for manifests found in the bag.

func (*FileMap) ValidateChecksumsWithCallback 

func (fm *FileMap) ValidateChecksumsWithCallback(algs []string, callback func(string, string)) map[string]string

ValidateChecksumsWithCallback validates all checksums for all files in this FileMap. Param algs is a list of algorithms for manifests found in the bag. Param callback is a function to send Event messages back to the front end.

func (*FileMap) WriteManifest 

func (fm *FileMap) WriteManifest(writer io.Writer, fileType, alg, trimFromPath string) error

WriteManifest is used during bagging to write a manifest. Param fileType should be either constants.FileTypePayload or constants.FileTypeTag, depending on whether you're writing a payload manifest or a tag manifest. Param alg is the digest algorithm. Those are defined in constants. ("md5", "sha256", etc)

Param trimFromPath is an optional prefix to remove from file paths when writing the manifest. For example, tarred bags include an additional top-level directory that matches the bag name, but this directory should not be included in the manifest paths.

What appears in the tar file as bag_name/data/file.txt should appear in the manifest as data/file.txt. To make that happen, pass "bag_name/" as the last param, and it will be trimmed. Pass an empty string if you don't need to trim leading paths.

type FileRecord 

type FileRecord struct {
	Size      int64
	Checksums []*Checksum
}

FileRecord contains a collection of checksums for a single file.

func NewFileRecord 

func NewFileRecord() *FileRecord

NewFileRecord returns a pointer to a new FileRecord object.

func (*FileRecord) AddChecksum 

func (fr *FileRecord) AddChecksum(source, alg, digest string)

AddChecksum adds a checksum to this FileRecord.

func (*FileRecord) DigestAlgorithms 

func (fr *FileRecord) DigestAlgorithms() []string

DigestAlgorithms returns a list of digest algorithms calculated for this file.

func (*FileRecord) GetChecksum 

func (fr *FileRecord) GetChecksum(alg, source string) *Checksum

GetChecksum returns the checksum with the specified algorithm and source.

func (*FileRecord) Validate 

func (fr *FileRecord) Validate(fileType string, algs []string) error

Validate validates the following about the current file:

  • file is present in the payload directory
  • file is listed in the payload manifests matching the specified algorithms.
  • the checksums that the validator calculated on the file itself match the checksums in the manifests.

Validate returns true if the checksums we calculated for the file match the checksums in the manifests.

type Form 

type Form struct {
	ObjType       string
	ObjectID      string
	Fields        map[string]*Field
	UserCanDelete bool
	Errors        map[string]string
}

Form represents an HTML form.

func NewBagItProfileCreationForm 

func NewBagItProfileCreationForm() (*Form, error)

NewBagItProfileCreationForm returns a form for starting the BagIt Profile creation process. This form contains only a single element: a select list containing a list of base profiles on which to base the new profile.

func NewForm 

func NewForm(objType, objectId string, errors map[string]string) *Form

NewForm creates a new form for the specified object type and id. The errors map is used to set error messages on the form's fields.

func (*Form) AddField 

func (f *Form) AddField(name, label, value string, required bool) *Field

AddField adds a new field to the form.

func (*Form) AddMultiValueField 

func (f *Form) AddMultiValueField(name, label string, values []string, required bool) *Field

func (*Form) String 

func (f *Form) String() string

String converts the form to a string for logging purposes.

type InternalSetting 

type InternalSetting struct {
	ID     string            `json:"id"`
	Name   string            `json:"name"`
	Value  string            `json:"value"`
	Errors map[string]string `json:"errors"`
}

InternalSetting is set by DART and cannot be edited by user. These settings may record when migrations were run, or other internal info. These settings cannot be created or edited by users.

func NewInternalSetting 

func NewInternalSetting(name, value string) *InternalSetting

func (*InternalSetting) GetErrors 

func (setting *InternalSetting) GetErrors() map[string]string

func (*InternalSetting) IsDeletable 

func (setting *InternalSetting) IsDeletable() bool

func (*InternalSetting) ObjID 

func (setting *InternalSetting) ObjID() string

func (*InternalSetting) ObjName 

func (setting *InternalSetting) ObjName() string

func (*InternalSetting) ObjType 

func (setting *InternalSetting) ObjType() string

func (*InternalSetting) String 

func (setting *InternalSetting) String() string

func (*InternalSetting) ToForm 

func (setting *InternalSetting) ToForm() *Form

ToForm returns a form to comply with the PersistentObject interface, but internal settings are not editable, so the app never displays this form.

func (*InternalSetting) Validate 

func (setting *InternalSetting) Validate() bool

Validate validates this setting, returning true if it's valid, false if not. If false, this sets specific error messages in the Errors map, which are suitable for display on the form.

type Job 

type Job struct {
	ID               string               `json:"id"`
	BagItProfile     *BagItProfile        `json:"bagItProfile"`
	ByteCount        int64                `json:"byteCount"`
	DirCount         int64                `json:"dirCount"`
	Errors           map[string]string    `json:"errors"`
	PayloadFileCount int64                `json:"fileCount"` // still fileCount in JSON for legacy compatibility
	TotalFileCount   int64                `json:"totalFileCount"`
	PackageOp        *PackageOperation    `json:"packageOp"`
	UploadOps        []*UploadOperation   `json:"uploadOps"`
	ValidationOp     *ValidationOperation `json:"validationOp"`
	WorkflowID       string               `json:"workflowId"`
	ArtifactsDir     string               `json:"artifactsDir"`
}

func JobFromJson 

func JobFromJson(pathToFile string) (*Job, error)

JobFromJson converts the JSON data in file pathToFile into a Job object.

func JobFromWorkflow 

func JobFromWorkflow(workflow *Workflow) *Job

JobFromWorkflow creates a new job based on the specified workflow. Note that the new job will have no name or source files.

func NewJob 

func NewJob() *Job

NewJob creates a new Job with a unique ID.

func (*Job) GetErrors 

func (job *Job) GetErrors() map[string]string

func (*Job) GetResultMessages 

func (job *Job) GetResultMessages() (stdoutMessage, stdErrMessage string)

func (*Job) HasPackageOp 

func (job *Job) HasPackageOp() bool

HasPackageOp returns true if this job includes a package operation.

func (*Job) HasUploadOps 

func (job *Job) HasUploadOps() bool

HasUploadOps returns true if this job includes one or more upload operations.

func (*Job) IsDeletable 

func (job *Job) IsDeletable() bool

func (*Job) Name 

func (job *Job) Name() string

Name returns a name for this job, which is usually the file name of the package being built, validated, or uploaded.

func (*Job) ObjID 

func (job *Job) ObjID() string

ObjID returns this job's object id (uuid).

func (*Job) ObjName 

func (job *Job) ObjName() string

ObjName returns this object's name, so names will be searchable and sortable in the DB.

func (*Job) ObjType 

func (job *Job) ObjType() string

ObjType returns this object's type name.

func (*Job) Outcome 

func (job *Job) Outcome() JobOutcomeSummary

Outcome returns a JobOutcomeSummary describing whether or not the job was run and whether or not it succeeded.

func (*Job) PackageAttempted 

func (job *Job) PackageAttempted() bool

PackageAttempted returns true if DART attempted to execute this job's package operation.

func (*Job) PackageFormat 

func (job *Job) PackageFormat() string

PackageFormat returns the name of the format in which this job will package its files. Usually, this will be constants.PackageFormatBagIt, but if the job has no package operation (i.e. an upload-only or validation-only job), it will be constants.PackageFormatNone.

func (*Job) PackageSucceeded 

func (job *Job) PackageSucceeded() bool

PackageSucceeded returns true if DART successfully completed this job's package operation. Note that this will return false if packaging failed and if packaging was never attempted, so check PackageAttempted as well.

func (*Job) PackagedAt 

func (job *Job) PackagedAt() time.Time

PackagedAt returns the datetime on which this job's package operation completed.

func (*Job) RuntimeErrors 

func (job *Job) RuntimeErrors() map[string]string

RuntimeErrors returns a list of errors from all of this job's operations.

func (*Job) String 

func (job *Job) String() string

func (*Job) ToForm 

func (job *Job) ToForm() *Form

func (*Job) UpdatePayloadStats 

func (job *Job) UpdatePayloadStats()

func (*Job) UploadAttempted 

func (job *Job) UploadAttempted() bool

UploadAttempted returns true if DART attempted to execute any of this job's upload operations.

func (Job) UploadSucceeded 

func (job Job) UploadSucceeded() bool

UploadSucceeded returns true if DART successfully completed all of this job's upload operations. See UploadAttempted as well.

func (*Job) UploadedAt 

func (job *Job) UploadedAt() time.Time

UploadedAt returns the datetime on which this job's last upload operation completed.

func (*Job) Validate 

func (job *Job) Validate() bool

Validate returns true or false, indicating whether this object contains complete and valid data. If it returns false, check the errors property for specific errors.

func (*Job) ValidatedAt 

func (job *Job) ValidatedAt() time.Time

ValidatedAt returns the datetime on which this job's validation operation completed.

func (*Job) ValidationAttempted 

func (job *Job) ValidationAttempted() bool

ValidationAttempted returns true if DART attempted to execute this job's validation operation.

func (*Job) ValidationSucceeded 

func (job *Job) ValidationSucceeded() bool

ValidationSucceeded returns true if DART successfully completed this job's validation operation. See ValidationAttempted as well.

type JobOutcomeSummary 

type JobOutcomeSummary struct {
	JobName           string
	JobWasRun         bool
	JobSucceeded      bool
	Stage             string
	Message           string
	BagItProfileName  string
	SuccessfulUploads []string
	FailedUploads     []string
	LastActivity      time.Time
}

type JobParams 

type JobParams struct {
	Errors      map[string]string `json:"errors"`
	Files       []string          `json:"files"`
	PackageName string            `json:"packageName"`
	OutputPath  string            `json:"outputPath"`
	Tags        []*Tag            `json:"tags"`
	Workflow    *Workflow         `json:"workflow"`
}

JobParams describes what we need to assemble to create a Job for the JobRunner. This structure allows to create a job object by describing what needs to be done (the workflow), what set of files we're operating on (the files) and where we'll store a local copy of the result (the output path).

func NewJobParams 

func NewJobParams(workflow *Workflow, packageName, outputPath string, files []string, tags []*Tag) *JobParams

NewJobParams creates a new JobParams object.

Param workflow is the workflow to run. packageName is the name of the output file (for now, a tarred bag, so something like "my_bag.tar"). outputPath is the directory into which we should write the bag. files is a list of files to bag.

func (*JobParams) ToJob 

func (p *JobParams) ToJob() *Job

ToJob converts a JobParams object to a Job object, which can be run directly by the JobRunner.

type JobResult 

type JobResult struct {
	JobID             string             `json:"jobId"`
	JobName           string             `json:"jobName"`
	PayloadByteCount  int64              `json:"payloadByteCount"`
	PayloadFileCount  int64              `json:"payloadFileCount"`
	Succeeded         bool               `json:"succeeded"`
	PackageResult     *OperationResult   `json:"packageResult"`
	ValidationResults []*OperationResult `json:"validationResults"`
	UploadResults     []*OperationResult `json:"uploadResults"`
	ValidationErrors  map[string]string  `json:"validationErrors"`
}

JobResult collects the results of an attempted job for reporting purposes.

func NewJobResult 

func NewJobResult(job *Job) *JobResult

NewJobResult creates an object containing the results of all actions attempted in a job. The WorkflowRunner prints a json representation of this object to stdout upon completion or termination of each job.

func NewJobResultFromUploadJob 

func NewJobResultFromUploadJob(uploadJob *UploadJob) *JobResult

func NewJobResultFromValidationJob 

func NewJobResultFromValidationJob(valJob *ValidationJob) *JobResult

NewJobResultFromValidationJob creates a new JobResult containing info about the outcome of a ValidationJob.

func (*JobResult) ToJson 

func (r *JobResult) ToJson() (string, error)

ToJson returns a JSON string describing the results of this job's operations.

type JobSummary 

type JobSummary struct {
	ID                      string   `json:"id"`
	JobType                 string   `json:"jobType"`
	Name                    string   `json:"name"`
	HasPackageOp            bool     `json:"hasPackageOp"`
	HasUploadOps            bool     `json:"hasUploadOps"`
	HasBagItProfile         bool     `json:"hasBagItProfile"`
	PackageFormat           string   `json:"packageFormat"`
	PackageName             string   `json:"packageName"`
	BagItProfileName        string   `json:"bagItProfileName"`
	BagItProfileDescription string   `json:"bagItProfileDescription"`
	DirectoryCount          int64    `json:"directoryCount"`
	PayloadFileCount        int64    `json:"payloadFileCount"`
	ByteCount               int64    `json:"byteCount"`
	ByteCountFormatted      string   `json:"byteCountFormatted"`
	ByteCountHuman          string   `json:"byteCountHuman"`
	SourceFiles             []string `json:"sourceFiles"`
	OutputPath              string   `json:"outputPath"`
	UploadTargets           []string `json:"uploadTargets"`
	PathSeparator           string   `json:"pathSeparator"`
}

JobSummary contains info about a job to be displayed in the front end. All of this info appears on the job_run page and the batch_run page. It may appear elsewhere as well. Note that it takes some logic to extract some of this information. We don't want to force that logic into HTML templates and front-end JavaScript, so we do it here, where it's easy to test.

func NewJobSummary 

func NewJobSummary(job *Job) *JobSummary

NewJobSummary creates a new JobSummaryInfo object based on the given job.

NOTE: To get accurate info about the job's payload, call job.UpdatePayloadStats() before calling this constructor.

func NewUploadJobSummary 

func NewUploadJobSummary(uploadJob *UploadJob) *JobSummary

func NewValidationJobSummary 

func NewValidationJobSummary(valJob *ValidationJob, profile *BagItProfile) *JobSummary

type LOCKSSClientV2 

type LOCKSSClientV2 struct {
	// contains filtered or unexported fields
}

LOCKSSClientV2 is a remote repository client that can talk to the LOCKSS API. This client satisfies the RemoteRepoClient interface and includes additional methods for depositing materials into LOCKSS repos.

func NewLOCKSSClientV2 

func NewLOCKSSClientV2(config *RemoteRepository) *LOCKSSClientV2

NewLOCKSSClientV2 returns a new instance of the LOCKSS remote repo client.

func (*LOCKSSClientV2) APIVersion 

func (client *LOCKSSClientV2) APIVersion() string

APIVersion returns the version number of the API that this client can talk to.

func (*LOCKSSClientV2) AvailableHTMLReports 

func (client *LOCKSSClientV2) AvailableHTMLReports() []util.NameValuePair

AvailableHTMLReports returns a list of available HTML report names. In the NameValuePair, Name is the name of the report and Value is a description.

func (*LOCKSSClientV2) Description 

func (client *LOCKSSClientV2) Description() string

Description returns a short description of this client.

func (*LOCKSSClientV2) ID 

func (client *LOCKSSClientV2) ID() string

ID returns this client's UUID.

func (*LOCKSSClientV2) Name 

func (client *LOCKSSClientV2) Name() string

Name returns the client name.

func (*LOCKSSClientV2) RunHTMLReport 

func (client *LOCKSSClientV2) RunHTMLReport(name string) (string, error)

RunHTMLReport runs the named report and returns HTML suitable for display on the DART dashboard. For a list of available report names, call AvailableHTMLReports().

func (*LOCKSSClientV2) TestConnection 

func (client *LOCKSSClientV2) TestConnection() error

TestConnection tests a connection to the remote repo. It returns true or false to describe whether the connection succeeded. Check the error if the connection did not succeed.

type LOCOrderedProfile 

type LOCOrderedProfile struct {
	Tags []map[string]LOCTagDef `json:"ordered"`
}

LOCOrderedProfile represents a Library of Congress BagIt profile containing an ordered list of tags. In this format, each entry in the list is a map containing a single key and value, where key is the tag name and value is the LOC tag definition.

type LOCTagDef 

type LOCTagDef struct {
	Required      bool     `json:"fieldRequired,omitempty"`
	DefaultValue  string   `json:"defaultValue,omitempty"`
	Values        []string `json:"valueList,omitempty"`
	RequiredValue string   `json:"requiredValue,omitempty"`
}

LOCTagDef represents a Library of Congress tag definition. These may appear in both ordered and unordered LOC profiles. Unordered LOC profiles are simply a map in format map[string]LOCTagDef

type NameIDPair 

type NameIDPair struct {
	Name string
	ID   string
}

func ArtifactNameIDList 

func ArtifactNameIDList(jobID string) ([]NameIDPair, error)

func ObjNameIdList 

func ObjNameIdList(objType string) []NameIDPair

ObjNameIdList returns a list of all items of type objType in the database. The return value is a slice of NameIDPair objects containing the name and id (uuid) of each object. The slice is in alpha order by name.

func RepoClientList 

func RepoClientList() []NameIDPair

RepoClientList returns a list of NameIDPair objects, listing available remote repository clients. This is useful for creating HTML select lists.

func UserSettableTagsForProfile 

func UserSettableTagsForProfile(bagItProfileID string) ([]NameIDPair, error)

UserSettableTagsForProfile returns a list of tags defined in the BagIt profile with the specified ID. The list inlcudes only those tags whose values users can set. Tags such as Payload-Oxum will be excluded because users can't set their value.

type OperationResult 

type OperationResult struct {
	Attempt          int               `json:"attempt"`
	Completed        time.Time         `json:"completed"`
	Errors           map[string]string `json:"errors"`
	FileMTime        time.Time         `json:"fileMtime"`
	FilePath         string            `json:"filepath"`
	FileSize         int64             `json:"filesize"`
	Info             string            `json:"info"`
	Operation        string            `json:"operation"`
	Provider         string            `json:"provider"`
	RemoteTargetName string            `json:"remoteTargetName"`
	RemoteChecksum   string            `json:"remoteChecksum"`
	RemoteURL        string            `json:"remoteURL"`
	Started          time.Time         `json:"started"`
	Warning          string            `json:"warning"`
}

func NewOperationResult 

func NewOperationResult(operation, provider string) *OperationResult

func (*OperationResult) Finish 

func (r *OperationResult) Finish(errors map[string]string)

func (*OperationResult) HasErrors 

func (r *OperationResult) HasErrors() bool

func (*OperationResult) Reset 

func (r *OperationResult) Reset()

func (*OperationResult) Start 

func (r *OperationResult) Start()

func (*OperationResult) Succeeded 

func (r *OperationResult) Succeeded() bool

func (*OperationResult) WasAttempted 

func (r *OperationResult) WasAttempted() bool

func (*OperationResult) WasCompleted 

func (r *OperationResult) WasCompleted() bool

type Options 

type Options struct {
	WorkflowFilePath  string
	BatchFilePath     string
	OutputDir         string
	StdinData         []byte
	Concurrency       int
	DeleteAfterUpload bool
	ShowHelp          bool
	Version           bool
}

func ParseOptions 

func ParseOptions() *Options

func (Options) AreValid 

func (opts Options) AreValid() bool

AreValid returns true if options are valid. That is, they contain enough info for DART Runner to proceed.

type PackageOperation 

type PackageOperation struct {
	BagItSerialization string            `json:"bagItSerialization"`
	Errors             map[string]string `json:"errors"`
	OutputPath         string            `json:"outputPath"`
	PackageName        string            `json:"packageName"`
	PackageFormat      string            `json:"packageFormat"`
	PayloadSize        int64             `json:"payloadSize"`
	Result             *OperationResult  `json:"result"`
	SourceFiles        []string          `json:"sourceFiles"`
}

func NewPackageOperation 

func NewPackageOperation(packageName, outputPath string, sourceFiles []string) *PackageOperation

func (*PackageOperation) PruneSourceFiles 

func (p *PackageOperation) PruneSourceFiles()

PruneSourceFiles removes non-existent files and directories from SouceFiles and de-dupes duplicates.

func (*PackageOperation) ToForm 

func (p *PackageOperation) ToForm() *Form

func (*PackageOperation) Validate 

func (p *PackageOperation) Validate() bool

type PersistentObject 

type PersistentObject interface {
	GetErrors() map[string]string
	IsDeletable() bool
	ObjID() string
	ObjName() string
	ObjType() string
	String() string
	ToForm() *Form
	Validate() bool
}

type ProfileInfo 

type ProfileInfo struct {
	BagItProfileIdentifier string `json:"bagItProfileIdentifier" form:"InfoIdentifier"`
	BagItProfileVersion    string `json:"bagItProfileVersion" form:"InfoProfileVersion"`
	ContactEmail           string `json:"contactEmail" form:"InfoContactEmail"`
	ContactName            string `json:"contactName" form:"InfoContactName"`
	ExternalDescription    string `json:"externalDescription" form:"InfoExternalDescription"`
	SourceOrganization     string `json:"sourceOrganization" form:"InfoSourceOrganization"`
	Version                string `json:"version" form:"InfoVersion"`
}

ProfileInfo contains meta info about the profile.

func CopyProfileInfo 

func CopyProfileInfo(info ProfileInfo) ProfileInfo

type QueryResult 

type QueryResult struct {
	AppSettings        []*AppSetting
	BagItProfiles      []*BagItProfile
	Error              error
	ExportSettings     []*ExportSettings
	InternalSettings   []*InternalSetting
	Jobs               []*Job
	Limit              int
	ObjCount           int
	ObjType            string
	Offset             int
	OrderBy            string
	RemoteRepositories []*RemoteRepository
	ResultType         string
	StorageServices    []*StorageService
	UploadJobs         []*UploadJob
	ValidationJobs     []*ValidationJob
	Workflows          []*Workflow
	WorkflowBatches    []*WorkflowBatch
}

func NewQueryResult 

func NewQueryResult(resultType string) *QueryResult

func ObjByNameAndType 

func ObjByNameAndType(objName, objType string) *QueryResult

ObjByNameAndType returns the object of the specified type with the matching name. Note that the DB has a unique constraint on obj_type + obj_name, so this should return at most one row.

func ObjFind 

func ObjFind(uuid string) *QueryResult

func ObjList 

func ObjList(objType, orderBy string, limit, offset int) *QueryResult

func (*QueryResult) AppSetting 

func (qr *QueryResult) AppSetting() *AppSetting

func (*QueryResult) BagItProfile 

func (qr *QueryResult) BagItProfile() *BagItProfile

func (*QueryResult) ExportSetting 

func (qr *QueryResult) ExportSetting() *ExportSettings

func (*QueryResult) GetForm 

func (qr *QueryResult) GetForm() (*Form, error)

func (*QueryResult) InternalSetting 

func (qr *QueryResult) InternalSetting() *InternalSetting

func (*QueryResult) Job 

func (qr *QueryResult) Job() *Job

func (*QueryResult) RemoteRepository 

func (qr *QueryResult) RemoteRepository() *RemoteRepository

func (*QueryResult) ResultCount 

func (qr *QueryResult) ResultCount() int

Result count returns the total number of records returned by the query. This is different from ObjectCount, which is the total number of objects of a specified type in the DB.

func (*QueryResult) StorageService 

func (qr *QueryResult) StorageService() *StorageService

func (*QueryResult) UploadJob 

func (qr *QueryResult) UploadJob() *UploadJob

func (*QueryResult) ValidationJob 

func (qr *QueryResult) ValidationJob() *ValidationJob

func (*QueryResult) Workflow 

func (qr *QueryResult) Workflow() *Workflow

func (*QueryResult) WorkflowBatch 

func (qr *QueryResult) WorkflowBatch() *WorkflowBatch

type RemoteRepoClient 

type RemoteRepoClient interface {
	ID() string
	Name() string
	APIVersion() string
	Description() string
	AvailableHTMLReports() []util.NameValuePair
	RunHTMLReport(string) (string, error)
	TestConnection() error
}

RemoteRepoClient interface defines the basics for talking to a remote repository. Each client talks to one specific type of remote repository, usually through a REST API. We don't want to know the internal workings of these clients. We just want to be able to get reports to display on the dashboard.

func GetRemoteRepoClient 

func GetRemoteRepoClient(repo *RemoteRepository) (RemoteRepoClient, error)

GetRemoteRepoClient returns a client that can talk to a RemoteRepository.

func RemoteRepoClientAPTrust 

func RemoteRepoClientAPTrust(repo *RemoteRepository) RemoteRepoClient

RemoteRepoClientAPTrust returns an APTrustlient as a basic RemoteRepoClient to support automated discovery and generation of clients using the factory method GetRemoteRepoClient().

If you need access to APTrust client methods outside of the RemoteRepoClient interface, which is just for reporting, use NewLOCKSSClient() instead.

func RemoteRepoClientLOCKSS 

func RemoteRepoClientLOCKSS(config *RemoteRepository) RemoteRepoClient

RemoteRepoClientLOCKSS returns a LOCKSSClient as a basic RemoteRepoClient to support automated discovery and generation of clients using the factory method GetRemoteRepoClient().

If you need access to LOCKSS client methods outside of the RemoteRepoClient interface, which is just for reporting, use NewLOCKSSClient() instead.

type RemoteRepository 

type RemoteRepository struct {
	ID         string            `json:"id"`
	Name       string            `json:"name"`
	Url        string            `json:"url"`
	UserID     string            `json:"userId"`
	APIToken   string            `json:"apiToken"`
	LoginExtra string            `json:"loginExtra"`
	PluginID   string            `json:"pluginId"`
	Errors     map[string]string `json:"errors"`
}

RemoteRepository contains config settings describing how to connect to a remote repository, such as APTrust. Presumably, this is a repository into which you are ingesting data, and the repository has a REST API.

The repo config allows you to connect to the repo so you can see the state of bags you uploaded. The logic for performing those requests and parsing the responses has to be implemented elsewhere. In DART 2.x, this was done with plugins, and APTrust was the only existing plugin. In DART 3.x, the way to add new repo implementations is to be determined. One suggestion is to generate clients with Swagger/OpenAPI.

func CreateRemoteRepos 

func CreateRemoteRepos(howMany int) ([]*RemoteRepository, error)

func NewRemoteRepository 

func NewRemoteRepository() *RemoteRepository

func (*RemoteRepository) GetAPIToken 

func (repo *RemoteRepository) GetAPIToken() string

GetUserAPIToken returns the API token for logging into this remote repo. If the token comes from an environment variable, this returns the value of the ENV var. Otherwise, it returns the value of this object's APIToken property.

When authenticating with a remote repo, call this instead of accessing APIToken directly.

func (*RemoteRepository) GetErrors 

func (repo *RemoteRepository) GetErrors() map[string]string

func (*RemoteRepository) GetUserID 

func (repo *RemoteRepository) GetUserID() string

GetUserID returns the UserID for logging into this remote repo. If the UserID comes from an environment variable, this returns the value of the ENV var. Otherwise, it returns the value of this object's UserID property.

When authenticating with a remote repo, call this instead of accessing UserID directly.

func (*RemoteRepository) HasPlaintextAPIToken 

func (repo *RemoteRepository) HasPlaintextAPIToken() bool

HasPlaintextAPIToken returns true if this repo's API token is non-empty and does not come from an environment variable.

func (*RemoteRepository) IsDeletable 

func (repo *RemoteRepository) IsDeletable() bool

func (*RemoteRepository) ObjID 

func (repo *RemoteRepository) ObjID() string

ObjID returns this remote repo's UUID.

func (*RemoteRepository) ObjName 

func (repo *RemoteRepository) ObjName() string

ObjName returns the name of this remote repo.

func (*RemoteRepository) ObjType 

func (repo *RemoteRepository) ObjType() string

ObjType returns this object's type.

func (*RemoteRepository) ReportsAvailable 

func (repo *RemoteRepository) ReportsAvailable() ([]util.NameValuePair, error)

ReportsAvailable returns a list of reports that can be retrieved from this remote repository.

func (*RemoteRepository) String 

func (repo *RemoteRepository) String() string

func (*RemoteRepository) TestConnection 

func (repo *RemoteRepository) TestConnection() error

func (*RemoteRepository) ToForm 

func (repo *RemoteRepository) ToForm() *Form

func (*RemoteRepository) Validate 

func (repo *RemoteRepository) Validate() bool

Validate returns true if this RemoteRepository config contains valid settings, false if not. Check the Errors map if this returns false.

type Runner 

type Runner struct {
	Job            *Job
	MessageChannel chan *EventMessage
}

func (*Runner) RunPackageOp 

func (r *Runner) RunPackageOp() bool

func (*Runner) RunUploadOps 

func (r *Runner) RunUploadOps() bool

func (*Runner) RunValidationOp 

func (r *Runner) RunValidationOp() bool

func (*Runner) ValidateJob 

func (r *Runner) ValidateJob() bool

type StandardProfile 

type StandardProfile struct {
	AcceptBagItVersion   []string                         `json:"Accept-BagIt-Version"`
	AcceptSerialization  []string                         `json:"Accept-Serialization"`
	AllowFetchTxt        bool                             `json:"Allow-Fetch.txt"`
	Serialization        string                           `json:"Serialization"`
	ManifestsAllowed     []string                         `json:"Manifests-Allowed"`
	ManifestsRequired    []string                         `json:"Manifests-Required"`
	TagManifestsAllowed  []string                         `json:"Tag-Manifests-Allowed"`
	TagManifestsRequired []string                         `json:"Tag-Manifests-Required"`
	TagFilesAllowed      []string                         `json:"Tag-Files-Allowed"`
	TagFilesRequired     []string                         `json:"Tag-Files-Required"`
	BagItProfileInfo     StandardProfileInfo              `json:"BagIt-Profile-Info"`
	BagInfo              map[string]StandardProfileTagDef `json:"Bag-Info"`
}

StandardProfile represents a BagIt Profile in BagIt Profile Spec version 1.3.0. See https://bagit-profiles.github.io/bagit-profiles-specification/ and https://github.com/bagit-profiles/bagit-profiles-specification

The BagIt Profile spec can describe a limited subset of the properties DART BagIt profiles can describe.

func NewStandardProfile 

func NewStandardProfile() *StandardProfile

NewStandardProfile creates a new StandardProfile object with all internal structs, slices, and maps allocated and initialized to empty values.

func StandardProfileFromJson 

func StandardProfileFromJson(jsonBytes []byte) (*StandardProfile, error)

StandardProfileFromJson converts the JSON in jsonBytes into a StandardProfile object.

func (*StandardProfile) ToDartProfile 

func (sp *StandardProfile) ToDartProfile() *BagItProfile

ToDartProfile converts a StandardProfile to a DART BagItProfile object.

func (*StandardProfile) ToJSON 

func (sp *StandardProfile) ToJSON() (string, error)

ToJSON returns the StandardProfile object in pretty-printed JSON format.

type StandardProfileInfo 

type StandardProfileInfo struct {
	BagItProfileIdentifier string `json:"BagIt-Profile-Identifier"`
	BagItProfileVersion    string `json:"BagIt-Profile-Version"`
	ContactEmail           string `json:"Contact-Email"`
	ContactName            string `json:"Contact-Name"`
	ExternalDescription    string `json:"External-Description"`
	SourceOrganization     string `json:"Source-Organization"`
	Version                string `json:"Version"`
}

StandardProfileInfo is structurally identical to BagItProfileInfo, but it serializes to and from JSON differently.

type StandardProfileTagDef 

type StandardProfileTagDef struct {
	Required    bool     `json:"required"`
	Recommended bool     `json:"recommended"`
	Values      []string `json:"values"`
	Description string   `json:"description"`
}

StandardProfileTagDef represents a tag definition in BagIt Profile Spec version 1.3.0.

type StorageService 

type StorageService struct {
	ID             string            `json:"id"`
	AllowsDownload bool              `json:"allowsDownload"`
	AllowsUpload   bool              `json:"allowsUpload"`
	Bucket         string            `json:"bucket"`
	Description    string            `json:"description"`
	Errors         map[string]string `json:"-"`
	Host           string            `json:"host"`
	Login          string            `json:"login"`
	LoginExtra     string            `json:"loginExtra"`
	Name           string            `json:"name"`
	Password       string            `json:"password"`
	Port           int               `json:"port"`
	Protocol       string            `json:"protocol"`
}

func CreateStorageServices 

func CreateStorageServices(howMany int) ([]*StorageService, error)

func LoadStorageServiceFixture 

func LoadStorageServiceFixture(filename string) (*StorageService, error)

For testing only. Valid fixtures are:

storage_service_local_minio.json storage_service_local_sftp.json

func NewStorageService 

func NewStorageService() *StorageService

func (*StorageService) Copy 

func (ss *StorageService) Copy() *StorageService

Copy returns a pointer to a new StorageService whose values are the same as this service. The copy will have the same ID as the original, so if you want to change it, you'll have to do that yourself.

func (*StorageService) GetErrors 

func (ss *StorageService) GetErrors() map[string]string

func (*StorageService) GetLogin 

func (ss *StorageService) GetLogin() string

GetLogin returns the login name or AccessKeyID to connect to this storage service. Per the DART docts, if the login begins with "env:", we fetch it from the environment. For example, "env:MY_SS_LOGIN" causes us to fetch the env var "MY_SS_LOGIN". This allows us to copy Workflow info across the wire without exposing sensitive credentials.

If the login does not begin with "env:", this returns it verbatim.

func (*StorageService) GetPassword 

func (ss *StorageService) GetPassword() string

GetPassword returns this storage service's password from the StorageService record or from the environment as necessary. See the documentation for StorageService.GetLogin() for more info.

func (*StorageService) HasPlaintextPassword 

func (ss *StorageService) HasPlaintextPassword() bool

HasPlaintextPassword returns true if this StorageService has a non-empty password and does not use an environment variable for the password.

func (*StorageService) HostAndPort 

func (ss *StorageService) HostAndPort() string

HostAndPort returns the host and port for connecting to a remote service. Use this when creating a connection to a remote S3 service.

func (*StorageService) IsDeletable 

func (ss *StorageService) IsDeletable() bool

func (*StorageService) ObjID 

func (ss *StorageService) ObjID() string

ObjID returns this remote ss's UUID.

func (*StorageService) ObjName 

func (ss *StorageService) ObjName() string

ObjName returns the name of this remote ss.

func (*StorageService) ObjType 

func (ss *StorageService) ObjType() string

ObjType returns this object's type.

func (*StorageService) String 

func (ss *StorageService) String() string

func (*StorageService) TestConnection 

func (ss *StorageService) TestConnection() error

func (*StorageService) ToForm 

func (ss *StorageService) ToForm() *Form

func (*StorageService) URL 

func (ss *StorageService) URL(filename string) string

URL returns the URL to which the file will be uploaded.

func (*StorageService) Validate 

func (ss *StorageService) Validate() bool

type StreamProgress 

type StreamProgress struct {
	Total          int64
	Current        int64
	Percent        int
	MessageChannel chan *EventMessage
	// contains filtered or unexported fields
}

StreamProgress sends info from an upload/download stream into a message channel so we can report upload/download progress to the user. When running in GUI mode, the S3 and SFTP clients send EventMessages to the front-end via server-sent events, and the front end updates a progress bar.

func NewStreamProgress 

func NewStreamProgress(byteCount int64, messageChannel chan *EventMessage) *StreamProgress

NewStreamProgress returns a new StreamProgress object. Param byteCount is the number of bytes we need to send or receive to complete the upload/download/copy operation.

func (*StreamProgress) Read 

func (p *StreamProgress) Read(b []byte) (int, error)

Read satisfies the progress interface for the Minio client, which passes progress information as it uploads. Each time Minio calls Read(), the length of b equals the total number of bytes read thus far.

See https://github.com/search?q=repo%3Aminio%2Fminio-go%20progress&type=code

func (*StreamProgress) SetTotalBytesCompleted 

func (p *StreamProgress) SetTotalBytesCompleted(byteCount int64) (int, error)

SetTotalBytesCompleted satisfies the progress interface for SFTP uploads, where the progress meter periodically tells us the number of total bytes uploaded thus far.

type Tag 

type Tag struct {
	TagFile string `json:"tagFile"`
	TagName string `json:"tagName"`
	Value   string `json:"value"`
}

Tag describes a tag parsed from a BagIt file such as bag-info.txt.

func NewTag 

func NewTag(sourceFile, label, value string) *Tag

NewTag returns a new Tag object. Params are self-explanatory.

func ParseTagFile 

func ParseTagFile(reader io.Reader, relFilePath string) ([]*Tag, error)

ParseTagFile parses the tags in a tag file that conforms to the BagIt spec, which uses colon-separated Label:Value tags that may span multiple lines. See the BagIt spec at https://tools.ietf.org/html/draft-kunze-bagit-17#section-2.2.2 for more info.

Param reader can be an open file, a buffer, or any other io.Reader. If the reader needs to be closed, the user is responsible for closing it.

Param relFilePath is the relative path of the file in the bag. For example, "bag-info.txt" or "custom-tags/custom-info.txt".

This returns a slice of Tag objects as parsed from the bag. Note that the BagIt spec permits some tags to appear more than once in a file, so you may get multiple tags with the same label.

func (*Tag) FullyQualifiedName 

func (t *Tag) FullyQualifiedName() string

FullyQualifiedName returns the tag's fully qualified name, which is the tag file followed by a slash followed by the tag name. E.g. bag-info.txt/Source-Organization.

type TagDefinition 

type TagDefinition struct {
	DefaultValue    string            `json:"defaultValue"`
	EmptyOK         bool              `json:"emptyOK"`
	Errors          map[string]string `json:"-"`
	Help            string            `json:"help"`
	ID              string            `json:"id"`
	IsBuiltIn       bool              `json:"isBuiltIn"`
	IsUserAddedFile bool              `json:"isUserAddedFile"`
	IsUserAddedTag  bool              `json:"isUserAddedTag"`
	Required        bool              `json:"required"`
	TagFile         string            `json:"tagFile"`
	TagName         string            `json:"tagName"`
	UserValue       string            `json:"userValue"`
	Values          []string          `json:"values"`
	WasAddedForJob  bool              `json:"wasAddedForJob"`
}

TagDefinition describes a tag in a BagItProfile, whether it's required, what values are allowed, etc.

func (*TagDefinition) Copy 

func (t *TagDefinition) Copy() *TagDefinition

Copy returns a pointer to a new TagDefinition whose values are the same as this TagDefinition.

func (*TagDefinition) FullyQualifiedName 

func (t *TagDefinition) FullyQualifiedName() string

FullyQualifiedName returns the tag defintion's fully qualified name, which is the tag file followed by a slash followed by the tag name. E.g. bag-info.txt/Source-Organization.

func (*TagDefinition) GetValue 

func (t *TagDefinition) GetValue() string

GetValue returns this tag's UserValue, if that's non-empty, or its DefaultValue.

func (*TagDefinition) IsLegalValue 

func (t *TagDefinition) IsLegalValue(val string) bool

IsLegalValue returns true if val is a legal value for this tag definition. If TagDefinition.Values is empty, all values are legal.

func (*TagDefinition) SystemMustSet 

func (t *TagDefinition) SystemMustSet() bool

SystemMustSet returns true if the value for this tag must be set by the bagger at runtime. This applies to tags like Bag-Size, Payload-Oxum, Bagging-Software and a few others in the bag-info.txt file. See TagsSetBySystem for a full list.

func (*TagDefinition) ToForm 

func (t *TagDefinition) ToForm() *Form

func (*TagDefinition) ToFormattedString 

func (t *TagDefinition) ToFormattedString() string

ToFormattedString returns the tag as string in a format suitable for writing to a tag file. Following LOC's bagit.py, this function does not break lines into 79 character chunks. It prints the whole tag on a single line, replacing newlines with spaces.

func (*TagDefinition) Validate 

func (t *TagDefinition) Validate() bool

type TarredBagReader 

type TarredBagReader struct {
	// contains filtered or unexported fields
}

TarredBagReader reads a tarred BagIt file to collect metadata for bagit and ingest processing. See ProcessNextEntry() below.

func NewTarredBagReader 

func NewTarredBagReader(validator *Validator) (*TarredBagReader, error)

NewTarredBagReader creates a new TarredBagReader.

func (*TarredBagReader) Close 

func (r *TarredBagReader) Close()

Close closes the underlying reader (the file). If you neglect this call in a long-running worker process, you'll run the system out of filehandles.

func (*TarredBagReader) ScanMetadata 

func (r *TarredBagReader) ScanMetadata() error

ScanMetadata does the following:

* gets a list of all files and creates a FileRecord for each * parses all payload and tag manifests * parses all parsable tag files

func (*TarredBagReader) ScanPayload 

func (r *TarredBagReader) ScanPayload() error

ScanPayload scans the entire bag, adding checksums for all files.

type TarredBagWriter 

type TarredBagWriter struct {
	// contains filtered or unexported fields
}

func NewTarredBagWriter 

func NewTarredBagWriter(outputPath string, digestAlgs []string) *TarredBagWriter

func (*TarredBagWriter) AddFile 

func (writer *TarredBagWriter) AddFile(xFileInfo *util.ExtendedFileInfo, pathWithinArchive string) (map[string]string, error)

AddFile as a file to a tar archive. Returns a map of checksums where key is the algorithm and value is the digest. E.g. checksums["md5"] = "0987654321"

func (*TarredBagWriter) Close 

func (writer *TarredBagWriter) Close() error

func (*TarredBagWriter) DigestAlgs 

func (writer *TarredBagWriter) DigestAlgs() []string

DigestAlgs returns a list of digest algoritms that the writer calculates as it writes. E.g. ["md5", "sha256"]. These are defined in the contants package.

func (*TarredBagWriter) Open 

func (writer *TarredBagWriter) Open() error

func (*TarredBagWriter) OutputPath 

func (writer *TarredBagWriter) OutputPath() string

type UploadJob 

type UploadJob struct {
	ID                string
	PathsToUpload     []string
	StorageServiceIDs []string
	UploadOps         []*UploadOperation
	Name              string
	Errors            map[string]string
}

UploadJob represents an upload-only job, in which we may be sending multiple files to multiple targets. This type of job has no packaging or validation step.

func NewUploadJob 

func NewUploadJob() *UploadJob

func (*UploadJob) GetErrors 

func (job *UploadJob) GetErrors() map[string]string

GetErrors returns a map of errors describing why this UploadJob is not valid.

func (*UploadJob) IsDeletable 

func (job *UploadJob) IsDeletable() bool

IsDeletable describes whether users can delete this object from the database. All UploadJobs are deletable.

func (*UploadJob) ObjID 

func (job *UploadJob) ObjID() string

ObjID returns this job's object id (uuid).

func (*UploadJob) ObjName 

func (job *UploadJob) ObjName() string

ObjName returns this object's name, so names will be searchable and sortable in the DB.

func (*UploadJob) ObjType 

func (job *UploadJob) ObjType() string

ObjType returns this object's type name.

func (*UploadJob) Run 

func (job *UploadJob) Run(messageChannel chan *EventMessage) int

func (*UploadJob) String 

func (job *UploadJob) String() string

String returns a string representation of this UploadJob, which is the same as Name().

func (*UploadJob) ToForm 

func (job *UploadJob) ToForm() *Form

ToForm returns a form object through which users can edit this UploadJob.

func (*UploadJob) Validate 

func (job *UploadJob) Validate() bool

Validate returns true if this UploadJob is valid, false if not. Check the value of Errors or GetErrors() after calling this to see why validation failed.

type UploadOperation 

type UploadOperation struct {
	Errors         map[string]string  `json:"errors"`
	PayloadSize    int64              `json:"payloadSize"`
	Result         *OperationResult   `json:"result"`
	SourceFiles    []string           `json:"sourceFiles"`
	StorageService *StorageService    `json:"storageService"`
	MessageChannel chan *EventMessage `json:"-"`
}

func NewUploadOperation 

func NewUploadOperation(ss *StorageService, files []string) *UploadOperation

func (*UploadOperation) CalculatePayloadSize 

func (u *UploadOperation) CalculatePayloadSize() error

func (*UploadOperation) DoUpload 

func (u *UploadOperation) DoUpload(messageChannel chan *EventMessage) bool

func (*UploadOperation) Validate 

func (u *UploadOperation) Validate() bool

type ValidationJob 

type ValidationJob struct {
	ID              string
	BagItProfileID  string
	PathsToValidate []string
	ValidationOps   []*ValidationOperation
	Name            string
	Errors          map[string]string
}

ValidationJob is a job that only validates bags. This type of job may validate multiple bags, but it includes no package or upload operations.

func NewValidationJob 

func NewValidationJob() *ValidationJob

func (*ValidationJob) GetErrors 

func (job *ValidationJob) GetErrors() map[string]string

GetErrors returns a map of errors describing why this ValidationJob is not valid.

func (*ValidationJob) IsDeletable 

func (job *ValidationJob) IsDeletable() bool

IsDeletable describes whether users can delete this object from the database. All ValidationJobs are deletable.

func (*ValidationJob) ObjID 

func (job *ValidationJob) ObjID() string

ObjID returns this job's object id (uuid).

func (*ValidationJob) ObjName 

func (job *ValidationJob) ObjName() string

ObjName returns this object's name, so names will be searchable and sortable in the DB.

func (*ValidationJob) ObjType 

func (job *ValidationJob) ObjType() string

ObjType returns this object's type name.

func (*ValidationJob) Run 

func (job *ValidationJob) Run(messageChannel chan *EventMessage) int

Run runs this validation job, validating all PathsToValidate against the selected BagItProfile. This returns constants.ExitOK if all operations succeed. If any params are invalid or the required profile cannot be found, it returns constants.ExitUsage error because the user didn't supply valid info to complete the job. For any other runtime failure, it returns constants.ExitRuntimeError.

Note that as long as the user supplies a valid profile and paths to validate, this will attempt to validate all bags. It's possible that some bags will be valid and some will not. Check the results of each ValidationOperation.Result if you get a non-zero exit code.

func (*ValidationJob) String 

func (job *ValidationJob) String() string

String returns a string representation of this ValidationJob, which is the same as Name().

func (*ValidationJob) ToForm 

func (job *ValidationJob) ToForm() *Form

ToForm returns a form object through which users can edit this ValidationJob.

func (*ValidationJob) Validate 

func (job *ValidationJob) Validate() bool

Validate returns true if this ValidationJob is valid, false if not. Check the value of Errors or GetErrors() after calling this to see why validation failed.

type ValidationOperation 

type ValidationOperation struct {
	Errors    map[string]string `json:"errors"`
	PathToBag string            `json:"pathToBag"`
	Result    *OperationResult  `json:"result"`
}

func NewValidationOperation 

func NewValidationOperation(pathToBag string) *ValidationOperation

func (*ValidationOperation) Validate 

func (op *ValidationOperation) Validate() bool

type Validator 

type Validator struct {
	MessageChannel     chan *EventMessage `json:""`
	PathToBag          string
	Profile            *BagItProfile
	PayloadFiles       *FileMap
	PayloadManifests   *FileMap
	TagFiles           *FileMap
	TagManifests       *FileMap
	Tags               []*Tag
	UnparsableTagFiles []string
	Errors             map[string]string

	IgnoreOxumMismatch bool
	// contains filtered or unexported fields
}

func NewValidator 

func NewValidator(pathToBag string, profile *BagItProfile) (*Validator, error)

TODO: Deprecate this. New version should always use channel.

func (*Validator) AssertOxumsMatch 

func (v *Validator) AssertOxumsMatch() error

func (*Validator) ErrorJSON 

func (v *Validator) ErrorJSON() string

func (*Validator) ErrorString 

func (v *Validator) ErrorString() string

func (*Validator) GetTags 

func (v *Validator) GetTags(tagFile, tagName string) []*Tag

func (*Validator) MapForPath 

func (v *Validator) MapForPath(pathInBag string) *FileMap

func (*Validator) PayloadManifestAlgs 

func (v *Validator) PayloadManifestAlgs() ([]string, error)

func (*Validator) ScanBag 

func (v *Validator) ScanBag() error

ScanBag scans the bag's metadata and payload, recording file names, tag values, checksums, and errors. This will not run checksums on the payload if Payload-Oxum doesn't match because that's expensive. You can force checksum calculation here by setting Validator.IgnoreOxumMismatch to true. We will still flag the Oxum mismatch when you call Validate(), but you'll get to see which extra or missing files may be triggering the Oxum mismatch.

func (*Validator) TagManifestAlgs 

func (v *Validator) TagManifestAlgs() ([]string, error)

func (*Validator) Validate 

func (v *Validator) Validate() bool

Validate validates the bag and returns true if it's valid. If this returns false, check the errors in Validator.Errors. That collection should contain an exhaustive list of errors, though it maxes out at 30 checksum validation errors.

type Workflow 

type Workflow struct {
	ID                string            `json:"id"`
	BagItProfile      *BagItProfile     `json:"bagItProfile"`
	Description       string            `json:"description"`
	Errors            map[string]string `json:"-"`
	Name              string            `json:"name"`
	PackageFormat     string            `json:"packageFormat"`
	StorageServiceIDs []string          `json:"storageServiceIds"`
	StorageServices   []*StorageService `json:"storageServices"`
}

func WorkFlowFromJob 

func WorkFlowFromJob(job *Job) (*Workflow, error)

WorkflowFromJob creates a new workflow based on param job. It does not save the workflow. That's up to the caller.

When a user defines a job that does what they want, they will often convert it to a workflow, so they can run the same packaging and upload operations on a large of set of materials.

func WorkflowFromJson 

func WorkflowFromJson(pathToFile string) (*Workflow, error)

func (*Workflow) Copy 

func (w *Workflow) Copy() *Workflow

func (*Workflow) ExportJson 

func (w *Workflow) ExportJson() ([]byte, error)

ExportJson returns formatted JSON describing this workflow, including full StorageService records. Use this to export a workflow to be run on an external DART Runner server.

func (*Workflow) GetErrors 

func (w *Workflow) GetErrors() map[string]string

func (*Workflow) HasPlaintextPasswords 

func (w *Workflow) HasPlaintextPasswords() bool

HasPlaintextPasswords returns true if any of the storage services in this workflow contain plaintext passwords. The front end will warn the user about plain text passwords when they export a workflow.

func (*Workflow) IsDeletable 

func (w *Workflow) IsDeletable() bool

func (*Workflow) ObjID 

func (w *Workflow) ObjID() string

ObjID returns this w's object id (uuid).

func (*Workflow) ObjName 

func (w *Workflow) ObjName() string

ObjName returns this object's name, so names will be searchable and sortable in the DB.

func (*Workflow) ObjType 

func (w *Workflow) ObjType() string

ObjType returns this object's type name.

func (*Workflow) String 

func (w *Workflow) String() string

func (*Workflow) ToForm 

func (w *Workflow) ToForm() *Form

func (*Workflow) Validate 

func (w *Workflow) Validate() bool

type WorkflowBatch 

type WorkflowBatch struct {
	ID            string
	Workflow      *Workflow
	PathToCSVFile string
	Errors        map[string]string
}

func NewWorkflowBatch 

func NewWorkflowBatch(workflow *Workflow, pathToCSVFile string) *WorkflowBatch

func (*WorkflowBatch) GetErrors 

func (wb *WorkflowBatch) GetErrors() map[string]string

func (*WorkflowBatch) IsDeletable 

func (wb *WorkflowBatch) IsDeletable() bool

func (*WorkflowBatch) ObjID 

func (wb *WorkflowBatch) ObjID() string

ObjID returns this items's object id (uuid).

func (*WorkflowBatch) ObjName 

func (wb *WorkflowBatch) ObjName() string

ObjName returns this object's name, so names will be searchable and sortable in the DB.

func (*WorkflowBatch) ObjType 

func (wb *WorkflowBatch) ObjType() string

ObjType returns this object's type name.

func (*WorkflowBatch) String 

func (wb *WorkflowBatch) String() string

func (*WorkflowBatch) ToForm 

func (wb *WorkflowBatch) ToForm() *Form

func (*WorkflowBatch) Validate 

func (wb *WorkflowBatch) Validate() bool

type WorkflowCSVEntry 

type WorkflowCSVEntry struct {
	BagName string
	RootDir string
	Tags    []*Tag
}

WorkflowCSVEntry represents a single entry from a workflow CSV file. Bag up whatever's in RootDir and run it through the workflow.

func NewWorkflowCSVEntry 

func NewWorkflowCSVEntry(bagName, rootDir string) *WorkflowCSVEntry

NewWorkflowCSVEntry creates a new WorkflowCSVEntry.

func (*WorkflowCSVEntry) AddTag 

func (entry *WorkflowCSVEntry) AddTag(tagFile, tagName, value string)

AddTag adds a tag to this entry. Tag values from the CSV file will be written into the bag.

func (*WorkflowCSVEntry) FindTags 

func (entry *WorkflowCSVEntry) FindTags(tagFile, tagName string) []*Tag

FindTags returns all tags with the given name from the specified tag file. Returns an empty list if there are no matches. If multiple instances of a tag exist, they'll be returned in the order they were defined in the CSV file. This is in keeping with the BagIt spec that says order may be important.

type WorkflowCSVFile 

type WorkflowCSVFile struct {
	PathToFile string
	// contains filtered or unexported fields
}

WorkflowCSVFile reads a CSV file that describes what items should be run through a workflow.

func NewWorkflowCSVFile 

func NewWorkflowCSVFile(pathToFile string) (*WorkflowCSVFile, error)

NewWorkflowCSVFile creates a new WorkflowCSVFile object. See the ReadNext() function for extracting data from the file.

func (*WorkflowCSVFile) Close 

func (csvFile *WorkflowCSVFile) Close()

Close closes the CSV's underlying file object.

func (*WorkflowCSVFile) HeaderTags 

func (csvFile *WorkflowCSVFile) HeaderTags() []*Tag

HeaderTags returns the headers from the first line, converted to tag objects with empty values. This is primarily for internal use, but may be useful in debugging and error reporting.

Tags Bag-Name and Root-Directory are essentially throw-aways. The bagger won't use them. All other tags will have TagFile and TagName attributes, with an empty Value. These are used internally to help construct the tags in each WorkflowCSVEntry.

func (*WorkflowCSVFile) Headers 

func (csvFile *WorkflowCSVFile) Headers() []string

Headers returns the headers (column names) from the first line of the file. Other than the required headers Bag-Name and Root-Directory, all headers should be in the format FileName/TagName. For example, "bag-info.txt/Source-Organization".

func (*WorkflowCSVFile) ReadNext 

func (csvFile *WorkflowCSVFile) ReadNext() (*WorkflowCSVEntry, error)

ReadNext reads the next entry in the CSV file, returning a WorkflowCSVEntry. If this returns error io.EOF, it's finished reading and there are no more entries. If it returns any other error, something went wrong.

type WorkflowRunner 

type WorkflowRunner struct {
	Workflow     *Workflow
	CSVFile      *WorkflowCSVFile
	OutputDir    string
	Cleanup      bool
	Concurrency  int
	SuccessCount int
	FailureCount int
	// contains filtered or unexported fields
}

WorkflowRunner runs all jobs in CSVFile through Workflow.

func NewWorkflowRunner 

func NewWorkflowRunner(workflowFile, csvFile, outputDir string, cleanup bool, concurrency int) (*WorkflowRunner, error)

NewWorkflowRunner creates a new WorkFlowRunner object. Param workflowFile is the path the JSON file that contains a description of the workflow. Param csvFile is the path to the file contains a CSV list of directories to package. (That file also contains tag values for each package.) Param outputDir is the path to the directory into which the packages should be written. Param concurrency is the number of jobs to run in parallel.

This creates (1 * concurrency) goroutines to do the work. You probably shouldn't set concurrency too high because bagging and other forms of packaging do a lot of disk reading and writing. Concurrency significantly above 2 will probably lead to disk thrashing.

Note that this constructor is for running a workflow without a GUI. See NewWorkflowRunnerWithMessageChannel() to run a workflow that sends a running stream of status events back to the GUI.

Param workflowFile is the path to a file containing a JSON representation of the workflow you want to run.

csvFile is the path to the csv file containing info about what items to bag and what tag values to apply to each bag. See https://aptrust.github.io/dart-docs/users/workflows/batch_jobs/ for a description of the csv file and an example of what it looks like. This should be an absolute path.

outputDir is the path the output directory where DART will write the bags it creates. This should be an absolute path.

cleanup describes whether or not DART should delete the bags it creates after successful upload.

concurrency describes how many jobs you want to run at once. In some environments, you can set this to 2 or more to get better throughput. (E.g. if you're reading from and writing to network attached storage on a high-performance file server). But in most cases where you're reading from and writing to a single local disk, you'll want to set this to 1.

func NewWorkflowRunnerWithMessageChannel 

func NewWorkflowRunnerWithMessageChannel(workflowID, csvFile, outputDir string, cleanup bool, messageChannel chan *EventMessage) (*WorkflowRunner, error)

NewWorkflowRunnerWithMessageChannel creates a new workflow runner that sends status events back to the GUI. These events tell the user which files are being packaged and validated, and they update the packaging, validation, and upload progress bars.

This constructor force concurrency to 1 because at the moment, the UI is capable of reporting on only one process at a time.

Param workflowID is the id of the workflow you want to run.

csvFile is the path to the csv file containing info about what items to bag and what tag values to apply to each bag. See https://aptrust.github.io/dart-docs/users/workflows/batch_jobs/ for a description of the csv file and an example of what it looks like. This should be an absolute path.

outputDir is the path the output directory where DART will write the bags it creates. This should be an absolute path.

cleanup describes whether or not DART should delete the bags it creates after successful upload.

messageChannel is a channel to send status/progress messages back to the front end, so the user can see what's happening.

func (*WorkflowRunner) Run 

func (r *WorkflowRunner) Run() int

Run runs the workflow, writing one line of JSON output per job to STDOUT. The output is a serialized JobResult object. Errors will be written to STDERR, though there **should** also be JobResult written to STDOUT if a job fails.

func (*WorkflowRunner) SetStdErr 

func (r *WorkflowRunner) SetStdErr(b *bytes.Buffer)

SetStdErr causes messages to be written to buffer b instead of os.Stderr. See the doc comments for SetStdOut above for why this exists.

func (*WorkflowRunner) SetStdOut 

func (r *WorkflowRunner) SetStdOut(b *bytes.Buffer)

SetStdOut causes messages to be written to buffer b instead of os.Stdout. We use this in testing to capture output that would go to os.Stdout. On Windows, redirecting os.Stdout to a pipe using os.Pipe() causes some writes to hang forever. This happens deep inside Syscall6, so there's nothing we can do about it. The pipe works fine on Linux and MacOS, but not on Windows. Hence, this.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.