objects

package
v2.6.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 27, 2019 License: Apache-2.0 Imports: 30 Imported by: 1

Documentation

Index

Constants

View Source 
const (
	ErrMessageGPGVerificationFailed = "error while verifying GPG signature"
)

Some common values for ListEntriesError.Message that are always accompanied by an Inner error.

View Source 
const StatusSwiftRateLimit = 498

StatusSwiftRateLimit is the non-standard HTTP status code used by Swift to indicate Too Many Requests.

Variables

View Source 
var ErrListAllFilesNotSupported = &ListEntriesError{
	Message: "ListAllFiles not supported by this source",
}

ErrListAllFilesNotSupported is returned by ListAllFiles() for sources that do not support it.

Functions

This section is empty.

Types

type AgeSpec 

type AgeSpec time.Duration

AgeSpec is a timestamp that is deserialized from a duration in the format "<value> <unit>", e.g. "4 days" or "2 weeks".

func (*AgeSpec) UnmarshalYAML 

func (a *AgeSpec) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type AuthPassword 

type AuthPassword string

AuthPassword contains the password for some OpenStack Authentication credentials.

func (*AuthPassword) UnmarshalYAML 

func (p *AuthPassword) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type CleanupConfiguration 

type CleanupConfiguration struct {
	Strategy CleanupStrategy `yaml:"strategy"`
}

CleanupConfiguration contains the "cleanup" section of a JobConfiguration.

type CleanupStrategy 

type CleanupStrategy string

CleanupStrategy is an enum of legal values for the jobs[].cleanup.strategy configuration option. 

const (
	//KeepUnknownFiles is the default cleanup strategy.
	KeepUnknownFiles CleanupStrategy = ""
	//DeleteUnknownFiles is another strategy.
	DeleteUnknownFiles CleanupStrategy = "delete"
	//ReportUnknownFiles is another strategy.
	ReportUnknownFiles CleanupStrategy = "report"
)

type Configuration 

type Configuration struct {
	Swift        SwiftLocation `yaml:"swift"`
	WorkerCounts struct {
		Transfer uint
	} `yaml:"workers"`
	Statsd     StatsdConfiguration `yaml:"statsd"`
	JobConfigs []JobConfiguration  `yaml:"jobs"`
	Jobs       []*Job              `yaml:"-"`
}

Configuration contains the contents of the configuration file.

func ReadConfiguration 

func ReadConfiguration(path string) (*Configuration, []error)

ReadConfiguration reads the configuration file.

type DebianSource 

type DebianSource struct {
	//options from config file
	URLString                string   `yaml:"url"`
	ClientCertificatePath    string   `yaml:"cert"`
	ClientCertificateKeyPath string   `yaml:"key"`
	ServerCAPath             string   `yaml:"ca"`
	Distributions            []string `yaml:"dist"`
	Architectures            []string `yaml:"arch"`
	VerifySignature          *bool    `yaml:"verify_signature"`
	// contains filtered or unexported fields
}

DebianSource is a URLSource containing a Debian repository. This type reuses the Validate() and Connect() logic of URLSource, but adds a custom scraping implementation that reads the Debian repository metadata instead of relying on directory listings.

func (*DebianSource) Connect 

func (s *DebianSource) Connect() error

Connect implements the Source interface.

func (*DebianSource) GetFile 

func (s *DebianSource) GetFile(directoryPath string, requestHeaders schwift.ObjectHeaders) (body io.ReadCloser, sourceState FileState, err error)

GetFile implements the Source interface.

func (*DebianSource) ListAllFiles 

func (s *DebianSource) ListAllFiles() ([]FileSpec, *ListEntriesError)

ListAllFiles implements the Source interface.

func (*DebianSource) ListEntries 

func (s *DebianSource) ListEntries(directoryPath string) ([]FileSpec, *ListEntriesError)

ListEntries implements the Source interface.

func (*DebianSource) Validate 

func (s *DebianSource) Validate(name string) []error

Validate implements the Source interface.

type Directory 

type Directory struct {
	Job  *Job
	Path string
	//RetryCounter is increased by the actors.Scraper when scraping of this
	//directory fails.
	RetryCounter uint
}

Directory describes a directory on the source side which can be scraped.

type ExpirationConfiguration 

type ExpirationConfiguration struct {
	EnabledIn    *bool  `yaml:"enabled"`
	Enabled      bool   `yaml:"-"`
	DelaySeconds uint32 `yaml:"delay_seconds"`
}

ExpirationConfiguration contains the "expiration" section of a JobConfiguration.

type File 

type File struct {
	Job  *Job
	Spec FileSpec
}

File describes a single file which is mirrored as part of a Job.

func (File) PerformTransfer 

func (f File) PerformTransfer() (TransferResult, int64)

PerformTransfer transfers this file from the source to the target. It returns the TransferResult (which indicates if the transfer finished successfully) and the number of bytes transferred.

func (File) TargetObject 

func (f File) TargetObject() *schwift.Object

TargetObject returns the object corresponding to this file in the target container.

type FileSpec 

type FileSpec struct {
	Path        string
	IsDirectory bool
	//only set for files in Swift sources (otherwise nil)
	LastModified *time.Time
	//only set for symlinks (refers to a path below the ObjectPrefix in the same container)
	SymlinkTargetPath string
	//results of GET on this file
	Contents []byte
	Headers  http.Header
}

FileSpec contains metadata for a File. The only required field is Path. Sources that download some files early (during scraping) can pass the downloaded contents and metadata in the remaining fields of the FileSpec to avoid double download.

type FileState 

type FileState struct {
	Etag         string
	LastModified string
	SizeBytes    int64      //-1 if not known
	ExpiryTime   *time.Time //nil if not set
	//the following fields are only used in `sourceState`, not `targetState`
	SkipTransfer bool
	ContentType  string
}

FileState is used by Source.GetFile() to describe the state of a file.

type Job 

type Job struct {
	Source     Source
	Target     *SwiftLocation
	Matcher    Matcher
	Segmenting *SegmentingConfiguration
	Expiration ExpirationConfiguration
	Cleanup    CleanupConfiguration
}

Job describes a transfer job at runtime.

type JobConfiguration 

type JobConfiguration struct {
	//basic options
	Source SourceUnmarshaler `yaml:"from"`
	Target *SwiftLocation    `yaml:"to"`
	//behavior options
	ExcludePattern       string                   `yaml:"except"`
	IncludePattern       string                   `yaml:"only"`
	ImmutableFilePattern string                   `yaml:"immutable"`
	Match                MatchConfiguration       `yaml:"match"`
	Segmenting           *SegmentingConfiguration `yaml:"segmenting"`
	Expiration           ExpirationConfiguration  `yaml:"expiration"`
	Cleanup              CleanupConfiguration     `yaml:"cleanup"`
	// contains filtered or unexported fields
}

JobConfiguration describes a transfer job in the configuration file.

func (JobConfiguration) Compile 

func (cfg JobConfiguration) Compile(name string, swift SwiftLocation) (job *Job, errors []error)

Compile validates the given JobConfiguration, then creates and prepares a Job from it.

type ListEntriesError 

type ListEntriesError struct {
	//the location of the directory (e.g. an URL)
	Location string
	//error message (either of those is optional, but at least one must be set)
	Message string
	Inner   error
}

ListEntriesError is an error that occurs while scraping a directory.

func (ListEntriesError) FullMessage 

func (e ListEntriesError) FullMessage() string

FullMessage returns the full error message for this error. This merges the .Messages with the .Inner.Error().

type MatchConfiguration 

type MatchConfiguration struct {
	NotOlderThan         *AgeSpec `yaml:"not_older_than"`
	SimplisticComparison *bool    `yaml:"simplistic_comparison"`
}

MatchConfiguration contains the "match" section of a JobConfiguration.

type MatchError 

type MatchError struct {
	Path   string
	Reason string
}

MatchError is returned by the functions on type Matcher.

func (MatchError) Error 

func (e MatchError) Error() string

Error implements the builtin/error interface.

type Matcher 

type Matcher struct {
	ExcludeRx            *regexp.Regexp //pointers because nil signifies absence
	IncludeRx            *regexp.Regexp
	ImmutableFileRx      *regexp.Regexp
	NotOlderThan         *time.Time
	SimplisticComparison *bool
}

Matcher determines if files shall be included or excluded in a transfer.

func (Matcher) Check 

func (m Matcher) Check(path string, lastModified *time.Time) error

Check checks whether the directory at `path` should be scraped, or whether the file at `path` should be transferred. If not, a MatchError is returned that contains the concerning `path` and a human-readable message describing the exclusion.

If `path` is a directory, `path` must have a trailing slash. If `path` is a file, `path` must not have a trailing slash.

For directories, `lastModified` must be nil. For files, `lastModified` may be non-nil and will then be checked against `m.NotOlderThan`.

func (Matcher) CheckFile 

func (m Matcher) CheckFile(spec FileSpec) error

CheckFile is like CheckRecursive, but uses `spec.Path` and appends a slash if `spec.IsDirectory`.

func (Matcher) CheckRecursive 

func (m Matcher) CheckRecursive(path string, lastModified *time.Time) error

CheckRecursive is like Check(), but also checks each directory along the way as well.

For example, CheckRecursive("a/b/c") calls Check("a/"), "Check("a/b/") and Check("a/b/c").

type SegmentingConfiguration 

type SegmentingConfiguration struct {
	MinObjectSize uint64 `yaml:"min_bytes"`
	SegmentSize   uint64 `yaml:"segment_bytes"`
	ContainerName string `yaml:"container"`
	//Container is initialized by JobConfiguration.Compile().
	Container *schwift.Container `yaml:"-"`
}

SegmentingConfiguration contains the "segmenting" section of a JobConfiguration.

type Source 

type Source interface {
	//Validate reports errors if this source is malspecified.
	Validate(name string) []error
	//Connect performs source-specific one-time setup.
	Connect() error
	//ListAllFiles returns all files in the source (as paths relative to the
	//source's root). If this returns ErrListAllFilesNotSupported, ListEntries
	//must be used instead.
	ListAllFiles() ([]FileSpec, *ListEntriesError)
	//ListEntries returns all files and subdirectories at this path in the
	//source. Each result value must have a "/" prefix for subdirectories, or
	//none for files.
	ListEntries(directoryPath string) ([]FileSpec, *ListEntriesError)
	//GetFile retrieves the contents and metadata for the file at the given path
	//in the source. The `headers` map contains additional HTTP request headers
	//that shall be passed to the source in the GET request.
	GetFile(directoryPath string, headers schwift.ObjectHeaders) (body io.ReadCloser, sourceState FileState, err error)
}

Source describes a place from which files can be fetched.

type SourceUnmarshaler 

type SourceUnmarshaler struct {
	// contains filtered or unexported fields
}

SourceUnmarshaler provides a yaml.Unmarshaler implementation for the Source interface.

func (*SourceUnmarshaler) UnmarshalYAML 

func (u *SourceUnmarshaler) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type StatsdConfiguration 

type StatsdConfiguration struct {
	HostName string `yaml:"hostname"`
	Port     int    `yaml:"port"`
	Prefix   string `yaml:"prefix"`
}

StatsdConfiguration contains the configuration options relating to StatsD metric emission.

type SwiftLocation 

type SwiftLocation struct {
	AuthURL           string       `yaml:"auth_url"`
	UserName          string       `yaml:"user_name"`
	UserDomainName    string       `yaml:"user_domain_name"`
	ProjectName       string       `yaml:"project_name"`
	ProjectDomainName string       `yaml:"project_domain_name"`
	Password          AuthPassword `yaml:"password"`
	RegionName        string       `yaml:"region_name"`
	ContainerName     string       `yaml:"container"`
	ObjectNamePrefix  string       `yaml:"object_prefix"`
	//configuration for Validate()
	ValidateIgnoreEmptyContainer bool `yaml:"-"`
	//Account and Container is filled by Connect(). Container will be nil if ContainerName is empty.
	Account   *schwift.Account   `yaml:"-"`
	Container *schwift.Container `yaml:"-"`
	//FileExists is filled by DiscoverExistingFiles(). The keys are object names
	//including the ObjectNamePrefix, if any.
	FileExists map[string]bool `yaml:"-"`
}

SwiftLocation contains all parameters required to establish a Swift connection. It implements the Source interface, but is also used on the target side.

func (*SwiftLocation) Connect 

func (s *SwiftLocation) Connect() error

Connect implements the Source interface. It establishes the connection to Swift.

func (*SwiftLocation) DiscoverExistingFiles 

func (s *SwiftLocation) DiscoverExistingFiles(matcher Matcher) error

DiscoverExistingFiles finds all objects that currently exist in this location (i.e. in this Swift container below the given object name prefix) and fills s.FileExists accordingly.

The given Matcher is used to find out which files are to be considered as belonging to the transfer job in question.

func (*SwiftLocation) GetFile 

func (s *SwiftLocation) GetFile(path string, requestHeaders schwift.ObjectHeaders) (io.ReadCloser, FileState, error)

GetFile implements the Source interface.

func (*SwiftLocation) ListAllFiles 

func (s *SwiftLocation) ListAllFiles() ([]FileSpec, *ListEntriesError)

ListAllFiles implements the Source interface.

func (*SwiftLocation) ListEntries 

func (s *SwiftLocation) ListEntries(path string) ([]FileSpec, *ListEntriesError)

ListEntries implements the Source interface.

func (*SwiftLocation) ObjectAtPath 

func (s *SwiftLocation) ObjectAtPath(path string) *schwift.Object

ObjectAtPath returns an Object instance for the object at the given path (below the ObjectNamePrefix, if any) in this container.

func (SwiftLocation) Validate 

func (s SwiftLocation) Validate(name string) []error

Validate returns an empty list only if all required credentials are present.

type TransferResult 

type TransferResult uint

TransferResult is the return type for PerformTransfer(). 

const (
	//TransferSuccess means that the file was newer on the source and was sent
	//to the target.
	TransferSuccess TransferResult = iota
	//TransferSkipped means that the file was the same on both sides and
	//nothing was transferred.
	TransferSkipped
	//TransferFailed means that an error occurred and was logged.
	TransferFailed
)

type URLSource 

type URLSource struct {
	URLString string   `yaml:"url"`
	URL       *url.URL `yaml:"-"`
	//auth options
	ClientCertificatePath    string       `yaml:"cert"`
	ClientCertificateKeyPath string       `yaml:"key"`
	ServerCAPath             string       `yaml:"ca"`
	HTTPClient               *http.Client `yaml:"-"`
	//transfer options
	SegmentingIn *bool  `yaml:"segmenting"`
	Segmenting   bool   `yaml:"-"`
	SegmentSize  uint64 `yaml:"segment_bytes"`
}

URLSource describes a source that's accessible via HTTP.

func (*URLSource) Connect 

func (u *URLSource) Connect() error

Connect implements the Source interface.

func (URLSource) GetFile 

func (u URLSource) GetFile(directoryPath string, requestHeaders schwift.ObjectHeaders) (io.ReadCloser, FileState, error)

GetFile implements the Source interface.

func (URLSource) ListAllFiles 

func (u URLSource) ListAllFiles() ([]FileSpec, *ListEntriesError)

ListAllFiles implements the Source interface.

func (URLSource) ListEntries 

func (u URLSource) ListEntries(directoryPath string) ([]FileSpec, *ListEntriesError)

ListEntries implements the Source interface.

func (*URLSource) Validate 

func (u *URLSource) Validate(name string) (result []error)

Validate implements the Source interface.

type YumSource 

type YumSource struct {
	//options from config file
	URLString                string   `yaml:"url"`
	ClientCertificatePath    string   `yaml:"cert"`
	ClientCertificateKeyPath string   `yaml:"key"`
	ServerCAPath             string   `yaml:"ca"`
	Architectures            []string `yaml:"arch"`
	VerifySignature          *bool    `yaml:"verify_signature"`
	// contains filtered or unexported fields
}

YumSource is a URLSource containing a Yum repository. This type reuses the Validate() and Connect() logic of URLSource, but adds a custom scraping implementation that reads the Yum repository metadata instead of relying on directory listings.

func (*YumSource) Connect 

func (s *YumSource) Connect() error

Connect implements the Source interface.

func (*YumSource) GetFile 

func (s *YumSource) GetFile(directoryPath string, requestHeaders schwift.ObjectHeaders) (body io.ReadCloser, sourceState FileState, err error)

GetFile implements the Source interface.

func (*YumSource) ListAllFiles 

func (s *YumSource) ListAllFiles() ([]FileSpec, *ListEntriesError)

ListAllFiles implements the Source interface.

func (*YumSource) ListEntries 

func (s *YumSource) ListEntries(directoryPath string) ([]FileSpec, *ListEntriesError)

ListEntries implements the Source interface.

func (*YumSource) Validate 

func (s *YumSource) Validate(name string) []error

Validate implements the Source interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.