server

package
v0.0.0-...-6a344f3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 9, 2025 License: MIT Imports: 37 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	AlertTypeGood = "GOOD"
	AlertTypeBad  = "BAD"
)

Alert.Type values

View Source 
const (
	FilenameAPIDB     = "api-keys.db"
	FilenameProjectDB = "projects.db"
)

Database filenames

View Source 
const (
	ExpirationUnitDefault = 0
	ExpirationUnitFile    = 1
	ExpirationUnitDay     = 2
)

"units" used in "every xx" expressions

View Source 
const (
	FileStatusNew       = "new"
	FileStatusQueued    = "queued"
	FileStatusUploading = "uploading"
	FileStatusUploaded  = "uploaded"
)

FileStatus list all possible status for a file in WaitList and ProjectDB

View Source 
const (
	MessageSuccess = "SUCCESS"
	MessageFailure = "FAILURE"
)

SUCCESS & FAILURE will end a client connection (no?)

View Source 
const (
	MessageError   = "ERROR"
	MessageWarning = "WARNING"
	MessageInfo    = "INFO"
	MessageTrace   = "TRACE"
	MessageNoop    = "NOOP" // MessageNoop is used for keep-alive messages
)

Message types

View Source 
const (
	MessagePrintTime    = true
	MessagePrintNoTime  = false
	MessagePrintTopic   = true
	MessagePrintNoTopic = false
)

Message.Print options

View Source 
const (
	SwiftObjectSealed    = "sealed"
	SwiftObjectUnsealing = "unsealing"
	SwiftObjectUnsealed  = "unsealed"
)

Possible object availability

View Source 
const CheckExpireEvery = 15 * time.Minute

CheckExpireEvery is the delay between each expire task

View Source 
const DiffAlertDisableIfLessThan = 100 * 1024 // 100KB

DiffAlertDisableIfLessThan is the minimum size of a file to be checked for diff alert

View Source 
const DiffAlertThresholdPerc = 20

DiffAlertThresholdPerc is the percentage of difference between two files to trigger an alert

View Source 
const FileStorageName = "files"

FileStorageName is the name of the storage sub-directory where local files are stored

View Source 
const KeepAliveDelayDays = 1

KeepAliveDelayDays is the number of days between each keep-alive/stats report

View Source 
const LogHistorySize = 5000

LogHistorySize is the maximum number of messages in app log history

View Source 
const (
	MsgGlob = ".GLOBAL"
)

Messages generic topics

View Source 
const NoBackupAlertSchedule = 1 * time.Hour

NoBackupAlertSchedule is the delay between each NoBackupAlert check

View Source 
const ProjectDefaultBackupEvery = 24 * time.Hour

ProjectDefaultBackupEvery is the approximate delay between each backup of a project (used by no-backup alerts)

View Source 
const ProjectNewestVersion = 1

ProjectNewestVersion is needed because each project record have a version and may be upgraded as application version goes. (see Upgrade() below) v0: original v1: added SchemaVersion + BackupEvery + LastNoBackupAlert

View Source 
const (
	PusherTypeMulch = "mulch"
)

Pusher types

View Source 
const QueueScanDelay = 1 * time.Minute

QueueScanDelay is the delay between consecutive queue scans

View Source 
const QueueStableDelay = 1*time.Minute + 30*time.Second

QueueStableDelay determine how long a file should stay the same (mtime+size) to be considered stable.

View Source 
const ReEncryptDelay = 1 * time.Hour

ReEncryptDelay is the delay after which a file will be re-encrypted after being decrypted

View Source 
const RetrievedStorageName = "retrieved"

RetrievedStorageName is the storage subfolder where retrieved files are stored

View Source 
const RetryDelay = 15 * time.Minute

RetryDelay is used when an upload/move/delete failed

View Source 
const SelfBackupDelay = 3 * time.Hour

SelfBackupDelay is the delay between each self-backup

Variables

This section is empty.

Functions

func AreDirsOnSameDevice 

func AreDirsOnSameDevice(path1, path2 string) (bool, error)

AreDirsOnSameDevice return true of both directories are on the same disk/device/partition, so a file can be moved from path1 to path2 without the need of data copy

func CreateDirIfNeeded 

func CreateDirIfNeeded(path string) error

CreateDirIfNeeded will create the directory if it does not exists

func IsDir 

func IsDir(path string) (bool, error)

IsDir return true if path exists and is a directory

func NewEncryptionsConfigFromToml 

func NewEncryptionsConfigFromToml(tEncryptions []*tomlEncryption, autogenerate bool, rand *rand.Rand, configPath string) (map[string]*EncryptionConfig, error)

NewEncryptionsConfigFromToml will "parse" TOML encryptions

func NewPushersConfigFromToml 

func NewPushersConfigFromToml(tDestinations []*tomlPushDestination) (map[string]*PusherConfig, error)

NewPushersConfigFromToml will "parse" TOML push destinations

func RandString 

func RandString(n int, rand *rand.Rand) string

RandString generate a random string of A-Za-z0-9 runes

Types

type APIConfig 

type APIConfig struct {
	Listen string
}

APIConfig describes API server configuration

type APIKey 

type APIKey struct {
	Comment string
	Key     string
}

APIKey describes an API key

type APIKeyDatabase 

type APIKeyDatabase struct {
	// contains filtered or unexported fields
}

APIKeyDatabase describes a persistent API Key database

func NewAPIKeyDatabase 

func NewAPIKeyDatabase(filename string, log *Log, rand *rand.Rand) (*APIKeyDatabase, error)

NewAPIKeyDatabase creates a new API key database

func (*APIKeyDatabase) AddNew 

func (db *APIKeyDatabase) AddNew(comment string) (*APIKey, error)

AddNew generates a new key and adds it to the database

func (*APIKeyDatabase) GetPath 

func (db *APIKeyDatabase) GetPath() string

GetPath of the database

func (*APIKeyDatabase) IsValidKey 

func (db *APIKeyDatabase) IsValidKey(key string) (bool, *APIKey)

IsValidKey return true if the key exists in the database (and returns the key as the second return value)

func (*APIKeyDatabase) List 

func (db *APIKeyDatabase) List() []*APIKey

List returns all keys NOTE: This function signature may change in the future, since the current one does not offer much safety to interal structures.

func (*APIKeyDatabase) Save 

func (db *APIKeyDatabase) Save() error

Save the database on the disk

func (*APIKeyDatabase) SaveToWriter 

func (db *APIKeyDatabase) SaveToWriter(writer io.Writer) error

SaveToWriter save the database to a io.Writer

type Alert 

type Alert struct {
	Type    string
	Subject string
	Content string
}

Alert are used only for background big "events" (seed download failure, vm autorebuild failure, etc)

type AlertSender 

type AlertSender struct {
	// contains filtered or unexported fields
}

AlertSender will be attached to the application

func NewAlertSender 

func NewAlertSender(configPath string, log *Log) (*AlertSender, error)

NewAlertSender creates a new AlertSender

func (*AlertSender) Send 

func (sender *AlertSender) Send(alert *Alert) error

Send an alert using all alert scripts (etc/alerts/*.sh)

type App 

type App struct {
	StartTime   time.Time
	Config      *AppConfig
	ProjectDB   *ProjectDatabase
	WaitList    *WaitList
	Uploader    *Uploader
	Encrypter   *Encrypter
	Swift       *Swift
	Log         *Log
	LogHistory  *LogHistory
	AlertSender *AlertSender
	Stats       *Stats
	APIKeysDB   *APIKeyDatabase
	Rand        *rand.Rand
	MuxAPI      *http.ServeMux
	// contains filtered or unexported fields
}

App describes an application

func NewApp 

func NewApp(config *AppConfig, rand *rand.Rand) (*App, error)

NewApp create a new application

func (*App) AddRoute 

func (app *App) AddRoute(route *Route) error

AddRoute adds a new route to the given route muxer

func (*App) DecryptFile 

func (app *App) DecryptFile(srcFilename string, dstFilename string) error

DecryptFile decrypt a file

func (*App) DecryptFileInPlace 

func (app *App) DecryptFileInPlace(filename string, log *Log) error

DecryptFileInPlace decrypt a file in place (using a temp file)

func (*App) Init 

func (app *App) Init(trace bool, pretty bool) error

Init the application

func (*App) LocalStoragePath 

func (app *App) LocalStoragePath(dir string, filename string) (string, error)

LocalStoragePath builds a path based on LocalStoragePath, and will create the (last) directory if needed

func (*App) MakeFileAvailable 

func (app *App) MakeFileAvailable(file *File) (common.APIFileStatus, error)

MakeFileAvailable will do all the work needed to make the file available (downloadable and decypted) This action is asynchronous, the function will return current file status with an ETA. This function is designed to be called repetitively.

func (*App) MoveFileToStorage 

func (app *App) MoveFileToStorage(file *File) error

MoveFileToStorage will move a file from the queue to our storage

func (*App) Run 

func (app *App) Run()

Run will start the app servers (foreground)

func (*App) RunKeepAliveStats 

func (app *App) RunKeepAliveStats(daysInterval int)

RunKeepAliveStats will send a keepalive alert with stats every X days

func (*App) ScheduleScan 

func (app *App) ScheduleScan()

ScheduleScan of the WaitList (block, will never return)

func (*App) ScheduleSelfBackup 

func (app *App) ScheduleSelfBackup()

ScheduleSelfBackup will backup our databases on a regular basis

func (*App) SelfRestore 

func (app *App) SelfRestore() error

SelfRestore will retrieve database backups from the self_backup_container

func (*App) Status 

func (app *App) Status() (*common.APIStatus, error)

Status returns informations about server

func (*App) UploadAndStore 

func (app *App) UploadAndStore(projectName string, file *File) error

UploadAndStore will upload and store a file

type AppConfig 

type AppConfig struct {
	QueuePath           string
	LocalStoragePath    string
	TempPath            string
	NumUploaders        int
	NumEncrypters       int
	SelfBackupContainer string
	Expiration          *ExpirationConfig
	Swift               *SwiftConfig
	API                 *APIConfig
	Containers          []*Container
	Pushers             map[string]*PusherConfig
	Encryptions         map[string]*EncryptionConfig
	// contains filtered or unexported fields
}

AppConfig describes the general configuration of an App

func NewAppConfigFromTomlFile 

func NewAppConfigFromTomlFile(configPath string, autogenKey bool, rand *rand.Rand) (*AppConfig, error)

NewAppConfigFromTomlFile return a AppConfig using a TOML file in configPath

func (*AppConfig) GetDefaultEncryption 

func (conf *AppConfig) GetDefaultEncryption() *EncryptionConfig

GetDefaultEncryption return the default encryption, or nil if none

func (*AppConfig) GetEncryption 

func (conf *AppConfig) GetEncryption(name string) (*EncryptionConfig, error)

GetEncryption return an encryption by name, or an error if not found

type Container 

type Container struct {
	Name     string
	CostExpr *govaluate.EvaluableExpression
}

Container describes a storage Container

func NewContainersConfigFromToml 

func NewContainersConfigFromToml(tContainers []*tomlContainer) ([]*Container, error)

NewContainersConfigFromToml return a list of Container based on TOML [[container]] settings

func (*Container) Cost 

func (c *Container) Cost(size int64, duration time.Duration) (float64, error)

Cost of a file in this container, based on its size and storage duration

type Encrypt 

type Encrypt struct {
	// input parameters
	EncryptionConfig *EncryptionConfig
	Filename         string

	// output chan
	Result chan error
}

func NewEncrypt 

func NewEncrypt(encryptionConfig *EncryptionConfig, filename string) *Encrypt

NewEncrypt initialize a new instance

type Encrypter 

type Encrypter struct {
	NumWorkers int
	Channel    chan *Encrypt
	Log        *Log
	Rand       *rand.Rand
	Status     []string
}

func NewEncrypter 

func NewEncrypter(numWorkers int, log *Log, rand *rand.Rand) *Encrypter

NewEncrypter initialize a new instance

func (*Encrypter) Start 

func (enc *Encrypter) Start()

Start the Encrypter (run workers)

type EncryptionConfig 

type EncryptionConfig struct {
	Name     string
	Filename string
	Key      []byte
	Default  bool
}

func (*EncryptionConfig) EncryptFile 

func (enc *EncryptionConfig) EncryptFile(srcFilename string, dstFilename string, rand *rand.Rand) error

EncryptFile encrypt a file

func (*EncryptionConfig) EncryptFileInPlace 

func (enc *EncryptionConfig) EncryptFileInPlace(filename string, rand *rand.Rand, log *Log) error

EncryptFileInPlace encrypt a file in place (using a temp file)

type Expiration 

type Expiration struct {
	Custom        bool // was customized? (versus "cloned from config")
	FileCount     int
	ReferenceDate time.Time
	Lines         []ExpirationLine
}

Expiration host an expiration cycle, each project has two (local, remote)

func ParseExpiration 

func ParseExpiration(linesIn []string) (Expiration, error)

ParseExpiration will parse an array of strings and return an Expiration

func (*Expiration) GetNext 

func (exp *Expiration) GetNext(modTime time.Time) ExpirationResult

GetNext return the next expiration duration

func (*Expiration) String 

func (exp *Expiration) String() string

type ExpirationConfig 

type ExpirationConfig struct {
	Local  Expiration
	Remote Expiration
}

ExpirationConfig is the expiration configuration at application level

func NewExpirationConfigFromToml 

func NewExpirationConfigFromToml(tConfig *tomlExpiration) (*ExpirationConfig, error)

NewExpirationConfigFromToml will check tomlExpiration and create an ExpirationConfig

type ExpirationLine 

type ExpirationLine struct {
	Original  string
	Keep      time.Duration
	Every     int
	EveryUnit int
}

ExpirationLine is a line of expiration inside an Expiration

type ExpirationResult 

type ExpirationResult struct {
	Original string
	Keep     time.Duration
}

ExpirationResult is the output of the whole Expiration thing :) (see GetNext)

type File 

type File struct {
	Filename        string
	Path            string
	ModTime         time.Time
	Size            int64
	AddedAt         time.Time
	Status          string
	ExpireLocal     time.Time // expiration date
	ExpireRemote    time.Time // (same)
	ExpireLocalOrg  string    // original expire string
	ExpireRemoteOrg string    // (same)
	RemoteKeep      time.Duration
	ExpiredLocal    bool
	ExpiredRemote   bool
	Container       string
	Cost            float64
	Encrypted       bool
	ReEncryptDate   time.Time
	RetrievedPath   string
	RetrievedDate   time.Time
	// contains filtered or unexported fields
}

File is a file in our DB (final leaf)

func (*File) GetLocalPath 

func (file *File) GetLocalPath(app *App) (string, error)

GetLocalPath will return the hypothetical local path of the file (valid only if file is available)

func (*File) GetPusher 

func (file *File) GetPusher(destination string) Pusher

func (*File) ProjectName 

func (file *File) ProjectName() string

ProjectName is a small helper to return the dir/project name of the file

type FileMap 

type FileMap map[string]*File

FileMap is a map of File

type FileMtimeSort 

type FileMtimeSort []*File

FileMtimeSort is a slice of File

func NewFileMtimeSort 

func NewFileMtimeSort(files FileMap) FileMtimeSort

NewFileMtimeSort crate a new FileMtimeSort from a File map

func (FileMtimeSort) Len 

func (d FileMtimeSort) Len() int

Len return the len of the slice (needed by sort.Interface)

func (FileMtimeSort) Less 

func (d FileMtimeSort) Less(i, j int) bool

Less returns true if ModTime of i is before j (needed by sort.Interface)

func (FileMtimeSort) Swap 

func (d FileMtimeSort) Swap(i, j int)

Swap entries (needed by sort.Interface)

type Log 

type Log struct {
	// contains filtered or unexported fields
}

Log host logs for the application

func NewLog 

func NewLog(trace bool, pretty bool, history *LogHistory) *Log

NewLog creates a new Log

func (*Log) Error 

func (log *Log) Error(topic, message string)

Error sends a MessageError Message

func (*Log) Errorf 

func (log *Log) Errorf(topic, format string, args ...interface{})

Errorf sends a formated string MessageError Message

func (*Log) Failure 

func (log *Log) Failure(topic, message string)

Failure sends an MessageFailure Message

func (*Log) Failuref 

func (log *Log) Failuref(topic, format string, args ...interface{})

Failuref sends a formated string MessageFailure Message

func (*Log) Info 

func (log *Log) Info(topic, message string)

Info sends an MessageInfo Message

func (*Log) Infof 

func (log *Log) Infof(topic, format string, args ...interface{})

Infof sends a formated string MessageInfo Message

func (*Log) Log 

func (log *Log) Log(message *Message)

Log is a low-level function for sending a Message

func (*Log) Success 

func (log *Log) Success(topic, message string)

Success sends an MessageSuccess Message

func (*Log) Successf 

func (log *Log) Successf(topic, format string, args ...interface{})

Successf sends a formated string MessageSuccess Message

func (*Log) Trace 

func (log *Log) Trace(topic, message string)

Trace sends an MessageTrace Message

func (*Log) Tracef 

func (log *Log) Tracef(topic, format string, args ...interface{})

Tracef sends a formated string MessageTrace Message

func (*Log) Warning 

func (log *Log) Warning(topic, message string)

Warning sends a MessageWarning Message

func (*Log) Warningf 

func (log *Log) Warningf(topic, format string, args ...interface{})

Warningf sends a formated string MessageWarning Message

type LogHistory 

type LogHistory struct {
	// contains filtered or unexported fields
}

LogHistory stores messages in a limited size double chain list

func NewLogHistory 

func NewLogHistory(elems int) *LogHistory

NewLogHistory will create and initialize a new log message history

func (*LogHistory) Dump 

func (lh *LogHistory) Dump()

Dump all logs in the buffer (temporary test)

func (*LogHistory) Push 

func (lh *LogHistory) Push(message *Message)

Push a new message in the list

func (*LogHistory) Search 

func (lh *LogHistory) Search(maxMessages int, topic string) []*Message

Search return an array of messages (latest messages, up to maxMessages, for a specific topic)

type Message 

type Message struct {
	Time    time.Time `json:"time"`
	Type    string    `json:"type"`
	Topic   string    `json:"topic"`
	Message string    `json:"message"`
}

Message describe a message between client and server

func NewMessage 

func NewMessage(mtype string, topic string, message string) *Message

NewMessage creates a new Message instance

func (*Message) MatchTarget 

func (message *Message) MatchTarget(topic string) bool

MatchTarget returns true if the message matches the topic (or is global)

func (*Message) Print 

func (message *Message) Print(showTime bool, showTopic bool) error

Print the formatted message

type Project 

type Project struct {
	Path              string
	Files             FileMap
	FileCount         int
	SizeCount         int64
	CostCount         float64
	LocalExpiration   Expiration
	RemoteExpiration  Expiration
	BackupEvery       time.Duration
	LastNoBackupAlert time.Time
	Archived          bool

	SchemaVersion int
}

Project is a project (directory with a leat one File) in our DB

func NewProject 

func NewProject(path string, expirationConfig *ExpirationConfig) *Project

NewProject create a new Project struct

func (*Project) GetLatestFile 

func (p *Project) GetLatestFile() *File

GetLatestFile return the lastest file of the project If the project is empty, result is nil

func (*Project) ModTime 

func (p *Project) ModTime() time.Time

ModTime is the modification time of the project (aka the latest file's ModTime) If the project is empty, result is 0 (see IsZero())

type ProjectDBDeleteLocalFunc 

type ProjectDBDeleteLocalFunc func(file *File, filePath string)

ProjectDBDeleteLocalFunc is called when a local file expires (as a goroutine)

type ProjectDBDeleteRemoteFunc 

type ProjectDBDeleteRemoteFunc func(file *File)

ProjectDBDeleteRemoteFunc is called when a remote file expirtes (as a goroutine)

type ProjectDBNoBackupAlertFunc 

type ProjectDBNoBackupAlertFunc func(projects []*Project)

ProjectDBNoBackupAlertFunc is called when a backup is missing for a project

type ProjectDBStats 

type ProjectDBStats struct {
	ProjectCount int
	FileCount    int
	TotalSize    int64
	TotalCost    float64
}

ProjectDBStats hosts stats about the projects and files

type ProjectDatabase 

type ProjectDatabase struct {
	// contains filtered or unexported fields
}

ProjectDatabase is a Project database holder

func NewProjectDatabase 

func NewProjectDatabase(
	filename string,
	localStoragePath string,
	defaultExpiration *ExpirationConfig,
	alertSender *AlertSender,
	deleteLocalFunc ProjectDBDeleteLocalFunc,
	deleteRemoteFunc ProjectDBDeleteRemoteFunc,
	noBackupAlertFunc ProjectDBNoBackupAlertFunc,
	log *Log,
) (*ProjectDatabase, error)

NewProjectDatabase allocates a new ProjectDatabase

func (*ProjectDatabase) AddFile 

func (db *ProjectDatabase) AddFile(projectName string, file *File) error

AddFile will add a file to the database to a specific project

func (*ProjectDatabase) FileExists 

func (db *ProjectDatabase) FileExists(projectName string, fileName string) bool

FileExists returns true if the file exists in the project

func (*ProjectDatabase) FindFile 

func (db *ProjectDatabase) FindFile(projectName string, fileName string) *File

FindFile finds a file based on projectName and its fileName

func (*ProjectDatabase) FindOrCreateProject 

func (db *ProjectDatabase) FindOrCreateProject(projectName string) (*Project, error)

FindOrCreateProject will return an existing project or create a new one if needed

func (*ProjectDatabase) GetByName 

func (db *ProjectDatabase) GetByName(name string) (*Project, error)

GetByName returns a project, using its name Warning: do not mutate returned project, it's not thread safe

func (*ProjectDatabase) GetFilenames 

func (db *ProjectDatabase) GetFilenames(projectName string) ([]string, error)

GetFilenames returns all filenames of a project, sorted by mtime (newer last)

func (*ProjectDatabase) GetNames 

func (db *ProjectDatabase) GetNames() []string

GetNames returns all projects names, sorted

func (*ProjectDatabase) GetPath 

func (db *ProjectDatabase) GetPath() string

GetPath of the database

func (*ProjectDatabase) GetProjectNextExpiration 

func (db *ProjectDatabase) GetProjectNextExpiration(project *Project, file *File) (ExpirationResult, ExpirationResult, error)

GetProjectNextExpiration return next (= for next file) expiration values

func (*ProjectDatabase) NoBackupAlerts 

func (db *ProjectDatabase) NoBackupAlerts()

NoBackupAlerts will alert when no backup is found in time for a project

func (*ProjectDatabase) Save 

func (db *ProjectDatabase) Save() error

Save the database to the disk (public mutex-protected version of save())

func (*ProjectDatabase) SaveToWriter 

func (db *ProjectDatabase) SaveToWriter(writer io.Writer) error

SaveToWriter will save the database to a writer (mutex-protected)

func (*ProjectDatabase) ScheduleExpireFiles 

func (db *ProjectDatabase) ScheduleExpireFiles()

ScheduleExpireFiles will schedule file expiration tasks (call as a goroutine)

func (*ProjectDatabase) ScheduleNoBackupAlerts 

func (db *ProjectDatabase) ScheduleNoBackupAlerts()

ScheduleNoBackupAlerts will schedule NoBackupAlerts task

func (*ProjectDatabase) ScheduleReEncryptFiles 

func (db *ProjectDatabase) ScheduleReEncryptFiles(app *App)

ScheduleReEncryptFiles will schedule file re-encryption tasks (call as a goroutine)

func (*ProjectDatabase) SetRemoteExpirationOverride 

func (db *ProjectDatabase) SetRemoteExpirationOverride(filePath string, exp ExpirationResult)

SetRemoteExpirationOverride force an expiration for a future coming file

func (*ProjectDatabase) Stats 

func (db *ProjectDatabase) Stats() (ProjectDBStats, error)

Stats return DB content stats using a ProjectDBStats object

type ProjectMap 

type ProjectMap map[string]*Project

ProjectMap is a map of Project

type Pusher 

type Pusher interface {
	GetETA() time.Duration
	GetError() error
	IsFinished() bool
}

Pusher is an interface that will host an uploading thread implementation

func NewPusherMulch 

func NewPusherMulch(file *File, path string, expire time.Duration, config *PusherConfig, log *Log) (Pusher, error)

NewPusherMulch create a new Pusher to mulch

type PusherConfig 

type PusherConfig struct {
	Name string
	Type string
	URL  string
	Key  string
}

PusherConfig is a decoded tomlPushDestination config

type PusherMulch 

type PusherMulch struct {
	ETA      time.Duration
	Error    error
	Finished bool
	// contains filtered or unexported fields
}

Pusher is a structure that will host an uploading thread

func (*PusherMulch) GetETA 

func (p *PusherMulch) GetETA() time.Duration

GetETA of upload end

func (*PusherMulch) GetError 

func (p *PusherMulch) GetError() error

GetError of upload (if any)

func (*PusherMulch) IsFinished 

func (p *PusherMulch) IsFinished() bool

IsFinished uploading?

type Request 

type Request struct {
	Route    *Route
	SubPath  string
	HTTP     *http.Request
	Response http.ResponseWriter
	App      *App
	APIKey   *APIKey
}

Request describes a request and allows to build a response

func (*Request) Printf 

func (req *Request) Printf(format string, args ...interface{})

Printf like helper for req.Response.Write

func (*Request) Println 

func (req *Request) Println(message string)

Println like helper for req.Response.Write

type Retriever 

type Retriever struct {
	Path     string
	ETA      time.Duration
	Error    error
	Finished bool
	// contains filtered or unexported fields
}

Retriever is a structure that will host a Swift downloading thread

func NewRetriever 

func NewRetriever(file *File, swift *Swift, outputFilename string) (*Retriever, error)

NewRetriever create a new Retriever

func (*Retriever) GetETA 

func (r *Retriever) GetETA() time.Duration

GetETA of download end

type Route 

type Route struct {
	Route        string
	Public       bool
	NoProtoCheck bool
	Handler      func(*Request)
	// contains filtered or unexported fields
}

Route describes a route to a handler

type Stats 

type Stats struct {
	LastReportTime time.Time
	FileCount      int
	SizeCount      int64
	// contains filtered or unexported fields
}

Stats hosts statisctics about the app

func NewStats 

func NewStats() *Stats

NewStats return a new Stats object

func (*Stats) Inc 

func (s *Stats) Inc(fileCount int, sizeCount int64)

Inc will increment stats

func (*Stats) Report 

func (s *Stats) Report(intro string) string

Report as a string and reset stats

type Swift 

type Swift struct {
	Config *AppConfig
	Conn   swift.Connection
}

Swift host connection and configuration

func NewSwift 

func NewSwift(config *AppConfig) (*Swift, error)

NewSwift will create a new Swift instance from config

func (*Swift) CheckContainer 

func (s *Swift) CheckContainer(name string) error

CheckContainer will returnb nil if container and container_segments exists

func (*Swift) Delete 

func (s *Swift) Delete(file *File) error

Delete a File

func (*Swift) FileGetContent 

func (s *Swift) FileGetContent(container string, path string, output io.Writer) error

FileGetContent will read a file to io.Writer

func (*Swift) FilePutContent 

func (s *Swift) FilePutContent(container string, path string, content io.Reader) error

FilePutContent will create / overwrite a file with a content

func (*Swift) GetObjetAvailability 

func (s *Swift) GetObjetAvailability(container string, path string) (string, time.Duration, error)

GetObjetAvailability returns availability, explained with two values: - state (sealed, unsealing, unsealed) - delay in seconds (0 meaning that is file is ready to be downloaded)

func (*Swift) ObjectOpen 

func (s *Swift) ObjectOpen(container string, path string) (io.ReadCloser, error)

ObjectOpen a Swift Object, returning a ReadCloser

func (*Swift) Unseal 

func (s *Swift) Unseal(container string, path string) (time.Duration, error)

Unseal a "cold" file, return availability ETA

func (*Swift) Upload 

func (s *Swift) Upload(file *File) error

Upload a local file to Swift provider

type SwiftConfig 

type SwiftConfig struct {
	UserName   string
	APIKey     string
	AuthURL    string
	Domain     string
	Region     string
	ChunckSize uint64
}

SwiftConfig stores final settings for Swift

func NewSwiftConfigFromToml 

func NewSwiftConfigFromToml(tConfig *tomlSwiftConfig) (*SwiftConfig, error)

NewSwiftConfigFromToml will check tomlSwiftConfig and create a SwiftConfig

type Upload 

type Upload struct {
	// input parameters
	ProjectName string
	File        *File

	// output chan
	Result chan error

	// lifecycle members
	Tries   int
	LastTry time.Time
}

Upload is for the whole lifecycle of an upload, from request to completion

func NewUpload 

func NewUpload(projectName string, file *File) *Upload

NewUpload initialize a new instance

type Uploader 

type Uploader struct {
	NumWorkers int
	Channel    chan *Upload
	Swift      *Swift
	Log        *Log
	Status     []string
}

Uploader will manage workers

func NewUploader 

func NewUploader(numWorkers int, swift *Swift, log *Log) *Uploader

NewUploader initialize a new instance

func (*Uploader) Start 

func (up *Uploader) Start()

Start the Uploader (run workers)

type WaitList 

type WaitList struct {
	// contains filtered or unexported fields
}

WaitList stores all files of the source path we're waiting for. (waiting means the fime must be stable [no change of size of mtime] for a determined period of time)

func NewWaitList 

func NewWaitList(watchPath string, filterFunc WaitListFilterFunc, queueFunc WaitListQueueFunc, log *Log) (*WaitList, error)

NewWaitList allocates a new WaitList

func (*WaitList) Dump 

func (wl *WaitList) Dump()

Dump list content on stdout (debug)

func (*WaitList) FileExists 

func (wl *WaitList) FileExists(projectName string, fileName string) bool

FileExists returns true if the file is in the WaitList

func (*WaitList) RemoveFile 

func (wl *WaitList) RemoveFile(projectName string, fileName string) error

RemoveFile from the WaitList (next Scan will discover the file again)

func (*WaitList) Scan 

func (wl *WaitList) Scan() error

Scan the source directory to detect new files and add them to the list TODO: delete files from the list when they're not found anymore during scans (it's a memory issue)

type WaitListFilterFunc 

type WaitListFilterFunc func(dirName string, fileName string) bool

WaitListFilterFunc is used to filter incomming files return true to add the file to the WaitList, false to reject it

type WaitListQueueFunc 

type WaitListQueueFunc func(projectName string, file File)

WaitListQueueFunc is called when a new file is ready and has been queued in the WaitList. MUST NO BLOCK the caller, use goroutines if needed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.