s3storage

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 18, 2023 License: MIT Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	DefaultPartSize     = 8 * common.MbToBytes
	DefaultUploadCutoff = 100 * common.MbToBytes
	DefaultConcurrency  = 5
	MaxPartSizeMb       = 5 * 1024
)

Variables

View Source 
var ContentTypes = map[string]string{
	".css":  "text/css",
	".pdf":  "application/pdf",
	".xml":  "text/xml",
	".csv":  "text/csv",
	".json": "application/json",
	".rtf":  "application/rtf",
	".txt":  "text/plain",
	".java": "text/plain",
	".dat":  "text/plain",

	".htm":  "text/html",
	".html": "text/html",

	".gif":  "image/gif",
	".jpeg": "image/jpeg",
	".jpg":  "image/jpeg",
	".png":  "image/png",
	".bmp":  "image/bmp",

	".js":   "application/javascript",
	".mjs":  "application/javascript",
	".svg":  "image/svg+xml",
	".wasm": "application/wasm",
	".webp": "image/webp",

	".wav":  "audio/wav",
	".mp3":  "audio/mpeg",
	".mpeg": "video/mpeg",
	".aac":  "audio/aac",
	".avi":  "video/x-msvideo",
	".m3u8": "application/x-mpegURL",
	".ts":   "video/MP2T",
	".mid":  "audio/midiaudio/x-midi",
	".3gp":  "video/3gpp",
	".mp4":  "video/mp4",

	".doc":  "application/msword",
	".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
	".ppt":  "application/vnd.ms-powerpoint",
	".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
	".xls":  "application/vnd.ms-excel",
	".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",

	".gz":  "application/x-gzip",
	".jar": "application/java-archive",
	".rar": "application/vnd.rar",
	".tar": "application/x-tar",
	".zip": "application/x-zip-compressed",
	".7z":  "application/x-7z-compressed",
	".3g2": "video/3gpp2",

	".sh":  "application/x-sh",
	".exe": "application/x-msdownload",
	".dll": "application/x-msdownload",
}

ContentTypeMap : Store file extension to content-type mapping

Functions

func News3storageComponent 

func News3storageComponent() internal.Component

Constructor to create object of this component

func ParseAndReadDynamicConfig 

func ParseAndReadDynamicConfig(s3 *S3Storage, opt Options, reload bool) error

ParseAndReadDynamicConfig : On config change read only the required config

func ParseAndValidateConfig 

func ParseAndValidateConfig(s3 *S3Storage, opt Options) error

ParseAndValidateConfig : Parse and validate config

Types

type Client 

type Client struct {
	Connection
	// contains filtered or unexported fields
}

func (*Client) Configure 

func (cl *Client) Configure(cfg Config) error

Configure : Initialize the awsS3Client

func (*Client) CreateDirectory 

func (cl *Client) CreateDirectory(name string) error

CreateDirectory : Create a new directory in the bucket/virtual directory

func (*Client) CreateFile 

func (cl *Client) CreateFile(name string, mode os.FileMode) error

CreateFile : Create a new file in the bucket/virtual directory 

func (cl *Client) CreateLink(source string, target string, isSymlink bool) error

CreateLink : Create a symlink in the bucket/virtual directory

func (*Client) DeleteDirectory 

func (cl *Client) DeleteDirectory(name string) error

DeleteDirectory : Recursively delete all objects with the given prefix. If name is given without a trailing slash, a slash will be added. If the directory does not exist, no error will be returned.

func (*Client) DeleteFile 

func (cl *Client) DeleteFile(name string) error

DeleteFile : Delete an object. if the file does not exist, this returns an error (ENOENT).

func (*Client) GetAttr 

func (cl *Client) GetAttr(name string) (*internal.ObjAttr, error)

GetAttr : Get attributes for a given file or folder. If name is a file, it should not have a trailing slash. If name is a directory, the trailing slash is optional.

func (*Client) GetFileBlockOffsets 

func (cl *Client) GetFileBlockOffsets(name string) (*common.BlockOffsetList, error)

GetFileBlockOffsets: store blocks ids and corresponding offsets.

func (*Client) List 

func (cl *Client) List(prefix string, marker *string, count int32) ([]*internal.ObjAttr, *string, error)

Wrapper for awsS3Client.ListObjectsV2 List : Get a list of objects matching the given prefix, up to the next "/", similar to listing a directory. For predictable results, include the trailing slash in the prefix. When prefix has no trailing slash, List has unintuitive behavior (e.g. prefix "file" would match "filet-o-fish"). This fetches the list using a marker so the caller code should handle marker logic. If count=0 - fetch max entries. the *string being returned is the token / marker and will be nil when the listing is complete.

func (*Client) ListBuckets 

func (cl *Client) ListBuckets() ([]string, error)

Wrapper for awsS3Client.ListBuckets

func (*Client) NewCredentialKey 

func (cl *Client) NewCredentialKey(key, value string) error

NewCredentialKey : Update the credential key specified by the user. Currently not implemented.

func (*Client) ReadBuffer 

func (cl *Client) ReadBuffer(name string, offset int64, length int64, isSymlink bool) ([]byte, error)

Download object with the given name and return the data as a byte array. Reads starting at a byte offset from the start of the object, with length in bytes = len. len = 0 reads to the end of the object. name is the file path

func (*Client) ReadInBuffer 

func (cl *Client) ReadInBuffer(name string, offset int64, length int64, data []byte) error

Download object to provided byte array. Reads starting at a byte offset from the start of the object, with length in bytes = len. len = 0 reads to the end of the object. name is the file path.

func (*Client) ReadToFile 

func (cl *Client) ReadToFile(name string, offset int64, count int64, fi *os.File) error

Download object data to a file handle. Read starting at a byte offset from the start of the object, with length in bytes = count. count = 0 reads to the end of the object.

func (*Client) RenameDirectory 

func (cl *Client) RenameDirectory(source string, target string) error

RenameDirectory : Rename the directory

func (*Client) RenameFile 

func (cl *Client) RenameFile(source string, target string, isSymLink bool) error

RenameFile : Rename the object (copy then delete).

func (*Client) SetPrefixPath 

func (cl *Client) SetPrefixPath(path string) error

Set the prefix path - this overrides "subdirectory" in config.yaml. This is only used for testing.

func (*Client) StageAndCommit 

func (cl *Client) StageAndCommit(name string, bol *common.BlockOffsetList) error

func (*Client) TruncateFile 

func (cl *Client) TruncateFile(name string, size int64) error

Truncate object to size in bytes. name is the file path.

func (*Client) UpdateConfig 

func (cl *Client) UpdateConfig(cfg Config) error

For dynamic configuration, update the config here.

func (*Client) Write 

func (cl *Client) Write(options internal.WriteFileOptions) error

Write : write data at given offset to an object

func (*Client) WriteFromBuffer 

func (cl *Client) WriteFromBuffer(name string, metadata map[string]string, data []byte) error

WriteFromBuffer : Upload from a buffer to an object. name is the file path.

func (*Client) WriteFromFile 

func (cl *Client) WriteFromFile(name string, metadata map[string]string, fi *os.File) error

Upload from a file handle to an object. The metadata parameter is not used.

type Config 

type Config struct {
	// contains filtered or unexported fields
}

type Connection 

type Connection struct {
	Config   Config
	Endpoint *url.URL
}

type Options 

type Options struct {
	BucketName                string                  `config:"bucket-name" yaml:"bucket-name,omitempty"`
	KeyID                     string                  `config:"key-id" yaml:"key-id,omitempty"`
	SecretKey                 string                  `config:"secret-key" yaml:"secret-key,omitempty"`
	Region                    string                  `config:"region" yaml:"region,omitempty"`
	Profile                   string                  `config:"profile" yaml:"region,omitempty"`
	Endpoint                  string                  `config:"endpoint" yaml:"endpoint,omitempty"`
	PrefixPath                string                  `config:"subdirectory" yaml:"subdirectory,omitempty"`
	RestrictedCharsWin        bool                    `config:"restricted-characters-windows" yaml:"-"`
	PartSizeMb                int64                   `config:"part-size-mb" yaml:"part-size-mb,omitempty"`
	UploadCutoffMb            int64                   `config:"upload-cutoff-mb" yaml:"upload-cutoff-mb,omitempty"`
	Concurrency               int                     `config:"concurrency" yaml:"concurrency,omitempty"`
	DisableConcurrentDownload bool                    `config:"disable-concurrent-download" yaml:"disable-concurrent-download,omitempty"`
	EnableChecksum            bool                    `config:"enable-checksum" yaml:"enable-checksum,omitempty"`
	ChecksumAlgorithm         types.ChecksumAlgorithm `config:"checksum-algorithm" yaml:"checksum-algorithm,omitempty"`
	UsePathStyle              bool                    `config:"use-path-style" yaml:"use-path-style,omitempty"`
}

type S3Connection 

type S3Connection interface {
	Configure(cfg Config) error
	UpdateConfig(cfg Config) error

	ListBuckets() ([]string, error)

	// This is just for test, shall not be used otherwise
	SetPrefixPath(string) error

	CreateFile(name string, mode os.FileMode) error
	CreateDirectory(name string) error
	CreateLink(source string, target string, isSymlink bool) error

	DeleteFile(name string) error
	DeleteDirectory(name string) error

	RenameFile(string, string, bool) error
	RenameDirectory(string, string) error

	GetAttr(name string) (attr *internal.ObjAttr, err error)

	// Standard operations to be supported by any account type
	List(prefix string, marker *string, count int32) ([]*internal.ObjAttr, *string, error)

	ReadToFile(name string, offset int64, count int64, fi *os.File) error
	ReadBuffer(name string, offset int64, length int64, isSymlink bool) ([]byte, error)
	ReadInBuffer(name string, offset int64, length int64, data []byte) error

	WriteFromFile(name string, metadata map[string]string, fi *os.File) error
	WriteFromBuffer(name string, metadata map[string]string, data []byte) error
	Write(options internal.WriteFileOptions) error
	GetFileBlockOffsets(name string) (*common.BlockOffsetList, error)

	TruncateFile(string, int64) error
	StageAndCommit(name string, bol *common.BlockOffsetList) error

	NewCredentialKey(_, _ string) error
}

func NewConnection 

func NewConnection(cfg Config) (S3Connection, error)

NewConnection : Create S3Connection Object

type S3Storage 

type S3Storage struct {
	internal.BaseComponent
	// contains filtered or unexported fields
}

S3Storage Wrapper type around aws-sdk-go-v2/service/s3

func (*S3Storage) Chmod 

func (s3 *S3Storage) Chmod(options internal.ChmodOptions) error

func (*S3Storage) Chown 

func (s3 *S3Storage) Chown(options internal.ChownOptions) error

func (*S3Storage) CloseFile 

func (s3 *S3Storage) CloseFile(options internal.CloseFileOptions) error

func (*S3Storage) Configure 

func (s3 *S3Storage) Configure(isParent bool) error

Configure : Pipeline will call this method after constructor so that you can read config and initialize yourself

func (*S3Storage) CopyFromFile 

func (s3 *S3Storage) CopyFromFile(options internal.CopyFromFileOptions) error

func (*S3Storage) CopyToFile 

func (s3 *S3Storage) CopyToFile(options internal.CopyToFileOptions) error

func (*S3Storage) CreateDir 

func (s3 *S3Storage) CreateDir(options internal.CreateDirOptions) error

Directory operations

func (*S3Storage) CreateFile 

func (s3 *S3Storage) CreateFile(options internal.CreateFileOptions) (*handlemap.Handle, error)

File operations 

func (s3 *S3Storage) CreateLink(options internal.CreateLinkOptions) error

Symlink operations

func (*S3Storage) DeleteDir 

func (s3 *S3Storage) DeleteDir(options internal.DeleteDirOptions) error

func (*S3Storage) DeleteFile 

func (s3 *S3Storage) DeleteFile(options internal.DeleteFileOptions) error

func (*S3Storage) FlushFile 

func (s3 *S3Storage) FlushFile(options internal.FlushFileOptions) error

func (*S3Storage) GetAttr 

func (s3 *S3Storage) GetAttr(options internal.GetAttrOptions) (*internal.ObjAttr, error)

Attribute operations

func (*S3Storage) GetFileBlockOffsets 

func (s3 *S3Storage) GetFileBlockOffsets(options internal.GetFileBlockOffsetsOptions) (*common.BlockOffsetList, error)

func (*S3Storage) IsDirEmpty 

func (s3 *S3Storage) IsDirEmpty(options internal.IsDirEmptyOptions) bool

func (*S3Storage) ListBuckets 

func (s3 *S3Storage) ListBuckets() ([]string, error)

------------------------- Bucket listing -------------------------------------------

func (*S3Storage) Name 

func (s3 *S3Storage) Name() string

func (*S3Storage) OnConfigChange 

func (s3 *S3Storage) OnConfigChange()

OnConfigChange : When config file is changed, this will be called by pipeline. Refresh required config here

func (*S3Storage) OpenFile 

func (s3 *S3Storage) OpenFile(options internal.OpenFileOptions) (*handlemap.Handle, error)

func (*S3Storage) Priority 

func (s3 *S3Storage) Priority() internal.ComponentPriority

func (*S3Storage) ReadDir 

func (s3 *S3Storage) ReadDir(options internal.ReadDirOptions) ([]*internal.ObjAttr, error)

func (*S3Storage) ReadFile 

func (s3 *S3Storage) ReadFile(options internal.ReadFileOptions) ([]byte, error)

Read and return file data as a buffer.

func (*S3Storage) ReadInBuffer 

func (s3 *S3Storage) ReadInBuffer(options internal.ReadInBufferOptions) (int, error)

Read file data into the buffer given in options.Data. 

func (s3 *S3Storage) ReadLink(options internal.ReadLinkOptions) (string, error)

func (*S3Storage) RenameDir 

func (s3 *S3Storage) RenameDir(options internal.RenameDirOptions) error

func (*S3Storage) RenameFile 

func (s3 *S3Storage) RenameFile(options internal.RenameFileOptions) error

func (*S3Storage) SetName 

func (s3 *S3Storage) SetName(name string)

func (*S3Storage) SetNextComponent 

func (s3 *S3Storage) SetNextComponent(c internal.Component)

func (*S3Storage) Start 

func (s3 *S3Storage) Start(ctx context.Context) error

Start : Initialize the go-sdk pipeline here and test auth is working fine

func (*S3Storage) Stop 

func (s3 *S3Storage) Stop() error

Stop : Disconnect all running operations here

func (*S3Storage) StreamDir 

func (s3 *S3Storage) StreamDir(options internal.StreamDirOptions) ([]*internal.ObjAttr, string, error)

func (*S3Storage) TruncateFile 

func (s3 *S3Storage) TruncateFile(options internal.TruncateFileOptions) error

func (*S3Storage) WriteFile 

func (s3 *S3Storage) WriteFile(options internal.WriteFileOptions) (int, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.