cas

Documentation

Index

• Constants
• func DefaultTransformPathFunc(data []byte) (prefix string, filename string)
• func NewTree(root string) ([]string, error)
• func PathExists(path string) bool
• func PrepareRawFile(path string, data []byte) []byte
• func RecreateTree(path string, files []*File) error
• func ZLibPack(data []byte) []byte
• func ZLibUnpack(data []byte) ([]byte, error)
• type DB
  • func NewDB(root string) (*DB, error)
  • func (db *DB) Add(key string, hashes []string) error
  • func (db *DB) GetByKey(key string) ([]string, error)
  • func (db *DB) GetKeysByChunks(offset int) ([]string, error)
  • func (db *DB) RemoveByKey(key string) error
• type File
• type KeyHashPair
• type PackFunc
• type Storage
  • func NewDefaultStorage(opts StorageOpts) (*Storage, error)
  • func (s *Storage) AddNewPath(key string, hash string) error
  • func (s *Storage) DeleteAll()
  • func (s *Storage) Get(key string) ([]*File, error)
  • func (s *Storage) GetByHash(hash string) ([]byte, error)
  • func (s *Storage) GetHashesByKey(key string) ([]string, error)
  • func (s *Storage) GetKeysByChunks(offset int) ([]string, error)
  • func (s *Storage) Has(path string) bool
  • func (s *Storage) MakePathFromHash(hash string) string
  • func (s *Storage) PrepareParentFolders(fullPath string) error
  • func (s *Storage) RemoveByHash(hash string) error
  • func (s *Storage) RemoveByKey(key string) error
  • func (s *Storage) Store(key string, paths ...string) error
  • func (s *Storage) Write(path string, data []byte) error
  • func (s *Storage) WriteFromRawData(data []byte) (string, error)
• type StorageOpts
• type TransformPathFunc
• type UnpackFunc

Constants

const DB_CHUNK_SIZE = 100

TODO: move it to config file

const DB_DRIVER = "sqlite3"

const DB_PATH = "meta.db"

const PREFIX_LENGTH = 5

Functions

func DefaultTransformPathFunc

func DefaultTransformPathFunc(data []byte) (prefix string, filename string)

func NewTree

func NewTree(root string) ([]string, error)

NewTree creates directory hierarchy.

func PathExists

func PathExists(path string) bool

func PrepareRawFile

func PrepareRawFile(path string, data []byte) []byte

PrepareRawFile adds a special header to the beginning of a file's content and returns the new content as raw bytes.

This function takes a file path and the original file data as input. It creates a header that consists of the file path followed by a null byte (`\0`), and prepends this header to the original data. The resulting byte slice containing the header and the original data is returned.

func RecreateTree

func RecreateTree(path string, files []*File) error

RecreateTree creates a directory structure and populates it with files based on the provided files ([]*File).

This method takes a destination path and a slice of File pointers. It ensures that the destination folder exists, creating it if necessary. For each file in the provided list, it constructs the full path and checks if the file already exists. If it does, it compares the existing file's content with the new data. If the contents differ, an error is returned. If the file does not exist, it creates a new file with the provided data.

func ZLibPack

func ZLibPack(data []byte) []byte

func ZLibUnpack

func ZLibUnpack(data []byte) ([]byte, error)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

func NewDB

func NewDB(root string) (*DB, error)

NewDB creates a new instance of DB and initializes the database connection.

This method takes a root directory path, constructs the full path to the database file, and attempts to open a connection to the database. If the connection is successful, it initializes the database and returns a pointer to the DB instance. In case of any errors during these processes, an error is returned.

func (*DB) Add

func (db *DB) Add(key string, hashes []string) error

Add inserts key-hash records into the database.

This method takes a key and a slice of hash strings and adds them to the `keys` table in the database. It validates that the key and hashes are not empty and constructs an SQL insert statement for the operation. If any of the inputs are invalid or if an error occurs during the database operations, an error is returned.

func (*DB) GetByKey

func (db *DB) GetByKey(key string) ([]string, error)

GetByKey retrieves hashes associated with a given key from the database.

This method takes a key as input and queries the `keys` table to retrieve all associated hash values. If the query is successful, it returns a slice of strings containing the hashes. If an error occurs during the query or while scanning the results, an error is returned.

func (*DB) GetKeysByChunks

func (db *DB) GetKeysByChunks(offset int) ([]string, error)

GetKeysByChunks retrieves a chunk of distinct keys from the database.

This method takes an offset as input and queries the `keys` table to fetch a limited number of distinct keys based on the specified offset. Chunk size is defined by DB_CHUNK_SIZE value. If the query is successful, it returns a slice of strings containing the keys. If an error occurs during the query or while scanning the results, an error is returned.

func (*DB) RemoveByKey

func (db *DB) RemoveByKey(key string) error

RemoveByKey deletes all records associated with a given key from the database.

This method takes a key as input and executes a delete operation on the `keys` table, removing all entries that match the specified key. If an error occurs during the preparation or execution of the SQL statement, an error is returned.

type File

type File struct {
	Path string
	Data []byte
}

type KeyHashPair

type KeyHashPair struct {
	Key  string
	Hash string
}

type PackFunc

type PackFunc func([]byte) []byte

type Storage

type Storage struct {
	Pack   PackFunc
	Unpack UnpackFunc
	// contains filtered or unexported fields
}

func NewDefaultStorage

func NewDefaultStorage(opts StorageOpts) (*Storage, error)

NewDefaultStorage creates a new instance of Storage.

This function initializes the storage by creating the specified base directory, setting up the database, and configuring any provided transformation functions for paths. If any errors occur during the directory creation or database initialization, an error is returned.

func (*Storage) AddNewPath

func (s *Storage) AddNewPath(key string, hash string) error

AddNewPath adds a new key-hash record to the storage database.

This method takes a key and a hash as input, and it invokes the Add method of the underlying database to store the association in the database. If an error occurs during the addition process, it returns the error.

func (*Storage) DeleteAll

func (s *Storage) DeleteAll()

func (*Storage) Get

func (s *Storage) Get(key string) ([]*File, error)

Get retrieves files associated with the given key from the storage.

This method takes a key as input and fetches the associated hash values from the database. It constructs file paths based on the hashes and reads the corresponding files from disk. If any errors occur during the database retrieval or file reading, the method returns an error.

func (*Storage) GetByHash

func (s *Storage) GetByHash(hash string) ([]byte, error)

GetByHash retrieves the content of a file as a byte slice using the provided hash.

This method takes a hash string as input, constructs the file path based on the hash, and reads the file's content from disk. If an error occurs during the file reading process, the method returns an error.

func (*Storage) GetHashesByKey

func (s *Storage) GetHashesByKey(key string) ([]string, error)

GetHashesByKey retrieves hash values associated with the specified key.

This method takes a key as input and queries the underlying database to fetch the corresponding hash values. If the operation is successful, it returns a slice of hashes; otherwise, it returns an error.

func (*Storage) GetKeysByChunks

func (s *Storage) GetKeysByChunks(offset int) ([]string, error)

func (*Storage) Has

func (s *Storage) Has(path string) bool

func (*Storage) MakePathFromHash

func (s *Storage) MakePathFromHash(hash string) string

MakePathFromHash constructs a file path from a given hash.

This method takes a hash string as input and creates a file path by splitting the hash into two parts: the prefix and the remaining characters. The resulting path is constructed by joining the base directory with the prefix and the rest of the hash.

func (*Storage) PrepareParentFolders

func (s *Storage) PrepareParentFolders(fullPath string) error

func (*Storage) RemoveByHash

func (s *Storage) RemoveByHash(hash string) error

RemoveByHash deletes the file associated with the specified hash.

This method takes a hash string as input and constructs the corresponding file path. It first checks if the file exists; if it does not, it returns an error. If the file exists, it attempts to remove the file from disk. After removing the file, it checks if the parent directory is empty and removes it if necessary. If any errors occur during these operations, the method returns an error.

func (*Storage) RemoveByKey

func (s *Storage) RemoveByKey(key string) error

RemoveByKey deletes all files associated with the specified key.

This method takes a key as input and removes all files that are linked to that key in the storage. It first checks if the key is empty, returning an error if it is. Then, it retrieves the associated hash values from the database and attempts to remove each file by its hash. If any operation fails during this process, an error is returned. Finally, the method removes the key entry from the database.

func (*Storage) Store

func (s *Storage) Store(key string, paths ...string) error

Store saves one or more file or directory paths to disk under the specified key.

This method takes a key and a variable number of paths as input. It creates a tree structure for each path and saves it in the storage using the provided key. If any errors occur during the creation of the tree or while saving, the method returns an error.

func (*Storage) Write

func (s *Storage) Write(path string, data []byte) error

Write saves the given data to a file at the specified path on disk.

This method takes a file path and the data to be written as input. It creates a new file at the specified path and writes the provided data to it. If any errors occur during file creation or writing, the method returns an error.

func (*Storage) WriteFromRawData

func (s *Storage) WriteFromRawData(data []byte) (string, error)

WriteFromRawData writes the provided raw data to a file after creating path and compressing the data.

This method takes raw data as input, creates path based on data, and compresses the data before saving it to disk. It ensures that the target directory exists, creating it if necessary. If a file with the same name already exists, it checks if the content is different to avoid overwriting. The method returns the transformed path of the saved file or an error if any operation fails.

type StorageOpts

type StorageOpts struct {
	BaseDir           string
	PathFunc          TransformPathFunc
	Pack              PackFunc
	Unpack            UnpackFunc
	ReplicationFactor int // TODO: implement locally
}

type TransformPathFunc

type TransformPathFunc func([]byte) (string, string)

type UnpackFunc

type UnpackFunc func([]byte) ([]byte, error)