Documentation
¶
Overview ¶
Package flatfs is a Datastore implementation that stores all objects in a two-level directory structure in the local file system, regardless of the hierarchy of the keys.
Package flatfs is a Datastore implementation that stores all objects in a two-level directory structure in the local file system, regardless of the hierarchy of the keys.
Index ¶
- Constants
- Variables
- func Create(path string, fun *ShardIdV1) error
- func DirIsEmpty(name string) (bool, error)
- func DowngradeV1toV0(path string) error
- func Move(oldPath string, newPath string, out io.Writer) error
- func UpgradeV0toV1(path string, prefixLen int) error
- func WriteReadme(dir string, id *ShardIdV1) error
- func WriteShardFunc(dir string, id *ShardIdV1) error
- type Datastore
- func (fs *Datastore) Accuracy() string
- func (fs *Datastore) Batch() (datastore.Batch, error)
- func (fs *Datastore) Close() error
- func (fs *Datastore) Delete(k key.Key) error
- func (fs *Datastore) DiskUsage() (uint64, error)
- func (fs *Datastore) Get(k key.Key) (value []byte, err error)
- func (fs *Datastore) GetSize(k key.Key) (size int, err error)
- func (fs *Datastore) Has(k key.Key) (exists bool, err error)
- func (fs *Datastore) Put(k key.Key, value []byte) error
- func (fs *Datastore) Query(q query.Query) (query.Results, error)
- func (fs *Datastore) ShardStr() string
- func (fs *Datastore) Sync(prefix key.Key) error
- type ShardFunc
- type ShardIdV1
Constants ¶
const PREFIX = "/repo/flatfs/shard/"
const README_FN = "_README"
const SHARDING_FN = "SHARDING"
const SyncThreadsMax = 16
don't block more than 16 threads on sync opearation 16 should be able to sataurate most RAIDs in case of two used disks per write (RAID 1, 5) and queue depth of 2, 16 concurrent Sync calls should be able to saturate 16 HDDs RAID TODO: benchmark it out, maybe provide tweak parmeter
Variables ¶
var ( // DiskUsageFile is the name of the file to cache the size of the // datastore in disk DiskUsageFile = "diskUsage.cache" // DiskUsageFilesAverage is the maximum number of files per folder // to stat in order to calculate the size of the datastore. // The size of the rest of the files in a folder will be assumed // to be the average of the values obtained. This includes // regular files and directories. DiskUsageFilesAverage = 2000 // DiskUsageCalcTimeout is the maximum time to spend // calculating the DiskUsage upon a start when no // DiskUsageFile is present. // If this period did not suffice to read the size of the datastore, // the remaining sizes will be stimated. DiskUsageCalcTimeout = 5 * time.Minute // RetryDelay is a timeout for a backoff on retrying operations // that fail due to transient errors like too many file descriptors open. RetryDelay = time.Millisecond * 200 // RetryAttempts is the maximum number of retries that will be attempted // before giving up. RetryAttempts = 6 )
var ( ErrDatastoreExists = errors.New("datastore already exists") ErrDatastoreDoesNotExist = errors.New("datastore directory does not exist") ErrShardingFileMissing = fmt.Errorf("%s file not found in datastore", SHARDING_FN) ErrClosed = errors.New("datastore closed") ErrInvalidKey = errors.New("key not supported by flatfs") )
var IPFS_DEF_SHARD = NextToLast(2)
var IPFS_DEF_SHARD_STR = IPFS_DEF_SHARD.String()
var README_IPFS_DEF_SHARD = `` /* 1123-byte string literal not displayed */
Functions ¶
func DirIsEmpty ¶
From: http://stackoverflow.com/questions/30697324/how-to-check-if-directory-on-path-is-empty
func DowngradeV1toV0 ¶
func UpgradeV0toV1 ¶
func WriteReadme ¶
func WriteShardFunc ¶
Types ¶
type Datastore ¶
type Datastore struct {
// contains filtered or unexported fields
}
Datastore implements the go-datastore Interface. Note this datastore cannot guarantee order of concurrent write operations to the same key. See the explanation in Put().
func CreateOrOpen ¶
convenience method
func (*Datastore) Accuracy ¶
Accuracy returns a string representing the accuracy of the DiskUsage() result, the value returned is implementation defined and for informational purposes only
func (*Datastore) Delete ¶
Delete removes a key/value from the Datastore. Please read the Put() explanation about the handling of concurrent write operations to the same key.
func (*Datastore) DiskUsage ¶
DiskUsage implements the PersistentDatastore interface and returns the current disk usage in bytes used by this datastore.
The size is approximative and may slightly differ from the real disk values.
func (*Datastore) Put ¶
Put stores a key/value in the datastore.
Note, that we do not guarantee order of write operations (Put or Delete) to the same key in this datastore.
For example. i.e. in the case of two concurrent Put, we only guarantee that one of them will come through, but cannot assure which one even if one arrived slightly later than the other. In the case of a concurrent Put and a Delete operation, we cannot guarantee which one will win.
type ShardIdV1 ¶
type ShardIdV1 struct {
// contains filtered or unexported fields
}