govfs

README

govfs

A Virtual Filesystem Library written in the Go programming language. Please note that this project is still under heavy development.

Synopsis

The govfs Virtual Filesystem is a ext3/4 Unix-based heirarchy type filesystem which begins from the root folder "/". Each file is represented by a meta header which may be referenced in O(1) time, i.e. this filesystem does not use a linked list to create a filesystem "tree", but rather a hash-table which allows for the quick reference of files.

In addition, govfs makes use of golang's gothreads by implementing very quick read/writes. I.E. there is a thread dispatched for each write operation (assuming it is a different file). So writing 1000 files concurrently is possible without breaking the filesystem.

Since this is an in-memory file system, upon unmount the filesystem data is stored in a serialized stream. This, too, is concurrent. So for each file that contains data, a thread will be spawned that will perform compression on the file and serialization of the meta data.

Features

1. 2^128 files
2. No file size limit
3. Concurrent reads/writes
4. O(1) reference time for finding a file header
5. Can print out file lists
6. File metadata serialization upon unmount (i.e. writing the entire filesystem to a physical file)
7. The filesystem file is OS independent
8. Creating one file will automatically create each subdirectory
9. Ideal for projects with heavy filesystem utilization
10. Compression and encryption of the raw filesystem file (unmounted file) is available

API

Main Filesystem Header

The FSHeader structure contains all data related to the virtual filesystem

type FSHeader struct {
    filename    string                             /* The raw file on disk */
    key         [16]byte                           /* RC4 key used to encrypt/decrypt the raw file */
    meta        map[string]*gofs_file              /* Hash table containing each file header */
    t_size      uint                               /* Total size of all files */
    [... Other structures/members omitted ...]
}

Create/Load Database

func CreateDatabase(name string, flags int) *FSHeader

Create New File

func (f *FSHeader) Create(name string) (*gofs_file, error)

Create I/O Reader

func (f *FSHeader) NewReader(name string) (*Reader, error)

Standard Read()

func (f *Reader) Read(p []byte) (int, error) 

I/O Reader

func (f *FSHeader) Read(name string) ([]byte, error)

Delete File

func (f *FSHeader) Delete(name string) error

Write to a file

func (f *FSHeader) Write(name string, d []byte) error

Writer interface

type Writer struct {
    Name string
    File *gofs_file
    Hdr *FSHeader
}

New Writer method

func (f *FSHeader) NewWriter(name string) (*Writer, error)

Write method

func (f *Writer) Write(p []byte) (int, error)

Disclaimer

Please see the LICENSE file for the detailed MIT license. All work written by Stan Ruzin stan [dot] ruzin [at] gmail [dot] com

Documentation

Index

• Constants
• type FSHeader
  • func CreateDatabase(name string, flags FlagVal) (*FSHeader, error)
  • func (f *FSHeader) Check(name string) bool
  • func (f *FSHeader) Commit() (*FSHeader, error)
  • func (f *FSHeader) Create(name string) error
  • func (f *FSHeader) Delete(name string) error
  • func (f *FSHeader) GetFileCount() uint
  • func (f *FSHeader) GetFileList() []string
  • func (f *FSHeader) GetFileListDirectory(dir string) ([]string, error)
  • func (f *FSHeader) GetFileSize(name string) (uint, error)
  • func (f *FSHeader) GetTotalFilesizes() int
  • func (f *FSHeader) NewReader(name string) (*Reader, error)
  • func (f *FSHeader) NewWriter(name string) (*Writer, error)
  • func (f *FSHeader) Read(name string) ([]byte, error)
  • func (f *FSHeader) StartIOController() error
  • func (f *FSHeader) UnmountDB(flags FlagVal) error
  • func (f *FSHeader) Write(name string, d []byte) error
• type FlagVal
• type RawFile
• type Reader
  • func (f *Reader) Len() int
  • func (f *Reader) Read(r []byte) (int, error)
• type Writer
  • func (f *Writer) Write(p []byte) (int, error)

Constants

const FS_SIGNATURE string = "govfs_header" /* Cannot exceed 64 */

const MAX_FILENAME_LENGTH int = 256

* Configurable constants

const REMOVE_FS_HEADER bool = false /* Removes the header at the beginning of the serialized file - leave false */

const STREAM_PAD_LEN int = 0 /* Length of the pad between two serialized RawFile structs */

Types

type FSHeader

type FSHeader struct {
	// contains filtered or unexported fields
}

func CreateDatabase

func CreateDatabase(name string, flags FlagVal) (*FSHeader, error)

* Creates or loads a filesystem database file. If the filename is nil, then create a new database * otherwise try to load an existing fs database file. * * Flags: FLAG_ENCRYPT, FLAG_COMPRESS

func (*FSHeader) Check

func (f *FSHeader) Check(name string) bool

* Exported method to check for object existence in db

func (*FSHeader) Commit

func (f *FSHeader) Commit() (*FSHeader, error)

* Commits in-memory objects to the disk

func (*FSHeader) Create

func (f *FSHeader) Create(name string) error

func (*FSHeader) Delete

func (f *FSHeader) Delete(name string) error

func (*FSHeader) GetFileCount

func (f *FSHeader) GetFileCount() uint

func (*FSHeader) GetFileList

func (f *FSHeader) GetFileList() []string

func (*FSHeader) GetFileListDirectory

func (f *FSHeader) GetFileListDirectory(dir string) ([]string, error)

* Retrieves the file listing in a specific directrory. NOTE: The * `dir` parameter must contain a trailing "/"

func (*FSHeader) GetFileSize

func (f *FSHeader) GetFileSize(name string) (uint, error)

func (*FSHeader) GetTotalFilesizes

func (f *FSHeader) GetTotalFilesizes() int

func (*FSHeader) NewReader

func (f *FSHeader) NewReader(name string) (*Reader, error)

func (*FSHeader) NewWriter

func (f *FSHeader) NewWriter(name string) (*Writer, error)

func (*FSHeader) Read

func (f *FSHeader) Read(name string) ([]byte, error)

func (*FSHeader) StartIOController

func (f *FSHeader) StartIOController() error

func (*FSHeader) UnmountDB

func (f *FSHeader) UnmountDB(flags FlagVal) error

func (*FSHeader) Write

func (f *FSHeader) Write(name string, d []byte) error

type FlagVal

type FlagVal int

const (
	IRP_PURGE  FlagVal = IRP_BASE + iota /* Flush the entire database and all files */
	IRP_DELETE                           /* Delete a file/folder */
	IRP_WRITE                            /* Write data to a file */
	IRP_CREATE                           /* Create a new file or folder */
)

const (
	FLAG_FILE      FlagVal = 1 << iota
	FLAG_DIRECTORY         /* The target file is a directory */
	FLAG_ENCRYPT           /* Encryption on the fs serialized output */
	FLAG_DB_LOAD           /* Loads the database */
	FLAG_DB_CREATE         /* Creates the database */
	FLAG_COMPRESS          /*
	 * Compresses files in the FS stream -- also used as a switch to determine
	 *  if a file should be compressed due to the chance of high entropy. If compression
	 *  takes places, then this flag is set on comp_file.Flags
	 */
)

const IRP_BASE FlagVal = 2 /*
 * Start the IRP controller ID count from n > 1.
 *  Altering this may break logic in the I/O controller
 */

type RawFile

type RawFile struct {
	RawSum      string
	Flags       FlagVal
	Name        string
	UnzippedLen int
}

* The meta header for each raw file * (govfsFile is the virtual, in-memory file header)

type Reader

type Reader struct {
	Name   string
	File   *govfsFile
	Hdr    *FSHeader
	Offset int
}

* Reader interface

func (*Reader) Len

func (f *Reader) Len() int

func (*Reader) Read

func (f *Reader) Read(r []byte) (int, error)

type Writer

type Writer struct {
	Name string
	File *govfsFile
	Hdr  *FSHeader
}

* Writer interface

func (*Writer) Write

func (f *Writer) Write(p []byte) (int, error)