gofile

package module
v1.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 23, 2023 License: MIT Imports: 8 Imported by: 1

Documentation

Overview

Package gofile provides wrappers to manage big numbers of files seamlessly.

It takes care of managing opened and closed files, holding their seek positions, flags etc. So you can use ManagedFile as any other file and "open" as many such files as you want. Also it provides the transaction API, so you can safely make subsequential calls to the same file across threads.

Index

Constants

View Source 
const (
	// How many file handles the system can hold simultaneously (applies if GetFilesLimit doesn't work)
	// mandatory: MAX_OPENED_FILES % 4 = 0
	DEFAULT_OPENED_FILES_LIMIT = 32

	// Deduced from tests falling on "failed to create new OS thread" error
	// when there are more than 4096 opened files on 8-threaded machine
	APPROX_MAX_GOROUTINES_PER_THREAD = 512
)
View Source 
const POS_REWIND = -1

Variables

View Source 
var ErrOutOfBound = errors.New("requested slice is out of the file's bounds")

Functions

func GetFilesLimit added in v1.2.0 

func GetFilesLimit() (nLimit uint64, err error)

GetFilesLimit returns the soft limit for simultaneously opened files

Types

type BufferedFile 

type BufferedFile struct {
	File

	BufferSize     int64
	BufferDataSize int64
	// contains filtered or unexported fields
}

A BufferedFile can be used to wrap any File compliant object and make buffered calls to it.

func NewBufferedFile 

func NewBufferedFile(file File, bufferSize int64) (b *BufferedFile, err error)

NewBufferedFile wraps existing File with BufferedFile and sets given bufferSize to it.

func (*BufferedFile) Append 

func (b *BufferedFile) Append(data []byte) (pos int64, err error)

Append appends given data at the end of the file

func (*BufferedFile) Close 

func (b *BufferedFile) Close()

Close closes the file (you can reopen it later).

func (*BufferedFile) PipeTo 

func (b *BufferedFile) PipeTo(dest Pipeable, chunkSize int64) (err error)

PipeTo pipes (copies) the data from start of the file and append it to another (dest) by chunks of size chunkSize. NOT BUFFERED YET

func (*BufferedFile) Position added in v1.2.0 

func (b *BufferedFile) Position() (pos int64)

Returns the seek position of the file

func (*BufferedFile) Read added in v1.2.0 

func (b *BufferedFile) Read(data []byte) (nRead int64, err error)

Read reads bytes from a file at the seek position.

It returns the number of bytes read and the error if there is one.

func (*BufferedFile) ReadAt 

func (b *BufferedFile) ReadAt(data []byte, offset int64) (nRead int64, err error)

ReadAt reads bytes from a file at given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "read the last byte of the file".

func (*BufferedFile) Seek 

func (b *BufferedFile) Seek(offset int64, whence int) (ret int64, err error)

Seek changes current read/write position of the file. The offset value is relative to the whence position which is file start, current pos or end (0, 1 or 2 respectively)

func (*BufferedFile) Size 

func (b *BufferedFile) Size() (length int64, err error)

Size returns current size of the file.

func (*BufferedFile) TRead 

func (b *BufferedFile) TRead(m func(txn Readable) (err error)) (err error)

TRead executes given function m locking the file for writing. You can only read from this transaction.

NOT BUFFERED YET

func (*BufferedFile) TReadWrite 

func (b *BufferedFile) TReadWrite(m func(txn Editable) (err error)) (err error)

TReadWrite executes given function m locking the file for writing/reading from other places.

NOT BUFFERED YET

func (*BufferedFile) TWrite 

func (b *BufferedFile) TWrite(m func(txn Writable) (err error)) (err error)

TWrite executes given function m locking the file for writing from other places. You can only write from this transaction.

NOT BUFFERED YET

func (*BufferedFile) Trunc 

func (b *BufferedFile) Trunc() (err error)

Trunc truncates given file and sets it's size to 0.

func (*BufferedFile) Write 

func (b *BufferedFile) Write(data []byte) (nWritten int64, err error)

Write writes the data to a file at the seek position.

It returns the number of bytes written and the error if there is one.

func (*BufferedFile) WriteAt 

func (b *BufferedFile) WriteAt(data []byte, offset int64) (nWritten int64, err error)

WriteAt writes bytes to a file at given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "write the last byte of the file".

type CommonMethods added in v1.2.0 

type CommonMethods interface {
	Seek(offset int64, whence int) (ret int64, err error)
	Size() (length int64, err error)
	Position() (pos int64)
}

type CommonTxn added in v1.2.0 

type CommonTxn struct {
	CommonMethods
	// contains filtered or unexported fields
}

func (*CommonTxn) Position added in v1.2.0 

func (t *CommonTxn) Position() (pos int64)

Returns the seek position of the file

func (*CommonTxn) Seek added in v1.2.0 

func (t *CommonTxn) Seek(offset int64, whence int) (ret int64, err error)

Seek changes current read/write position of the file. The offset value is relative to the whence position which is file start, current pos or end (0, 1 or 2 respectively)

func (*CommonTxn) Size added in v1.2.0 

func (t *CommonTxn) Size() (length int64, err error)

Size returns current size of the file.

type Editable 

type Editable interface {
	Readable
	Writable
}

type File 

type File interface {
	Editable
	Freeable
	Transactional

	Close()
}

type Freeable 

type Freeable interface {
	Trunc() (err error)
}

type ManagedFile 

type ManagedFile struct {
	Path string
	// contains filtered or unexported fields
}

A ManagedFile wraps a native go file, allowing it to be managed e.g. hold its size, position, state etc.

func NewFile 

func NewFile(filePath string, perm fs.FileMode) (file *ManagedFile, err error)

Creates a managed file.

Custom flags are unavailable here since the nature of the manager requires the access methods to be as uniform as possible.

func (*ManagedFile) Append 

func (f *ManagedFile) Append(data []byte) (pos int64, err error)

Append appends given data at the end of the file

It returns the position at which the data is written [Thread safe]

func (*ManagedFile) Close 

func (f *ManagedFile) Close()

Close closes the file. [Thread safe]

func (*ManagedFile) PipeTo 

func (f *ManagedFile) PipeTo(dest Pipeable, chunkSize int64) (err error)

PipeTo pipes (copies) the data from start of the file and append it to another (dest) by chunks of size chunkSize. [Thread safe]

func (*ManagedFile) Position added in v1.2.0 

func (f *ManagedFile) Position() (pos int64)

Returns the seek position of the file [Thread safe]

func (*ManagedFile) Read added in v1.2.0 

func (f *ManagedFile) Read(b []byte) (nRead int64, err error)

Read reads bytes from a file at the seek position.

It returns the number of bytes read and the error if there is one. [Thread safe]

func (*ManagedFile) ReadAt 

func (f *ManagedFile) ReadAt(b []byte, offset int64) (nRead int64, err error)

ReadAt reads bytes from a file at given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "read the last byte of the file". [Thread safe]

func (*ManagedFile) Seek 

func (f *ManagedFile) Seek(offset int64, whence int) (ret int64, err error)

Seek changes current read/write position of the file. The offset value is relative to the whence position which is file start, current pos or end (0, 1 or 2 respectively) [Thread safe]

func (*ManagedFile) Size 

func (f *ManagedFile) Size() (length int64, err error)

Size returns current size of the file. [Thread safe]

func (*ManagedFile) TRead 

func (f *ManagedFile) TRead(m func(txn Readable) (err error)) (err error)

TRead executes given function m locking the file for writing. You can only read from this transaction.

Also the transaction's txn methods are faster than ManagedFile methods [Thread safe]

func (*ManagedFile) TReadWrite 

func (f *ManagedFile) TReadWrite(m func(txn Editable) (err error)) (err error)

TReadWrite executes given function m locking the file for writing/reading from other places.

Also the transaction's txn methods are faster than ManagedFile methods [Thread safe]

func (*ManagedFile) TWrite 

func (f *ManagedFile) TWrite(m func(txn Writable) (err error)) (err error)

TWrite executes given function m locking the file for writing from other places. You can only write from this transaction.

Also the transaction's txn methods are faster than ManagedFile methods [Thread safe]

func (*ManagedFile) Trunc 

func (f *ManagedFile) Trunc() (err error)

Trunc truncates given file and sets it's size to 0 [Thread safe]

func (*ManagedFile) Write 

func (f *ManagedFile) Write(b []byte) (nWritten int64, err error)

Write writes the data to a file at the seek position.

It returns the number of bytes written and the error if there is one. [Thread safe]

func (*ManagedFile) WriteAt 

func (f *ManagedFile) WriteAt(b []byte, offset int64) (nWritten int64, err error)

WriteAt writes bytes to a file at given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "write the last byte of the file". [Thread safe]

type ManagedFileRWTxn 

type ManagedFileRWTxn struct {
	CommonTxn
	// contains filtered or unexported fields
}

func (*ManagedFileRWTxn) Append added in v1.2.0 

func (t *ManagedFileRWTxn) Append(data []byte) (pos int64, err error)

Append appends given data at the end of the file

It returns the position at which the data is written

func (*ManagedFileRWTxn) Read added in v1.2.0 

func (t *ManagedFileRWTxn) Read(b []byte) (nRead int64, err error)

Read reads bytes from a file at the seek position.

It returns the number of bytes read and the error if there is one.

func (*ManagedFileRWTxn) ReadAt added in v1.2.0 

func (t *ManagedFileRWTxn) ReadAt(b []byte, offset int64) (nRead int64, err error)

ReadAt reads bytes from a file at a given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "read the last byte of the file".

func (*ManagedFileRWTxn) Write added in v1.2.0 

func (t *ManagedFileRWTxn) Write(b []byte) (nWritten int64, err error)

Write writes the data to a file at the seek position.

It returns the number of bytes written and the error if there is one.

func (*ManagedFileRWTxn) WriteAt added in v1.2.0 

func (t *ManagedFileRWTxn) WriteAt(b []byte, offset int64) (nWritten int64, err error)

WriteAt writes bytes to a file at given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "write the last byte of the file".

type ManagedFileReadableTxn 

type ManagedFileReadableTxn struct {
	CommonTxn
	// contains filtered or unexported fields
}

func (*ManagedFileReadableTxn) Read added in v1.3.3 

func (t *ManagedFileReadableTxn) Read(b []byte) (nRead int64, err error)

Read reads bytes from a file at the seek position.

It returns the number of bytes read and the error if there is one.

func (*ManagedFileReadableTxn) ReadAt 

func (t *ManagedFileReadableTxn) ReadAt(b []byte, offset int64) (nRead int64, err error)

ReadAt reads bytes from a file at a given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "read the last byte of the file".

type ManagedFileWritableTxn 

type ManagedFileWritableTxn struct {
	CommonTxn
	// contains filtered or unexported fields
}

func (*ManagedFileWritableTxn) Append 

func (t *ManagedFileWritableTxn) Append(data []byte) (pos int64, err error)

Append appends given data at the end of the file

It returns the position at which the data is written

func (*ManagedFileWritableTxn) Write 

func (t *ManagedFileWritableTxn) Write(b []byte) (nWritten int64, err error)

Write writes the data to a file at the seek position.

It returns the number of bytes written and the error if there is one.

func (*ManagedFileWritableTxn) WriteAt 

func (t *ManagedFileWritableTxn) WriteAt(b []byte, offset int64) (nWritten int64, err error)

WriteAt writes bytes to a file at given offset. The offset value can be negative to specify offset from the end of the file.

I.e. offset = -2 will mean "write the last byte of the file".

type Pipeable added in v1.2.0 

type Pipeable interface {
	Transactional
	Writable
}

type Readable 

type Readable interface {
	CommonMethods

	ReadAt(b []byte, offset int64) (nRead int64, err error)
	Read(b []byte) (nRead int64, err error)
}

Readable objects should support reading operations

type Transactional added in v1.2.0 

type Transactional interface {
	TRead(m func(txn Readable) (err error)) (err error)
	TWrite(m func(txn Writable) (err error)) (err error)
	TReadWrite(m func(txn Editable) (err error)) (err error)
	PipeTo(dest Pipeable, chunkSize int64) (err error)
}

Transactional objects support methods that create any transactions

type Writable 

type Writable interface {
	CommonMethods

	Write(b []byte) (nWritten int64, err error)
	WriteAt(b []byte, offset int64) (nWritten int64, err error)
	Append(data []byte) (pos int64, err error)
}

Writable objects should support writing operations

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.