vfs

package
v0.23.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 11, 2025 License: MIT Imports: 24 Imported by: 4

README

Go SQLite VFS API

This package implements the SQLite OS Interface (aka VFS).

It replaces the default SQLite VFS with a pure Go implementation, and exposes interfaces that should allow you to implement your own custom VFSes.

Since it is a from scratch reimplementation, there are naturally some ways it deviates from the original.

The main differences are file locking and WAL mode support.

File Locking

POSIX advisory locks, which SQLite uses on Unix, are broken by design. Instead, on Linux and macOS, this package uses OFD locks to synchronize access to database files.

This package can also use BSD locks, albeit with reduced concurrency (BEGIN IMMEDIATE behaves like BEGIN EXCLUSIVE). BSD locks are the default on BSD and illumos, but you can opt into them with the sqlite3_flock build tag.

On Windows, this package uses LockFileEx and UnlockFileEx, like SQLite.

You can also opt into a cross-platform locking implementation with the sqlite3_dotlk build tag.

Otherwise, file locking is not supported, and you must use nolock=1 (or immutable=1) to open database files. To use the database/sql driver with nolock=1 you must disable connection pooling by calling db.SetMaxOpenConns(1).

You can use vfs.SupportsFileLocking to check if your build supports file locking.

Write-Ahead Logging

On Unix, this package may use mmap to implement shared-memory for the WAL-index, like SQLite.

On Windows, this package may use MapViewOfFile, like SQLite.

You can also opt into a cross-platform, in-process, memory sharing implementation with the sqlite3_dotlk build tag.

Otherwise, WAL support is limited, and EXCLUSIVE locking mode must be set to create, read, and write WAL databases. To use EXCLUSIVE locking mode with the database/sql driver you must disable connection pooling by calling db.SetMaxOpenConns(1).

You can use vfs.SupportsSharedMemory to check if your build supports shared memory.

Batch-Atomic Write

On Linux, this package may support batch-atomic writes on the F2FS filesystem.

Checksums

This package can be configured to add an 8-byte checksum to the end of every page in an SQLite database. The checksum is added as each page is written and verified as each page is read.
The checksum is intended to help detect database corruption caused by random bit-flips in the mass storage device.

The implementation is compatible with SQLite's Checksum VFS Shim.

Build Tags

The VFS can be customized with a few build tags:

  • sqlite3_flock forces the use of BSD locks.
  • sqlite3_dotlk forces the use of dot-file locks.

[!IMPORTANT] The default configuration of this package is compatible with the standard Unix and Windows SQLite VFSes; sqlite3_flock builds are compatible with the unix-flock VFS; sqlite3_dotlk builds are compatible with the unix-dotfile VFS. If incompatible file locking is used, accessing databases concurrently with other SQLite libraries will eventually corrupt data.

Custom VFSes

Documentation

Overview

Package vfs wraps the C SQLite VFS API.

Index

Constants

View Source 
const SupportsFileLocking = true

SupportsFileLocking is false on platforms that do not support file locking. To open a database file on those platforms, you need to use the nolock or immutable URI parameters.

View Source 
const SupportsSharedMemory = true

SupportsSharedMemory is false on platforms that do not support shared memory. To use WAL without shared-memory, you need to set EXCLUSIVE locking mode.

Variables

This section is empty.

Functions

func ExportHostFunctions 

func ExportHostFunctions(env wazero.HostModuleBuilder) wazero.HostModuleBuilder

ExportHostFunctions is an internal API users need not call directly.

ExportHostFunctions registers the required VFS host functions with the provided env module.

func Register 

func Register(name string, vfs VFS)

Register registers a VFS. Empty and "os" are reserved names.

https://sqlite.org/c3ref/vfs_find.html

func Unregister 

func Unregister(name string)

Unregister unregisters a VFS.

https://sqlite.org/c3ref/vfs_find.html

Types

type AccessFlag 

type AccessFlag uint32

AccessFlag is a flag for the VFS Access method.

https://sqlite.org/c3ref/c_access_exists.html 

const (
	ACCESS_EXISTS    AccessFlag = 0
	ACCESS_READWRITE AccessFlag = 1 /* Used by PRAGMA temp_store_directory */
	ACCESS_READ      AccessFlag = 2 /* Unused */
)

type DeviceCharacteristic 

type DeviceCharacteristic uint32

DeviceCharacteristic is a flag retuned by the File DeviceCharacteristics method.

https://sqlite.org/c3ref/c_iocap_atomic.html 

const (
	IOCAP_ATOMIC                DeviceCharacteristic = 0x00000001
	IOCAP_ATOMIC512             DeviceCharacteristic = 0x00000002
	IOCAP_ATOMIC1K              DeviceCharacteristic = 0x00000004
	IOCAP_ATOMIC2K              DeviceCharacteristic = 0x00000008
	IOCAP_ATOMIC4K              DeviceCharacteristic = 0x00000010
	IOCAP_ATOMIC8K              DeviceCharacteristic = 0x00000020
	IOCAP_ATOMIC16K             DeviceCharacteristic = 0x00000040
	IOCAP_ATOMIC32K             DeviceCharacteristic = 0x00000080
	IOCAP_ATOMIC64K             DeviceCharacteristic = 0x00000100
	IOCAP_SAFE_APPEND           DeviceCharacteristic = 0x00000200
	IOCAP_SEQUENTIAL            DeviceCharacteristic = 0x00000400
	IOCAP_UNDELETABLE_WHEN_OPEN DeviceCharacteristic = 0x00000800
	IOCAP_POWERSAFE_OVERWRITE   DeviceCharacteristic = 0x00001000
	IOCAP_IMMUTABLE             DeviceCharacteristic = 0x00002000
	IOCAP_BATCH_ATOMIC          DeviceCharacteristic = 0x00004000
	IOCAP_SUBPAGE_READ          DeviceCharacteristic = 0x00008000
)

type File 

type File interface {
	Close() error
	ReadAt(p []byte, off int64) (n int, err error)
	WriteAt(p []byte, off int64) (n int, err error)
	Truncate(size int64) error
	Sync(flags SyncFlag) error
	Size() (int64, error)
	Lock(lock LockLevel) error
	Unlock(lock LockLevel) error
	CheckReservedLock() (bool, error)
	SectorSize() int
	DeviceCharacteristics() DeviceCharacteristic
}

A File represents an open file in the OS interface layer.

Use sqlite3.ErrorCode or sqlite3.ExtendedErrorCode to return specific error codes to SQLite. In particular, sqlite3.BUSY is necessary to correctly implement lock methods.

https://sqlite.org/c3ref/io_methods.html

type FileBatchAtomicWrite 

type FileBatchAtomicWrite interface {
	File
	BeginAtomicWrite() error
	CommitAtomicWrite() error
	RollbackAtomicWrite() error
}

FileBatchAtomicWrite extends File to implement the SQLITE_FCNTL_BEGIN_ATOMIC_WRITE, SQLITE_FCNTL_COMMIT_ATOMIC_WRITE and SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE file control opcodes.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlbeginatomicwrite

type FileBusyHandler added in v0.22.0 

type FileBusyHandler interface {
	File
	BusyHandler(func() bool)
}

FileBusyHandler extends File to implement the SQLITE_FCNTL_BUSYHANDLER file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlbusyhandler

type FileCheckpoint added in v0.15.0 

type FileCheckpoint interface {
	File
	CheckpointStart()
	CheckpointDone()
}

FileCheckpoint extends File to implement the SQLITE_FCNTL_CKPT_START and SQLITE_FCNTL_CKPT_DONE file control opcodes.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlckptstart

type FileChunkSize added in v0.15.0 

type FileChunkSize interface {
	File
	ChunkSize(size int)
}

FileChunkSize extends File to implement the SQLITE_FCNTL_CHUNK_SIZE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlchunksize

type FileCommitPhaseTwo 

type FileCommitPhaseTwo interface {
	File
	CommitPhaseTwo() error
}

FileCommitPhaseTwo extends File to implement the SQLITE_FCNTL_COMMIT_PHASETWO file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlcommitphasetwo

type FileHasMoved 

type FileHasMoved interface {
	File
	HasMoved() (bool, error)
}

FileHasMoved extends File to implement the SQLITE_FCNTL_HAS_MOVED file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlhasmoved

type FileLockState 

type FileLockState interface {
	File
	LockState() LockLevel
}

FileLockState extends File to implement the SQLITE_FCNTL_LOCKSTATE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntllockstate

type FileOverwrite added in v0.9.0 

type FileOverwrite interface {
	File
	Overwrite() error
}

FileOverwrite extends File to implement the SQLITE_FCNTL_OVERWRITE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntloverwrite

type FilePersistWAL added in v0.22.0 

type FilePersistWAL interface {
	File
	PersistWAL() bool
	SetPersistWAL(bool)
}

FilePersistWAL extends File to implement the SQLITE_FCNTL_PERSIST_WAL file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal

type FilePowersafeOverwrite 

type FilePowersafeOverwrite interface {
	File
	PowersafeOverwrite() bool
	SetPowersafeOverwrite(bool)
}

FilePowersafeOverwrite extends File to implement the SQLITE_FCNTL_POWERSAFE_OVERWRITE file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpowersafeoverwrite

type FilePragma added in v0.15.0 

type FilePragma interface {
	File
	Pragma(name, value string) (string, error)
}

FilePragma extends File to implement the SQLITE_FCNTL_PRAGMA file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpragma

type FileSharedMemory added in v0.15.0 

type FileSharedMemory interface {
	File
	SharedMemory() SharedMemory
}

FileSharedMemory extends File to possibly implement shared-memory for the WAL-index. The same shared-memory instance must be returned for the entire life of the file. It's OK for SharedMemory to return nil.

type FileSizeHint 

type FileSizeHint interface {
	File
	SizeHint(size int64) error
}

FileSizeHint extends File to implement the SQLITE_FCNTL_SIZE_HINT file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlsizehint

type FileSync added in v0.22.0 

type FileSync interface {
	File
	SyncSuper(super string) error
}

FileSync extends File to implement the SQLITE_FCNTL_SYNC file control opcode.

https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlsync

type FileUnwrap added in v0.20.0 

type FileUnwrap interface {
	File
	Unwrap() File
}

FileUnwrap should be implemented by a File that wraps another File implementation.

type Filename added in v0.15.0 

type Filename struct {
	// contains filtered or unexported fields
}

Filename is used by SQLite to pass filenames to the Open method of a VFS.

https://sqlite.org/c3ref/filename.html

func GetFilename added in v0.18.0 

func GetFilename(ctx context.Context, mod api.Module, id ptr_t, flags OpenFlag) *Filename

GetFilename is an internal API users should not call directly.

func (*Filename) Database added in v0.15.0 

func (n *Filename) Database() string

Database returns the name of the corresponding database file.

https://sqlite.org/c3ref/filename_database.html

func (*Filename) DatabaseFile added in v0.15.0 

func (n *Filename) DatabaseFile() File

DatabaseFile returns the main database File corresponding to a journal.

https://sqlite.org/c3ref/database_file_object.html

func (*Filename) Journal added in v0.15.0 

func (n *Filename) Journal() string

Journal returns the name of the corresponding rollback journal file.

https://sqlite.org/c3ref/filename_database.html

func (*Filename) String added in v0.15.0 

func (n *Filename) String() string

String returns this filename as a string.

func (*Filename) URIParameter added in v0.15.0 

func (n *Filename) URIParameter(key string) string

URIParameter returns the value of a URI parameter.

https://sqlite.org/c3ref/uri_boolean.html

func (*Filename) URIParameters added in v0.15.0 

func (n *Filename) URIParameters() url.Values

URIParameters obtains values for URI parameters.

https://sqlite.org/c3ref/uri_boolean.html

func (*Filename) WAL added in v0.15.0 

func (n *Filename) WAL() string

Journal returns the name of the corresponding WAL file.

https://sqlite.org/c3ref/filename_database.html

type LockLevel 

type LockLevel uint32

LockLevel is a value used with File Lock and Unlock methods.

https://sqlite.org/c3ref/c_lock_exclusive.html 

const (
	// No locks are held on the database.
	// The database may be neither read nor written.
	// Any internally cached data is considered suspect and subject to
	// verification against the database file before being used.
	// Other processes can read or write the database as their own locking
	// states permit.
	// This is the default state.
	LOCK_NONE LockLevel = 0 /* xUnlock() only */

	// The database may be read but not written.
	// Any number of processes can hold SHARED locks at the same time,
	// hence there can be many simultaneous readers.
	// But no other thread or process is allowed to write to the database file
	// while one or more SHARED locks are active.
	LOCK_SHARED LockLevel = 1 /* xLock() or xUnlock() */

	// A RESERVED lock means that the process is planning on writing to the
	// database file at some point in the future but that it is currently just
	// reading from the file.
	// Only a single RESERVED lock may be active at one time,
	// though multiple SHARED locks can coexist with a single RESERVED lock.
	// RESERVED differs from PENDING in that new SHARED locks can be acquired
	// while there is a RESERVED lock.
	LOCK_RESERVED LockLevel = 2 /* xLock() only */

	// A PENDING lock means that the process holding the lock wants to write to
	// the database as soon as possible and is just waiting on all current
	// SHARED locks to clear so that it can get an EXCLUSIVE lock.
	// No new SHARED locks are permitted against the database if a PENDING lock
	// is active, though existing SHARED locks are allowed to continue.
	LOCK_PENDING LockLevel = 3 /* internal use only */

	// An EXCLUSIVE lock is needed in order to write to the database file.
	// Only one EXCLUSIVE lock is allowed on the file and no other locks of any
	// kind are allowed to coexist with an EXCLUSIVE lock.
	// In order to maximize concurrency, SQLite works to minimize the amount of
	// time that EXCLUSIVE locks are held.
	LOCK_EXCLUSIVE LockLevel = 4 /* xLock() only */
)

type OpenFlag 

type OpenFlag uint32

OpenFlag is a flag for the VFS Open method.

https://sqlite.org/c3ref/c_open_autoproxy.html 

const (
	OPEN_READONLY      OpenFlag = 0x00000001 /* Ok for sqlite3_open_v2() */
	OPEN_READWRITE     OpenFlag = 0x00000002 /* Ok for sqlite3_open_v2() */
	OPEN_CREATE        OpenFlag = 0x00000004 /* Ok for sqlite3_open_v2() */
	OPEN_DELETEONCLOSE OpenFlag = 0x00000008 /* VFS only */
	OPEN_EXCLUSIVE     OpenFlag = 0x00000010 /* VFS only */
	OPEN_AUTOPROXY     OpenFlag = 0x00000020 /* VFS only */
	OPEN_URI           OpenFlag = 0x00000040 /* Ok for sqlite3_open_v2() */
	OPEN_MEMORY        OpenFlag = 0x00000080 /* Ok for sqlite3_open_v2() */
	OPEN_MAIN_DB       OpenFlag = 0x00000100 /* VFS only */
	OPEN_TEMP_DB       OpenFlag = 0x00000200 /* VFS only */
	OPEN_TRANSIENT_DB  OpenFlag = 0x00000400 /* VFS only */
	OPEN_MAIN_JOURNAL  OpenFlag = 0x00000800 /* VFS only */
	OPEN_TEMP_JOURNAL  OpenFlag = 0x00001000 /* VFS only */
	OPEN_SUBJOURNAL    OpenFlag = 0x00002000 /* VFS only */
	OPEN_SUPER_JOURNAL OpenFlag = 0x00004000 /* VFS only */
	OPEN_NOMUTEX       OpenFlag = 0x00008000 /* Ok for sqlite3_open_v2() */
	OPEN_FULLMUTEX     OpenFlag = 0x00010000 /* Ok for sqlite3_open_v2() */
	OPEN_SHAREDCACHE   OpenFlag = 0x00020000 /* Ok for sqlite3_open_v2() */
	OPEN_PRIVATECACHE  OpenFlag = 0x00040000 /* Ok for sqlite3_open_v2() */
	OPEN_WAL           OpenFlag = 0x00080000 /* VFS only */
	OPEN_NOFOLLOW      OpenFlag = 0x01000000 /* Ok for sqlite3_open_v2() */
)

type SharedMemory added in v0.15.0 

type SharedMemory interface {
	io.Closer
	// contains filtered or unexported methods
}

SharedMemory is a shared-memory WAL-index implementation. Use NewSharedMemory to create a shared-memory.

func NewSharedMemory added in v0.15.0 

func NewSharedMemory(path string, flags OpenFlag) SharedMemory

NewSharedMemory returns a shared-memory WAL-index backed by a file with the given path. It will return nil if shared-memory is not supported, or not appropriate for the given flags. Only OPEN_MAIN_DB databases may need a WAL-index. You must ensure all concurrent accesses to a database use shared-memory instances created with the same path.

type SyncFlag 

type SyncFlag uint32

SyncFlag is a flag for the File Sync method.

https://sqlite.org/c3ref/c_sync_dataonly.html 

const (
	SYNC_NORMAL   SyncFlag = 0x00002
	SYNC_FULL     SyncFlag = 0x00003
	SYNC_DATAONLY SyncFlag = 0x00010
)

type VFS 

type VFS interface {
	Open(name string, flags OpenFlag) (File, OpenFlag, error)
	Delete(name string, syncDir bool) error
	Access(name string, flags AccessFlag) (bool, error)
	FullPathname(name string) (string, error)
}

A VFS defines the interface between the SQLite core and the underlying operating system.

Use sqlite3.ErrorCode or sqlite3.ExtendedErrorCode to return specific error codes to SQLite.

https://sqlite.org/c3ref/vfs.html

func Find 

func Find(name string) VFS

Find returns a VFS given its name. If there is no match, nil is returned. If name is empty, the default VFS is returned.

https://sqlite.org/c3ref/vfs_find.html

type VFSFilename added in v0.15.0 

type VFSFilename interface {
	VFS
	OpenFilename(name *Filename, flags OpenFlag) (File, OpenFlag, error)
}

VFSFilename extends VFS with the ability to use Filename objects for opening files.

https://sqlite.org/c3ref/filename.html

Source Files

View all Source files

Directories

Path Synopsis
Package adiantum wraps an SQLite VFS to offer encryption at rest.
 Package adiantum wraps an SQLite VFS to offer encryption at rest.
Package memdb implements the "memdb" SQLite VFS.
 Package memdb implements the "memdb" SQLite VFS.
Package readervfs implements an SQLite VFS for immutable databases.
 Package readervfs implements an SQLite VFS for immutable databases.
Package xts wraps an SQLite VFS to offer encryption at rest.
 Package xts wraps an SQLite VFS to offer encryption at rest.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.