pgpkg

package module
v0.0.0-...-b1666d1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2024 License: MIT Imports: 20 Imported by: 0

README

pgpkg - simplifies Postgresql stored procedure development.

pgpkg logo

pgpkg is a small and fast command-line tool (and Go library) that lets your pl/pgSQL functions live side-by-side with regular code, allowing you to use the exact same workflows, source code control, IDE (or non-IDE) and other development workflows for both SQL and non-SQL code. It automatically deploys your functions without the need to maintain migration scripts.

You can edit your SQL functions in the same IDE, commit them to the same Git repository, review them with PRs alongside native code changes, and deploy them seamlessly to production.

pgpkg also lets you package up your SQL code and incorporate it as a dependency into other projects, similar to Node packages or Go modules, but note that dependency support is still early days.

Documentation

The best place to start is the pgpkg man page.

Tutorial

The tutorial for using pgpkg contains a worked example for writing functions, unit tests and migration scripts. If you're familiar with Postgresql, it will only take a few minutes to work through.

Status

pgpkg is late alpha. I apologise in advance if documentation or examples are out of date.

I use pgpkg pretty much on a daily basis. It works really well for me, and I'm working to remove the rough edges.

I work on making it easier to use when I have time. The TODO list contains issues that I expect to fix over time.

PRs and issues are welcome.

Documentation

Initial set of documentation is here. I have been focusing on writing the manual and tutorial, so other documents may currently be out of date.

License

pgpkg is licensed under the same terms as Postgresql itself.

Contributing

Contributors welcome. Contact me at mark@commandquery.com

Documentation

Index

Constants

View Source 
const PGKSchemaName = "pgpkg"

PGKSchemaName is the name of the pgpkg schema itself.

Variables

View Source 
var CachePkgNotFound = errors.New("package not found in cache")
View Source 
var ErrUserRequest = errors.New("terminating due to user request")

Used when a --option requires the caller to quit, but there wasn't an error. e.g., --dry-run

View Source 
var Options struct {
	Verbose        bool           // print lots of stuff
	Summary        bool           // print a summary of the installation
	DryRun         bool           // rollback after installation (default)
	ShowTests      bool           // Show the result of each SQL test that was run.
	ShowSkipped    bool           // Show skipped tests
	SkipTests      bool           // Don't run the tests. Useful when fixing them!
	IncludePattern *regexp.Regexp // Pattern to use for running tests
	ExcludePattern *regexp.Regexp // Pattern to use for running tests
}
View Source 
var Stderr = log.New(os.Stderr, "pgpkg: ", log.LstdFlags)
View Source 
var Stdout = log.New(os.Stdout, "pgpkg: ", log.LstdFlags)
View Source 
var Verbose = Stdout

Functions

func AsString 

func AsString(node *pg_query.Node) string

AsString is a utility function to get the string value of a node.

func CheckPackageName 

func CheckPackageName(name string) error

CheckPackageName checks if the given string is a valid pgpkg package name and returns an error if not.

func Exit 

func Exit(err error)

Exit usually prints the error message (with context, if available), and then exits immediately with status 1. However, err == ErrUserRequest then no message is printed and we exit with status 0. project.Open() will return ErrUserRequest if the command-line options indicate a dry run or other condition that should not result in a program printing an error.

Applications should call pgpkg.Exit(err) after calling project.Open() if err is not nil.

func LogLouder 

func LogLouder()

func LogQuieter 

func LogQuieter()

func ParseArgs 

func ParseArgs(prefix string) error

ParseArgs parses the os.Args for a standard set of OS arguments. ParseArgs deletes matching arguments from os.Args so that the caller doesn't need to worry about them.

When embedding pgpkg into your own programs, set prefix to "pgpkg" to differentiate pgpkg arguments from your own. Doing this will make it possible to set pgpkg options from your code are runtime with prefixed options such as "--dry-run". If "prefix" is empty ("") then options will not be prefixed; ie, "--dry-run".

You should call ParseArgs before calling flag.Parse() if you are using the standard flag library.

func PrintError 

func PrintError(err error)

func QualifiedName 

func QualifiedName(nodes []*pg_query.Node) string

func Remote 

func Remote()

func Sanitize 

func Sanitize(pattern *regexp.Regexp, v string) string

func SanitizeSlice 

func SanitizeSlice(pattern *regexp.Regexp, values []string) []string

func WriteProject 

func WriteProject(z *zip.Writer, p *Project) error

Types

type Bundle 

type Bundle struct {
	Package *Package         // canonical name of the package.
	Path    string           // Path of this bundle, relative to the Package
	Index   map[string]*Unit // Index of location of each unit.
	Units   []*Unit          // Ordered list of build units within the bundle
}

func (*Bundle) HasUnits 

func (b *Bundle) HasUnits() bool

HasUnits indicates if any build units were found for this bundle.

func (*Bundle) Location 

func (b *Bundle) Location() string

func (*Bundle) Open 

func (b *Bundle) Open(path string) (fs.File, error)

Open an arbitrary file from the bundle.

func (*Bundle) PrintInfo 

func (b *Bundle) PrintInfo(w InfoWriter)

type Cache 

type Cache interface {
	GetCachedSource(pkgName string) (Source, error)
}

Cache represents a dependency cache which contains the source of any dependencies listed in the "Uses" section of pgpkg.toml.

Users can manually import project dependencies into the project cache with `pgpkg import`.

type DirSource 

type DirSource struct {
	// contains filtered or unexported fields
}

func NewDirSource 

func NewDirSource(path string) *DirSource

func (*DirSource) Cache 

func (ds *DirSource) Cache() (Cache, error)

func (*DirSource) Location 

func (ds *DirSource) Location() string

func (*DirSource) Open 

func (ds *DirSource) Open(name string) (fs.File, error)

func (*DirSource) Path 

func (ds *DirSource) Path() string

Path returns the path that this DirSource refers to, allowing discovery of the underlying filesystem location.

func (*DirSource) String 

func (ds *DirSource) String() string

func (*DirSource) Sub 

func (ds *DirSource) Sub(path string) (Source, error)

type FSSource 

type FSSource struct {
	// contains filtered or unexported fields
}

func NewFSSource 

func NewFSSource(efs fs.FS, dir string) (*FSSource, error)

func (*FSSource) Cache 

func (f *FSSource) Cache() (Cache, error)

func (*FSSource) Location 

func (f *FSSource) Location() string

func (*FSSource) Open 

func (f *FSSource) Open(name string) (fs.File, error)

func (*FSSource) String 

func (f *FSSource) String() string

func (*FSSource) Sub 

func (f *FSSource) Sub(path string) (Source, error)

type InfoWriter 

type InfoWriter struct {
	// contains filtered or unexported fields
}

func NewInfoWriter 

func NewInfoWriter(w io.Writer) InfoWriter

func (InfoWriter) Print 

func (i InfoWriter) Print(name string, value any)

func (InfoWriter) Printf 

func (i InfoWriter) Printf(format string, args ...any)

func (InfoWriter) Println 

func (i InfoWriter) Println(args ...any)

type MOB 

type MOB struct {
	*Bundle
	// contains filtered or unexported fields
}

func (*MOB) Apply 

func (m *MOB) Apply(tx *PkgTx) error

Apply performs the SQL required to create the objects listed in the MOB object, to register them in the pgpkg.object table. Since objects in an MOB may depend on one another, this function starts with a list of the statements to be executed, and attempts to execute them one at a time.

Each statement is executed in a savepoint. If a statement fails, we skip over it and keep trying.

The apply function will keep running until it's unable to create any statement, after which it will terminate.

func (*MOB) DefaultContext 

func (m *MOB) DefaultContext() *PKGErrorContext

func (*MOB) Location 

func (m *MOB) Location() string

func (*MOB) Parse 

func (m *MOB) Parse() error

func (*MOB) PrintInfo 

func (m *MOB) PrintInfo(w InfoWriter)

type ManagedObject 

type ManagedObject struct {
	ObjectSchema string
	ObjectType   string
	ObjectName   string
	ObjectArgs   []string
}

ManagedObject refers to a managed database object, with a schema name, object name and object type.

type PKGError 

type PKGError struct {
	Message string
	Object  PKGObject
	Context *PKGErrorContext
	Err     error

	// In the case of applying a migration, there may be multiple errors,
	// e.g. if a MOB can't be processed (typically because of dependencies).
	Errors []*PKGError
}

PKGError is the error type used internally by pgpkg.

func PKGErrorf 

func PKGErrorf(object PKGObject, err error, msg string, args ...any) *PKGError

func (*PKGError) Error 

func (e *PKGError) Error() string

func (*PKGError) GetContext 

func (e *PKGError) GetContext() *PKGErrorContext

func (*PKGError) PrintRootContext 

func (e *PKGError) PrintRootContext(contextLines int)

Print prints useful information about this error.

func (*PKGError) Unwrap 

func (e *PKGError) Unwrap() error

func (*PKGError) UnwrapAll 

func (e *PKGError) UnwrapAll() []*PKGError

UnwrapAll unwraps the errors until we get to the very last PKGError.

type PKGErrorContext 

type PKGErrorContext struct {
	Source     string
	LineNumber int
	Location   string
	Next       *PKGErrorContext // Indicates addtional stack traces.
}

func (*PKGErrorContext) Print 

func (c *PKGErrorContext) Print(contextLines int)

type PKGObject 

type PKGObject interface {
	Location() string
	DefaultContext() *PKGErrorContext
}

PKGObject is any object (statement, unit, package) that can tell us where a problem happened.

type Package 

type Package struct {
	Project     *Project
	Name        string   // canonical, unique name of the pgpkg package
	Location    string   // Location of this package
	Source      Source   // Source of the package (dir, zip, embedded, ...)
	SchemaNames []string // Packages participate in one or more schemas
	RoleName    string   // Associated role name

	StatFuncCount      int // Stat showing the number of functions in the package
	StatViewCount      int // Stat showing the number of views in the package
	StatTriggerCount   int // Stat showing the number of triggers in the package
	StatMigrationCount int // Stat showing how many migration scripts were run
	StatTestCount      int // Stat showing how many tests there are.

	Schema *Schema
	MOB    *MOB
	Tests  *Tests

	IsDependency bool // This package was loaded from .pgpkg cache
	// contains filtered or unexported fields
}

func (*Package) AddUses 

func (p *Package) AddUses(pkg string) bool

AddUses adds the given package name to the Uses clause of the package. Returns false if the package already exists in the Uses clause. Note that this does not update the config file; to do this, see WriteConfig.

func (*Package) Apply 

func (p *Package) Apply(tx *PkgTx) error

func (*Package) PrintInfo 

func (p *Package) PrintInfo(w InfoWriter)

func (*Package) WriteConfig 

func (p *Package) WriteConfig() error

type PkgTx 

type PkgTx struct {
	*sql.Tx
}

func (*PkgTx) Exec 

func (t *PkgTx) Exec(query string, args ...any) (sql.Result, error)

func (*PkgTx) Query 

func (t *PkgTx) Query(query string, args ...any) (*sql.Rows, error)

func (*PkgTx) QueryRow 

func (t *PkgTx) QueryRow(query string, args ...any) *sql.Row

type Project 

type Project struct {
	Root    *Package // the root (main package) of this project
	Sources []Source // All package sources, in no particular order.

	Cache  *WriteCache // primary cache for this project
	Search []Cache     // other caches to search for dependencies.
	// contains filtered or unexported fields
}

Project represents a collection of individual packages that are to be installed into a single database. This struct is responsible for tracking the package sources that make up a project, including dependencies and caches, and arranging for them to be installed in the correct order.

You work with a project by adding the "sources" you need - which might be directories, ZIP files, embedded filesystems, or embedded ZIP binaries. If a project- or search-cache is defined, then this will be used to find dependencies.

Once you've added all the sources for your project, p.Open() or p.Migrate() performs the migration.

The `pgpkg` package is always installed automatically, and is never exported.

func NewProject 

func NewProject() *Project

NewProject creates a new project. It adds the "pgpkg" package to the project, which is required to track the objects we create and remove.

func NewProjectFrom 

func NewProjectFrom(pkgPath string, searchCaches ...Cache) (*Project, error)

NewProjectFrom creates a new project and adds the package found at path. It also configures (and possibly creates) a project cache, also rooted at the given path. If searchCaches is not nil, these will be searched in order when resolving dependencies. Search caches take precedence over the project cache.

func (*Project) AddEmbeddedFS 

func (p *Project) AddEmbeddedFS(f fs.FS, path string) (*Package, error)

func (*Project) AddPackage 

func (p *Project) AddPackage(source Source, isDependency bool) (*Package, error)

AddPackage adds an individual package to the project.

func (*Project) AddSource 

func (p *Project) AddSource(src Source) (*Package, error)

func (*Project) Init 

func (p *Project) Init(tx *PkgTx) error

Init initialises the pgpkg schema itself. It effectively uses pgpkg's migration tools to bookstrap itself.

func (*Project) Migrate 

func (p *Project) Migrate(dsn string) error

func (*Project) Open 

func (p *Project) Open(dsn string) (*sql.DB, error)

Open opens the given database, installs the packages from the project, and returns the database connection.

Open is the main entry point for pgpkg.

Packages are installed within a single transaction. Migrations and tests are applied automatically. Package installation is atomic; it either fully succeeds or fails without changing the database.

If this method returns an error, you should call pgpkg.Exit(err) to exit. This call checks that the error was significant and will adjust the OS exit status accordingly. See pgpkg.Exit() for more details.

If dsn is an empty string, pgpkg will attempt to use the PGPKG_DSN environment variable. If PGPKG_DSN is not set, pgpkg will use the usual libpq PG environment variables.

func (*Project) Parse 

func (p *Project) Parse() error

Parse prepares a project for migrating or other processing by resolving any dependencies and parsing the schemas.

func (*Project) PrintInfo 

func (p *Project) PrintInfo(w InfoWriter)

type ReadCache 

type ReadCache struct {
	// contains filtered or unexported fields
}

func NewReadCache 

func NewReadCache(cfs fs.FS) *ReadCache

func (*ReadCache) GetCachedSource 

func (c *ReadCache) GetCachedSource(pkgName string) (Source, error)

type Schema 

type Schema struct {
	*Bundle
	// contains filtered or unexported fields
}

func (*Schema) Apply 

func (s *Schema) Apply(tx *PkgTx) error

Apply executes the schema statements in order.

func (*Schema) ApplyUnit 

func (s *Schema) ApplyUnit(tx *PkgTx, u *Unit) error

func (*Schema) PrintInfo 

func (s *Schema) PrintInfo(w InfoWriter)

type Source 

type Source interface {
	// FS is implemented by every Source.
	fs.FS

	// Sub returns a source representing a subdirectory within the source.
	Sub(dir string) (Source, error)

	// Location should return the actual path for a source, taking account
	// any subpaths that have been extracted from it. This is going to require a different
	// format and handling for directories, embeds, ZIPs, and other objects.
	Location() string

	// Cache returns the cache for this source, if one exists. You should return
	// a WriteCache from this function if your source supports writing. FIXME.
	Cache() (Cache, error)
}

Source represents the tree of files in a package; it's basically a wrapper around fs.FS, but adds context. Source lets us access filesystems in any format, which currently includes filesystems (eg, for use with go:embed), ZIP files (for packaging), and local directories.

Sources may include a cache, which could be either a read or write cache, depending on the type of source.

func NewSource 

func NewSource(pkgPath string) (Source, error)

NewSource returns a Source based on the given filesystem path. If the path name ends in ".zip", NewSource will return a ZipByteSource. Otherwise, NewSource returns a DirSource.

type Statement 

type Statement struct {
	Unit       *Unit             // Unit this statement appears in
	LineNumber int               // Line number within the Unit
	Source     string            // The actual SQL
	Tree       *pg_query.RawStmt // Parsed SQL statement.
	Error      error             // The most recent result from processing the statement.
	// contains filtered or unexported fields
}

Statement is a parsed SQL statement within a unit.

func (*Statement) DefaultContext 

func (s *Statement) DefaultContext() *PKGErrorContext

func (*Statement) GetManagedObject 

func (s *Statement) GetManagedObject() (*ManagedObject, error)

GetManagedObject returns identifying information about an object from a CREATE statement, such as function, view or trigger. NOTE: This functon might not support all object types, but you can add more as needed.

The result is cached since it's used repeatedly during MOB processing.

func (*Statement) Headline 

func (s *Statement) Headline() string

Headline returns the first line of the statement, eg, to provide context during debugging and logging.

func (*Statement) Location 

func (s *Statement) Location() string

func (*Statement) LocationOffset 

func (s *Statement) LocationOffset(offset int) string

func (*Statement) Try 

func (s *Statement) Try(tx *PkgTx) (bool, error)

Try executes a statement in a savepoint. This allows us to find context if statement execution fails.

Returns true if the statement succeeded, or true-with-error if it failed but could be retried (this depends on where the error occurred). Returns false if an error occurred that was not related to statement execution.

If an error occurs while executing the statement, the statement's Error field is also set.

type Tests 

type Tests struct {
	*Bundle

	NamedTests map[string]*Statement
	// contains filtered or unexported fields
}

func (*Tests) PrintInfo 

func (t *Tests) PrintInfo(w InfoWriter)

func (*Tests) Run 

func (t *Tests) Run(tx *PkgTx) error

type Unit 

type Unit struct {

	// The Bundle that this unit belongs to.
	Bundle *Bundle

	// Path is the filename within the Bundle FS that this Unit
	// should read from when it's parsed.
	Path string

	// The contents (SQL statements) declared in the unit.
	Source string

	// The list of parsed statements in this unit.
	Statements []*Statement
}

Unit (ie, build unit) represents potentially parsable tree of SQL source code taken from a single file. Units are lazily loaded, and don't parse their contents until requested, with the Statements() function. Once the unit is compiled, individual statements contain line number and other debugging information.

func (*Unit) DefaultContext 

func (u *Unit) DefaultContext() *PKGErrorContext

func (*Unit) Location 

func (u *Unit) Location() string

func (*Unit) Parse 

func (u *Unit) Parse() error

Parse a unit.

type WriteCache 

type WriteCache struct {
	ReadCache
	// contains filtered or unexported fields
}

func NewWriteCache 

func NewWriteCache(dir string) *WriteCache

func (*WriteCache) ImportProject 

func (c *WriteCache) ImportProject(srcProject *Project) error

ImportProject imports the given project into the cache. If the project has dependencies, these are imported from the child project's cache, unless they are already present in the target cache.

func (*WriteCache) RemovePackage 

func (c *WriteCache) RemovePackage(pkgName string) error

RemovePackage removes (deletes) a package from the cache.

type ZipByteSource 

type ZipByteSource struct {
	// contains filtered or unexported fields
}

func NewZipByteSource 

func NewZipByteSource(zipBytes []byte) (*ZipByteSource, error)

NewZipByteSource creates a new ZIP source from a byte slice.

func NewZipPathSource 

func NewZipPathSource(path string) (*ZipByteSource, error)

NewZipPathSource creates a new ZIP source from a filesystem path. This reads the whole ZIP file into memory and returns a ZipByteSource.

func (*ZipByteSource) Cache 

func (zs *ZipByteSource) Cache() (Cache, error)

func (*ZipByteSource) Location 

func (zs *ZipByteSource) Location() string

func (*ZipByteSource) Open 

func (zs *ZipByteSource) Open(name string) (fs.File, error)

func (*ZipByteSource) String 

func (zs *ZipByteSource) String() string

func (*ZipByteSource) Sub 

func (zs *ZipByteSource) Sub(dir string) (Source, error)

Sub returns a subtree of a ZipByteSource, as a ZipByteSource.

Source Files

View all Source files

Directories

Path Synopsis
cmd
pgpkg

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.