pgseeder

README

pgseeder - PostgreSQL Database seeder

Efficiently seed your postgres database from files using a hierarchical dependency structure. Supports parallel execution, file streaming, and query batching.

See a basic example here.

Data file format

Each line in your data files should have the information required to create one database row. The data can be any format but I recommend json. The file then needs to be gzipped.

Example:

users.json

[
    {"id": 1, "name": "John"},
    {"id": 2, "name": "Jane"}
]

jq -c '.[]' users.json | gzip > users.gz

Documentation

Index

• type Config
• type Seeder
  • func New(pool *pgxpool.Pool, opts ...SeederOption) *Seeder
  • func (s *Seeder) Run(ctx context.Context, cfgs []Config) error
• type SeederOption
  • func WithProgress(p *mpb.Progress) SeederOption

Types

type Config

type Config struct {
	// The path to the file to be seeded
	FileName string
	// ChunkSize is the number of lines to be read at a time (defaults to 100)
	ChunkSize int
	// HandleLine is called for each line in the file.
	// If HandleLine returns an error, the seeder will stop running.
	HandleLine func(batch *pgx.Batch, line []byte) error
	// A slice of Configs that will be run after the parent Config has finished.
	Children []Config
}

Config represents a file to be seeded.

type Seeder

type Seeder struct {
	// contains filtered or unexported fields
}

func New

func New(pool *pgxpool.Pool, opts ...SeederOption) *Seeder

New creates a new PgSeeder. Available options are:

• WithProgress : pass in an *mpb.Progress to get progress bars

func (*Seeder) Run

func (s *Seeder) Run(ctx context.Context, cfgs []Config) error

Run runs the seeder. It takes a context and a slice of Configs. Each Config represents a file to be seeded. The Configs are run concurrently, and each Config can have children Configs. The children Configs are run after the parent Config has finished.

type SeederOption

type SeederOption = func(s *Seeder)

func WithProgress

func WithProgress(p *mpb.Progress) SeederOption