dispatch

README

Simple task dispatcher

Provides an easy-to-use command to dispatch tasks described in a YAML file.

Common use cases:

• Launching multiple elementary tasks in parallel
• Add a condition with a task dependent on another
• Split SQL files to execute statements as elementary tasks
• Behave as \gexec on multiple connections

Usage

Usage:
  dispatch [command]

Available Commands:
  exec        Execute tasks from commands
  help        Help about any command
  run         Run tasks from configuration file

Global Flags:
  -c, --config string   configuration file
  -d, --dbname string   database name to connect to
      --help            show help
  -h, --host string     database server host or socket directory
  -j, --jobs int        number of workers (default 2)
  -l, --log string      log file
  -W, --password        force password prompt
  -p, --port int        database server port
  -U, --user string     database user name
  -v, --verbose         verbose mode

Usage:
  dispatch run [flags]

Flags:
  -t, --type string   parser type (default "sh")

Usage:
  dispatch exec [flags]

Flags:
  -C, --command string   loader command to execute
  -f, --file string      file containing commands to execute
  -T, --to string        executor type (default "sh")
  -t, --type string      parser type (default "sh")

Examples

Dispatch statements stored in a file:

cat <<EOF > statements.sql
CREATE INDEX t1_idx1 ON t1(col1);
CREATE INDEX t2_idx1 ON t2(col1);
EOF

dispatch exec -j 2 --type psql --file statements.sql

2023/05/22 18:19:08 Worker 1 completed Task 0 (query #1) (success: true, elapsed: 12ms)
2023/05/22 18:19:08 Worker 2 completed Task 0 (query #0) (success: true, elapsed: 12ms)

Dispatch generated statements from a command:

dispatch exec --type psql --to sh \
  --command "SELECT format('vacuumdb --table %s', tablename) FROM pg_tables WHERE tablename LIKE 'a%';"

Configuration

Use a valid YAML file to describe tasks.

Tasks declaration

• tasks: list of tasks to run
  • must be a valid array of tasks as described below

Elementary task

• id (required)
• name: as task description
• type: execution context in following choices
  • sh (default)
  • psql: needs PostgreSQL psql client to be installed
• command: instruction to be executed
• uri: connection string used by psql's database option (-d)
• connection: connection name as described below, overrides uri
• depends_on: a list of identifiers of others tasks declared upstream

All PostgreSQL environment variables can be used in place of uri as it used psql client.

# run the following shell commands simultaneously
tasks:
  - id: 1
    command: echo foo
  - id: 2
    command: echo bar

# execute SQL statement with psql on localhost with default credentials
tasks:
  - id: 1
    type: psql
    name: run this statement
    command: SELECT user;
    uri: postgresql://localhost

# make a task dependent from another
tasks:
  - id: 1
    command: echo foo
  - id: 2
    command: echo bar
    depends_on: [1]

Loader tasks

A loader is an extended task that dispatch instructions from a result command or a file. Delimiter detection is provided by Fragment package and only PgSQL and Shell languages are supported.

To read and dispatch instructions from a file, use this:

• file: instructions to be loaded from a file

# run queries from a file simultaneously
tasks:
  - id: 1
    type: psql
    name: dispatch queries from a file
    file: queries.sql

To dispatch commands from a specific result command, use the following configuration:

• loaded: in place of command
  • from: source execution context
  • command: instruction to be executed

# run queries generated by another query in parallel
tasks:
  - id: 1
    type: sh
    name: execute reindexdb for all table except log
    loaded:
      from: psql
      command: |
        SELECT format('reindexdb -v -t %I;', tablename) FROM pg_tables
        WHERE schemaname = 'public' AND tablename NOT IN ('log'
)

Traces

• logfile: summary of the tasks execution (default: disabled)
  • must be a valid path

logfile: result.out

Named connections

• connections: declares named connections used by tasks
  • name: connection name (default applied to any unattached tasks)
  • uri: a valid connection URI, takes precedence over following values
  • service: service defined in PGSERVICEFILE
  • host: database server host or socket directory
  • port: database server port
  • dbname: database name to connect to
  • user: database user name
  • password: user password

connections:
  - name: db
    uri: postgresql://remote
  - name: default
    host: localhost
    dbname: postgres
    user: postgres

tasks:
  - id: 1
    type: psql
    command: \conninfo
    connection: db

Parallelism

• workers: declares number of workers
  • explicit argument passed to command takes precedence
  • limited by the number of logical CPUs usable by the main process

workers: 1

# run the following tasks sequentially
tasks:
  - id: 1
    command: echo foo
  - id: 2
    command: echo bar

Testing

Bats testing framework is used. End-to-end tests are located under t/ directory A local PostgreSQL instance is required with postgres/postgres authentication or trust method in pg_hba.conf

go build -tags testing
bats t

# or using
make test

Unit tests are provided under internal packages.

go test ./...