slugcmplr

README

slugcmplr

slugcmplr allows you to compile Heroku compatible slugs using official and custom buildpacks, in the same manner as the Heroku Slug Compiler

This enables more control over the build process and allows for detaching building and releasing your Heroku applications.

Install

With the go toolchain installed:

# In order to use this as a library
go get github.com/cga1123/slugcmplr

# In order to install as a CLI
go install github.com/cga1123/slugcmplr@latest

There are also precompiled binaries and their associated checksums are available attached to tagged releases.

Info

slugcmplr has 3 steps/sub-commands:

prepare [APPLICATION] --build-dir [BUILD-DIR] --source-dir [SOURCE-DIR]

In the prepare step, slugcmplr will fetch the metadata required to compile your application. It will copy your project SOURCE-DIR into BUILD-DIR/app.

It will fetch the buildpacks as defined by your Heroku application, download, and decompress them into BUILD-DIR/buildpacks. If using official buildpacks (e.g. heroku/go, heroku/ruby) slugcmplr will use the same buildpack as currently deployed to Heroku's production environment.

It will fetch the config vars as defined by your Heroku application and put them into the BUILD-DIR/environment directory.

Finally, prepare writes metadata (such as the application name, stack, buildpack order, source version) to the BUILD-DIR/compile.json file to allow the compile step to bootstrap itself.

compile --build-dir [BUILD-DIR] --cache-dir [CACHE-DIR]

In the compile step, slugcmplr executes your buildpacks in the specified order, outputs your Heroku slug, and uploads it to Heroku for future release.

compile will output the slug for your application to BUILD-DIR/app.tgz

compile will output metadata about the compilation to BUILD-DIR/release.tgz, this contains information such as the slug ID as uploaded to Heroku.

The CACHE-DIR will be used by the buildpacks as their cache argument to speed up builds in the future, as per the Buildpack API

To guarantee full compatibility, it is recommended to run this step using Heroku's build containers. heroku/heroku:20-build or heroku/heroku:18-build.

release --build-dir [BUILD-DIR]

In the release step, slugcmplr triggers a release of your previously compiled slug.

It uses the BUILD-DIR/release.json file in order to fetch metadata in order to create this release.

You can optionally pass --app [APPLICATION] to target an application that is different from the one you built from. This will work as long as the applications are in the same Heroku team. (This is becuase the slug must be accessible to the application).

You can optionally pass --commit [COMMIT] to associate this release with a separate commit from the one used to build this slug initially.

Authentication

The slugcmplr CLI looks for credentials to api.heroku.com in your .netrc file, this is the same technique used by heroku/cli and so if you are currently making use of the heroku command during CI, you should already be logging in somehow and have no issues.

Otherwise populating your .netrc should be a case of adding something equivalent to the following script (assuming the HEROKU_EMAIL and HEROKU_API_KEY are correctly populated environment variables):

cat << EOF >> ${HOME}/.netrc
machine api.heroku.com
  login ${HEROKU_EMAIL}
  password ${HEROKU_API_KEY}
EOF

By default, slugcmplr will look in ${HOME}/.netrc for the credentials, however it will respect the ${NETRC} environment variable if set and non-empty.

Testing

The majority of tests for this project are acceptance tests that will create and release live Heroku applications. These require the correct credentials to be set in the environment as well as setting a sentinel value to execute the acceptance tests:

SLUGCMPLR_ACC=true \
  SLUGCMPLR_ACC_HEROKU_PASS=<HEROKU-API-KEY> \
  SLUGCMPLR_ACC_HEROKU_EMAIL=<HEROKU-EMAIL> \
  go test -v

Fixture application are hosted in separate repositories which will be cloned and created by using the withHarness function. Fixture repositories are expected to contain a app.json file which describes the Heroku applications.

See app.json Schema

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/CGA1123/slugcmplr

See issues if you want any inspiration as to what to help with or of any pending discussion/work.

This project is Codespaces compatible! If you want to get started quickly spin one up.

The script directory should help you get going with building and testing.

Some buildpacks are not compatible with arm/arm64 architectures, you may experience issues testing slugcmplr on an M1 Chip, even if using docker containers.

---

For more background on this you might find this Medium article helpful.

Documentation

Overview

Package slugcmplr exposes functions to transform a source repository to a Heroku compatible tarball known as a "slug" via a set of Buildpacks.

Index

• func Commit(dir string) (string, error)
• func UploadBlob(ctx context.Context, method, url, path string) error
• type BuildpackReference
• type CompileCmd
  • func (c *CompileCmd) Execute(ctx context.Context, out Outputter) (*CompileResult, error)
• type CompileResult
• type MetadataCmd
  • func (m *MetadataCmd) Execute(ctx context.Context, _ Outputter) (*MetadataResult, error)
• type MetadataResult
• type Outputter
  • func OutputterFromCmd(cmd *cobra.Command, verbose bool) Outputter
• type PrepareCmd
  • func (p *PrepareCmd) Execute(ctx context.Context, _ Outputter) (*PrepareResult, error)
• type PrepareResult
• type ReleaseCmd
  • func (r *ReleaseCmd) Execute(ctx context.Context, _ Outputter) (*ReleaseInfo, error)
• type ReleaseInfo
• type StdOutputter
  • func (o *StdOutputter) ErrOrStderr() io.Writer
  • func (o *StdOutputter) IsVerbose() bool
  • func (o *StdOutputter) OutOrStdout() io.Writer
• type Tarball
  • func Targz(srcDirPath, dstDirPath string) (*Tarball, error)
• type UploadCmd
  • func (u *UploadCmd) Execute(ctx context.Context, o Outputter) (*UploadResult, error)
• type UploadResult

Functions

func Commit

func Commit(dir string) (string, error)

Commit attempts to return the current resolved HEAD commit for the git repository at dir.

func UploadBlob

func UploadBlob(ctx context.Context, method, url, path string) error

UploadBlob uploads the file at the given path to the url using the given method.

Types

type BuildpackReference

type BuildpackReference struct {
	Name string
	URL  string
}

BuildpackReference is a reference to a buildpack, containing its raw URL and Name.

type CompileCmd

type CompileCmd struct {
	CacheDir      string
	BuildDir      string
	Stack         string
	SourceVersion string
	Buildpacks    []*buildpack.Buildpack
}

CompileCmd wraps up all the information required to compile the contents of SourceDir into a deployable artifact/slug.

func (*CompileCmd) Execute

func (c *CompileCmd) Execute(ctx context.Context, out Outputter) (*CompileResult, error)

Execute applies the buildpacks to the SourceDir in their specific order, before compressing the result of these applications into a GZipped Tar file within BuildDir.

type CompileResult

type CompileResult struct {
	SlugPath          string
	SlugChecksum      string
	SourceVersion     string
	Procfile          processfile.Procfile
	DetectedBuildpack string
	Stack             string
}

CompileResult contains metadata about the result of Executing CompileCmd.

type MetadataCmd

type MetadataCmd struct {
	Heroku      *heroku.Service
	Application string
	SourceDir   string
	BuildDir    string
}

MetadataCmd wraps up all the information required to fetch metadata require for compilation of application.

func (*MetadataCmd) Execute

func (m *MetadataCmd) Execute(ctx context.Context, _ Outputter) (*MetadataResult, error)

Execute fetches the applications name, stack, buildpacks, and config variables. It will also resolve the current HEAD commit of the source to be compiled.

type MetadataResult

type MetadataResult struct {
	ApplicationName string
	Stack           string
	Buildpacks      []*BuildpackReference
	ConfigVars      map[string]string
	SourceVersion   string
}

MetadataResult contains the result of resolving metadata for an application.

type Outputter

type Outputter interface {
	OutOrStdout() io.Writer
	ErrOrStderr() io.Writer
	IsVerbose() bool
}

Outputter mimics the interface implemented by *cobra.Command to inject custom Stdout and Stderr streams, while also allowing control over the verbosity of output.

func OutputterFromCmd

func OutputterFromCmd(cmd *cobra.Command, verbose bool) Outputter

OutputterFromCmd builds an Outputter based on a *cobra.Command.

type PrepareCmd

type PrepareCmd struct {
	SourceDir  string
	BuildDir   string
	ConfigVars map[string]string
	Buildpacks []*BuildpackReference
}

PrepareCmd wraps up all the information required to prepare an application for slug compilation.

func (*PrepareCmd) Execute

func (p *PrepareCmd) Execute(ctx context.Context, _ Outputter) (*PrepareResult, error)

Execute prepares an application for compilation by download all required buildpack, copying the source into the build directory, and writing out the application environment.

type PrepareResult

type PrepareResult struct {
	BuildDir   string
	Buildpacks []*buildpack.Buildpack
}

PrepareResult contains the result of preparing, including the path to the base build directory and metadata about the downloaded buildpacks -- their order and paths.

type ReleaseCmd

type ReleaseCmd struct {
	Heroku      *heroku.Service
	Application string
	SlugID      string
	Commit      string
}

ReleaseCmd wraps up all the information required to release a slug that has been uploaded to a Heroku application.

func (*ReleaseCmd) Execute

func (r *ReleaseCmd) Execute(ctx context.Context, _ Outputter) (*ReleaseInfo, error)

Execute will attempt to release a given slug to an application, returning the release ID and and OutputStreamURL, if there is one.

type ReleaseInfo

type ReleaseInfo struct {
	ID              string
	OutputStreamURL *string
}

ReleaseInfo contains the ID and OutputStreamURL of an attempted release.

type StdOutputter

type StdOutputter struct {
	Out     io.Writer
	Err     io.Writer
	Verbose bool
}

StdOutputter is an Outputter that will default to os.Stdout and os.Stderr.

func (*StdOutputter) ErrOrStderr

func (o *StdOutputter) ErrOrStderr() io.Writer

ErrOrStderr returns either the configured Err, or os.Stderr if it is nil.

func (*StdOutputter) IsVerbose

func (o *StdOutputter) IsVerbose() bool

IsVerbose returnes whether the StdOutputter is in Verbose mode.

func (*StdOutputter) OutOrStdout

func (o *StdOutputter) OutOrStdout() io.Writer

OutOrStdout returns either the configured Out, or os.Stdout if it is nil.

type Tarball

type Tarball struct {
	Path     string
	Checksum string
}

Tarball represents a GZipped Tar file.

func Targz

func Targz(srcDirPath, dstDirPath string) (*Tarball, error)

Targz will walk srcDirPath recursively and write the corresponding GZipped Tar Archive to the given writers.

type UploadCmd

type UploadCmd struct {
	Heroku            *heroku.Service
	Application       string
	Checksum          string
	Path              string
	DetectedBuildpack string
	SourceVersion     string
	Stack             string
	ProcessTypes      map[string]string
}

UploadCmd wraps up all the information required to upload a slug to a particular Heroku application.

func (*UploadCmd) Execute

func (u *UploadCmd) Execute(ctx context.Context, o Outputter) (*UploadResult, error)

Execute creates a new slug resource and uploads the compiled slug to it.

type UploadResult

type UploadResult struct {
	SlugID        string
	SourceVersion string
}

UploadResult returns metadata about the uploaded slug, so that it can be referred to or released later.