staticgen

README

Staticgen

A static website generator that lets you use HTTP servers and frameworks you already know. Just tell Staticgen how to start your server, then watch it crawl your site and generate a static version with all of the pages and assets required.

About

If you're unfamiliar, you can actually use the decades-old wget command to output a static website from a dynamic one, this project is purpose-built for the same idea, letting your team to use whatever HTTP servers and frameworks you're already familiar with, in any language.

I haven't done any scientific benchmarks or comparisons yet, but here are some results on my 2014 8-core MBP:

• Compiles 3,296 pages of the Signal v. Noise blog in 1 second
• Compiles my Apex Software site in 150ms

Installation

Via gobinaries.com:

$ curl -sf https://gobinaries.com/tj/staticgen/cmd/staticgen | sh

Configuration

Configuration is stored within a ./static.json file in your project's root directory. The following options are available:

• command — The server command executed before crawling.
• url — The target website to crawl. Defaults to "http://127.0.0.1:3000".
• dir — The static website output directory. Defaults to "build".
• pages — A list of paths added to crawl, typically including unlinked pages such as landing pages. Defaults to [].
• concurrency — The number of concurrent pages to crawl. Defaults to 30.

Guide

First create the ./static.json configuration file, for example here's the config for Go server, the only required property is command:

{
  "command": "go run main.go",
  "concurrency": 50,
  "dir": "dist"
}

Below is an example of a Node.js server, note that NODE_ENV is assigned to production so that optimizations such as Express template caches are used to improve serving performance.

{
  "command": "NODE_ENV=production node server.js"
}

Run the staticgen command to start the pre-rendering process:

$ staticgen

Staticgen executes the command you provided, waits for the server to become available on the url configured. The pages and assets are copied to the dir configured and then your server is shut down.

By default the timeout for the generation process is 15 minutes, depending on your situation you may want to increase or decrease this with the -t, --timeout flag, here are some examples:

$ staticgen -t 30s
$ staticgen -t 15m
$ staticgen -t 1h

When launching the command, Staticgen sets the STATICGEN environment variable to 1, allowing you to alter behaviour if necessary.

To view the pre-rendered site run the following command to start a static file server and open the browser:

$ staticgen serve

See the examples directory for full examples.

Notes

Staticgen does not pre-render using a headless browser, this makes it faster, however it means that you cannot rely on client-side JavaScript manipulating the page.

---

Sponsors

This project is sponsored by CTO.ai, making it easy for development teams to create and share workflow automations without leaving the command line.

And my GitHub sponsors:

Documentation

Overview

Package staticgen provides static website generation from a live server.

Index

• type Config
  • func (c *Config) Load(path string) error
• type Event
• type EventStartCrawl
• type EventStartedServer
• type EventStartingServer
• type EventStopCrawl
• type EventStoppingServer
• type EventVisitedResource
• type Generator
  • func (g *Generator) Report(ch chan<- Event)
  • func (g *Generator) Run(ctx context.Context) error
  • func (g *Generator) Start(ctx context.Context) error
  • func (g *Generator) Wait() error
• type Reporter
  • func (r *Reporter) Report(ch <-chan Event) <-chan struct{}
• type Target

Types

type Config

type Config struct {
	// URL is the target website to crawl. Defaults to "http://127.0.0.1:3000".
	URL string `json:"url"`

	// Dir is the static website output directory. Defaults to "build".
	Dir string `json:"dir"`

	// Command is the optional server command executed before crawling.
	Command string `json:"command"`

	// Pages is a list of paths added to crawl, typically
	// including unlinked pages such as error pages,
	// landing pages and so on.
	Pages []string `json:"pages"`

	// Concurrency is the number of concurrent pages to crawl. Defaults to 30.
	Concurrency int `json:"concurrency"`

	// Allow404 can be enabled to opt-in to pages resulting in a 404,
	// which otherwise lead to an error.
	Allow404 bool `json:"allow_404"`
}

Config is the static website generator configuration.

func (*Config) Load

func (c *Config) Load(path string) error

Load configuration from the given path.

type Event

type Event interface {
	// contains filtered or unexported methods
}

Event is an event.

type EventStartCrawl

type EventStartCrawl struct{}

EventStartCrawl .

type EventStartedServer

type EventStartedServer struct {
	Command string
	URL     string
}

EventStartedServer .

type EventStartingServer

type EventStartingServer struct {
	Command string
	URL     string
}

EventStartingServer .

type EventStopCrawl

type EventStopCrawl struct{}

EventStopCrawl .

type EventStoppingServer

type EventStoppingServer struct{}

EventStoppingServer .

type EventVisitedResource

type EventVisitedResource struct {
	Target
	Duration   time.Duration
	StatusCode int
	Filename   string
	Error      error
}

EventVisitedResource .

type Generator

type Generator struct {
	// Config used for crawling and producing the static website.
	Config

	// HTTPClient ...
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

Generator is a static website generator.

func (*Generator) Report

func (g *Generator) Report(ch chan<- Event)

Report registers a channel for reporting on events.

func (*Generator) Run

func (g *Generator) Run(ctx context.Context) error

Run starts the configured server command, starts to perform crawling, and waits for completion before shutting down the configured server.

func (*Generator) Start

func (g *Generator) Start(ctx context.Context) error

Start loads configuration from ./static.json, starts the configured server, and begins the crawling process.

func (*Generator) Wait

func (g *Generator) Wait() error

Wait for crawling to complete.

type Reporter

type Reporter struct {
	// contains filtered or unexported fields
}

A Reporter outputs a human-friendly report of events.

func (*Reporter) Report

func (r *Reporter) Report(ch <-chan Event) <-chan struct{}

Report on the given event channel. Returns a channel which is closed when the event channel is closed, and all reporting has been completed.

type Target

type Target struct {
	Parent *url.URL
	URL    *url.URL
}

Target is a target URL.