docsite

README

docsite

A documentation site generator that fits Sourcegraph's needs:

• Markdown source files that are browseable on the file system and readable as plain text (without custom directives or complex front matter or configuration)
• Served by an HTTP server, not generated as static HTML files, to eliminate the need for external static site host configuration (which we found to be error-prone)
• Provides built-in site search for all documentation versions

Usage

go get github.com/sourcegraph/docsite/cmd/docsite
docsite -h

• docsite check: check for common problems (such as broken links)
• docsite serve: serve the site over HTTP

To use docsite for docs.sourcegraph.com, see "Documentation site" in the Sourcegraph documentation.

Checks

The docsite check command runs various checks on your documentation site to find problems:

• invalid links
• broken links
• disconnected pages (with no inlinks from other pages)

If any problems are found, it exits with a non-zero status code.

To ignore the disconnected page check for a page, add YAML ignoreDisconnectedPageCheck: true to the top matter in the beginning of the .md file. For example:

---
ignoreDisconnectedPageCheck: true
---

# My page title

Site data

The site data describes the location of its templates, assets, and content. It is a JSON object with the following properties.

• content: a VFS URL for the Markdown content files.
• contentExcludePattern: a regular expression specifying Markdown content files to exclude.
• baseURLPath: the URL path where the site is available (such as / or /help/).
• rootURL: (optional) the root URL (scheme + host). Only used for rare cases where this is absolutely necessary, such as SEO tags fox example.
• templates: a VFS URL for the Go-style HTML templates used to render site pages.
• assets: a VFS URL for the static assets referred to in the HTML templates (such as CSS stylesheets).
• assetsBaseURLPath: the URL path where the assets are available (such as /assets/).
• redirects: an object mapping URL paths (such as /my/old/page) to redirect destination URLs (such as /my/new/page).
• check (optional): an object containing a single property ignoreURLPattern, which is a RE2 regexp of URLs to ignore when checking for broken URLs with docsite check.
• search (optional): an object containing a single proprety skipIndexURLPattern, which is a RE2 regexp pattern that if matching any content file URL will remove that file from the search index.

The possible values for VFS URLs are:

• A relative path to a local directory (such as ../myrepo/doc). The path is interpreted relative to the docsite.json file (if it exists) or the current working directory (if site data is specified in DOCSITE_CONFIG).

• An absolute URL to a Zip archive (with http or https scheme). The URL can contain a fragment (such as #mydir/) to refer to a specific directory in the archive.

  If the URL fragment contains a path component * (such as #*/templates/), it matches the first top-level directory in the Zip file. (This is useful when using GitHub Zip archive URLs, such as https://codeload.github.com/alice/myrepo/zip/myrev#*/templates/. GitHub produces Zip archives with a top-level directory $REPO-$REV, such as myrepo-myrev, and using #*/templates/ makes it easy to descend into that top-level directory without needing to duplicate the myrev in the URL fragment.)

  If the URL contains the literal string $VERSION, it is replaced by the user's requested version from the URL (e.g., the URL path /@foo/bar means the version is foo). ⚠️ If you are using GitHub codeload.github.com archive URLs, be sure your URL contains refs/heads/$VERSION (as in https://codeload.github.com/owner/repo/zip/refs/heads/$VERSION), not just $VERSION. This prevents someone from forking your repository, pushing a commit to their fork with unauthorized content, and then crafting a URL on your documentation site that would cause users to view that unauthorized content (which may contain malicious scripts or misleading information).

Templates

The templates use Go-style HTML templates.

• Document pages are rendered using a template named document.html.
• Search result pages are rendered using a template named search.html.
• The file root.html, if it exists, is loaded when rendering any template. You can define common templates in this file.

See the following examples:

• about.sourcegraph.com/handbook templates
• docs.sourcegraph.com templates

Redirects

In addition to the redirects property in site data, you can also specify redirects in a text file named redirects at the top level of the assets VFS. The format is as follows:

FROM-PATH TO-URL STATUS-CODE

For example:

# Comments are allowed
/my/old/page /my/new/page 308
/another/page https://example.com/page 308

Specifying site data

The docsite tool requires site data to be available in any of the following ways:

• A docsite.json file (or other file specified in the -config flag's search paths), as in the following example:

  {
    "content": "../sourcegraph/doc",
    "baseURLPath": "/",
    "templates": "templates",
    "assets": "assets",
    "assetsBaseURLPath": "/assets/",
    "check": {
      "ignoreURLPattern": "(^https?://)|(^#)|(^mailto:support@sourcegraph\\.com$)|(^chrome://)"
    }
  }
  

• In the DOCSITE_CONFIG env var, using Zip archive URLs for templates, assets, and content, as in the following example:

  DOCSITE_CONFIG='{"templates":"https://codeload.github.com/sourcegraph/sourcegraph/zip/refs/heads/main#*/doc/_resources/templates/","assets":"https://codeload.github.com/sourcegraph/sourcegraph/zip/refs/heads/main#*/doc/_resources/assets/","content":"https://codeload.github.com/sourcegraph/sourcegraph/zip/refs/heads/$VERSION#*/doc/","baseURLPath":"/","assetsBaseURLPath":"/assets/","defaultContentBranch":"main"}' docsite serve
  

Development

Release a new version

1. Build the Docker image for linux/amd64:

   docker build -t sourcegraph/docsite .
   
   # Use buildx if you're on M1
   docker buildx build --platform linux/amd64 -t sourcegraph/docsite .
   

2. Tag and push the image to Docker Hub and GCR:

   export VERSION= # e.g. v1.9.1
   docker tag sourcegraph/docsite sourcegraph/docsite:$VERSION
   docker tag sourcegraph/docsite gcr.io/sourcegraph-dogfood/docsite
   docker push sourcegraph/docsite
   docker push sourcegraph/docsite:$VERSION
   docker push gcr.io/sourcegraph-dogfood/docsite
   

3. For internal Sourcegraph usage:

   1. Bump the deployed version by updating the SHA-256 image digest in all files that define sourcegraph/docsite:latest@sha256.
   2. Once the pull request is merged, wait for the Buildkite build to pass.

4. For development, bump the version number in files that define DOCSITE_VERSION.

Documentation

Index

• func IsContentAsset(urlPath string) bool
• func ReadFile(fs http.FileSystem, path string) ([]byte, error)
• func WalkFileSystem(fs http.FileSystem, filterFn func(path string) bool, ...) error
• type ContentPage
• type PageData
• type Site
  • func (s *Site) AllContentPages(ctx context.Context, contentVersion string) ([]*ContentPage, error)
  • func (s *Site) Check(ctx context.Context, contentVersion string) (problems []string, err error)
  • func (s *Site) GetResources(dir, version string) (http.FileSystem, error)
  • func (s *Site) Handler() http.Handler
  • func (s *Site) RenderContentPage(page *PageData) ([]byte, error)
  • func (s *Site) ResolveContentPage(ctx context.Context, contentVersion, path string) (*ContentPage, error)
  • func (s *Site) Search(ctx context.Context, contentVersion string, queryStr string) (*search.Result, error)
• type VersionedFileSystem

Functions

func IsContentAsset

func IsContentAsset(urlPath string) bool

IsContentAsset reports whether the file in the site contents file system is a content asset (i.e., not a Markdown file). It typically matches .png, .gif, and .svg files.

func ReadFile

func ReadFile(fs http.FileSystem, path string) ([]byte, error)

func WalkFileSystem

func WalkFileSystem(fs http.FileSystem, filterFn func(path string) bool, walkFn func(path string) error) error

WalkFileSystem walks a file system and calls walkFn for each file that passes filterFn.

Types

type ContentPage

type ContentPage struct {
	Path        string            // the canonical URL path (without ".md" or "/index.md")
	FilePath    string            // the filename on disk
	Data        []byte            // the page's file contents
	Doc         markdown.Document // the Markdown doc
	Breadcrumbs []breadcrumbEntry // ancestor breadcrumb for this page
}

ContentPage represents a Markdown-formatted documentation page. To create a ContentPage, use one of the Site methods.

type PageData

type PageData struct {
	ContentVersion  string // content version string requested
	ContentPagePath string // content page path requested

	ContentVersionNotFoundError bool // whether the requested version was not found
	ContentPageNotFoundError    bool // whether the requested content page was not found

	// Content is the content page, when it is found.
	Content *ContentPage
}

PageData is the data available to the HTML template used to render a page.

type Site

type Site struct {
	// Content is the versioned file system containing the Markdown files and assets (e.g., images)
	// embedded in them.
	Content VersionedFileSystem

	// ContentExcludePattern is a regexp matching file paths to exclude in the content file system.
	ContentExcludePattern *regexp.Regexp

	// Base is the base URL (typically including only the path, such as "/" or "/help/") where the
	// site is available.
	Base *url.URL

	// Root is the root URL that is only used for specific cases where an absolute URL is mandatory,
	// such as for opengraph tags in the headers for example. It must include both the scheme and
	// host.
	Root *url.URL

	// AssetsBase is the base URL (sometimes only including the path, such as "/assets/") where the
	// assets are available.
	AssetsBase *url.URL

	// Redirects contains a mapping from URL path to redirect destination URL.
	Redirects map[string]*url.URL

	// CheckIgnoreURLPattern is a regexp matching URLs to ignore in the Check method.
	CheckIgnoreURLPattern *regexp.Regexp

	// SkipIndexURLPattern is a regexp matching URLs to ignore when searching. Any files that have a URL that match this
	// pattern will be ignored from the search index.
	SkipIndexURLPattern *regexp.Regexp
}

Site represents a documentation site, including all of its templates, assets, and content.

func (*Site) AllContentPages

func (s *Site) AllContentPages(ctx context.Context, contentVersion string) ([]*ContentPage, error)

AllContentPages returns a list of all content pages in the site.

func (*Site) Check

func (s *Site) Check(ctx context.Context, contentVersion string) (problems []string, err error)

Check checks the site content for common problems (such as broken links).

func (*Site) GetResources

func (s *Site) GetResources(dir, version string) (http.FileSystem, error)

func (*Site) Handler

func (s *Site) Handler() http.Handler

Handler returns an http.Handler that serves the site.

func (*Site) RenderContentPage

func (s *Site) RenderContentPage(page *PageData) ([]byte, error)

RenderContentPage renders a content page using the template.

func (*Site) ResolveContentPage

func (s *Site) ResolveContentPage(ctx context.Context, contentVersion, path string) (*ContentPage, error)

ResolveContentPage looks up the content page at the given version and path (which generally comes from a URL). The path may omit the ".md" file extension and the "/index" or "/index.md" suffix.

If the resulting ContentPage differs from the path argument, the caller should (if possible) communicate a redirect.

func (*Site) Search

func (s *Site) Search(ctx context.Context, contentVersion string, queryStr string) (*search.Result, error)

Search searches all documents at the version for a query.

type VersionedFileSystem

type VersionedFileSystem interface {
	OpenVersion(ctx context.Context, version string) (http.FileSystem, error)
}

VersionedFileSystem represents multiple versions of an http.FileSystem.