midas

README

Midas

What am I?

Midas is a web application that allows to generate static websites using SSGs using data from headless CMS.

How does it work?

Midas is listening for the webhooks from the "provider" - CMS - and then, depending on the payload, it modifies the site (e.g. adds a new post) and regenerates it (builds).

Features

• Listening to the changes from the headless CMS using the endpoint for the webhooks.
• Sanitizing and generating new data (post/page) for the site.
• Generating the static site using one of the supported SSGs.
• Deployment of the ready website to one of the cloud providers.

Supported technologies

Providers (CMS)

• Strapi

Receivers (Static site generators)

• Hugo - full support with creating/updating posts/pages.
• Astro - partial (build) support. Data fetching has to be done on the Astro end.

Deployment targets

• AWS
  • Files upload to AWS S3.
  • CloudFront distribution invalidation.
• SFTP server

Provider-receiver support matrix

	Strapi
Hugo	✔
Astro	✔

Installation

Pre-built binaries

You can find downloads in the releases section on GitHub.

Install using go

You may want to install the app using go itself. To do that, type following command:

go install github.com/kovansky/midas/cmd/midasd@latest

Usage

Midas configuration

You need to create a configuration file (can be in your home directory). Config file path needs to be provided while starting midasd webserver.

Sample configuration file

{
  // This is a address that the app will be listening on
  "addr": "127.0.0.1:8445",
  // You can paste the Rollbar token to receive internal errors reported there (https://rollbar.com/)
  "rollbarToken": "",
  // This is probably most important part of the config - here you specify where your static site code is
  "sites": {
    // We start with an API key, a.k.a. identifier of the site. In future the codes will be held in some database, not there
    "abcd-efgh-ijkl": {
      // Name of the site. May be passed to generator.
      "siteName": "Sample site",
      // Very important setting, specifies which SSG (receiver) is used. Required. Currently, hugo fully supported and astro just for build process.
      "service": "hugo",
      // Where the site code lives. Should be absolute path. Required.
      "rootDir": "/home/kitten/hugo-site",
      // You can enable to build a site with draft posts along with the main site (in the separate dir). Default: false.
      "buildDrafts": false,
      // If you enable the option above ^, here you need to pass the URL at which the site will be available, so the generator can build URLs properly.
      "draftsUrl": "http://preview.hugo.local",
      // Here you can set where the static site will be generated (can be absolute or relative - then will be placed under rootDir).
      "outputSettings": {
        // Main site will be generated to this directory. Default: public
        "build": "public",
        // Site with drafts will be generated to this directory. Default: publicDrafts
        "draft": "publicDrafts",
        // The environment that should be passed to the generator. Default: development
        "draftEnvironment": "development"
      },
      // You can specify deployment configuration to upload built site to the cloud.
      "deployment": {
        // Self-explainatory. If the deployment is enabled.
        "enabled": true,
        // Name of the provider to use. Possible: aws, sftp. Required.
        "target": "aws",
        // AWS-specific settings.
        "aws": {
          // Name of the bucket to use for upload.
          "bucketName": "hugo-test",
          // AWS Access and secret keys.
          "accessKey": "AWSSAMPLEACCESSKEY",
          "secretKey": "AWSSAMPLESECRETKEY",
          // AWS S3 bucket region.
          "region": "eu-central-1",
          // If provided, all files in the distribution will be invalidated after deployment.
          "cloudfrontDistribution": "E3SABCD1234",
        },
        // SFTP-specific settings.
        "sftp": {
          // Server address. Required.
          "host": "1.2.3.4",
          // SSH/SFTP server port. Default: 22.
          "port": 22,
          // Authentication method. Possible: none, password, key.
          "method": "password",
          // Username.
          "user": "me",
          // Password
          "password": "secret",
          // Private key file. Required if key method is used.
          "key": "/home/kitten/id_rsa",
          // Passphrase of the key file. Optional.
          "keyPassphrase": "super_secret_wow",
          // Path to the directory on the server. Required.
          "path": "/home/kitten/mysite/",
        }
      },
      // Same as the deployment above, using same config structure, but for drafts.
      "draftsDeployment": {},
      // Required. Midas keeps an id->filename mapping for created entries.
      "registry": {
        // Currently jsonfile storage is supported, as well as "none" to not keep registry at all.
        "type": "jsonfile",
        // Provide json filename where the mapping should be saved. Can be absolute or relative - then will be placed under site's rootDir 
        "location": "./midas-registry.json"
      },
      // List incoming types that should be treated as collections (multiple entries per type).
      "collectionTypes": {
        // ...here we are allowing "post" type as a collection type, because we will have many posts
        "post": {
          // We can choose the archetype used to generate content for this type.
          "archetypePath": "archetypes/default.md",
          // And specify the directory to which the entries will be saved.
          "outputDir": "content/posts/"
        }
      },
      // Same as above, but with single types (so type=one entry).
      "singleTypes": {
        "homepage": {
          // For single types a JSON file with entry data will be generated in the outputDir (named %typename%.json, i.e. homepage.json) with values passed through the HTML sanitizer.
          "outputDir": "data/cms/"
        }
      }
    }
    // Note, that types not listed in collectionTypes nor singleTypes will be ignored.
  }
}

Start midas

After creating the configuration file you need to start the Midas. The main command is midasd. It takes two arguments (optional):

• config - path to your config file. Default: config.json (in current directory).
• env - development or production. Used in rollbar logging. Default: production. Can also be set using MIDAS_ENV environmental variable.

So sample startup command could be:

midasd --config ~/midas.json --env development

CMS configuration

Strapi

For Strapi to send webhooks to Midas you need to head to the Settings in Strapi, then to Webhooks, and click " Create new webhook" (or just visit https://your-strapi-installation.com/admin/settings/webhooks/create) .

You can choose whatever name you want. In the URL you need to pass a Midas URL for the connection. It will have the following form: https://midas-installation.com/{{provider}}/{{receiver}}. For example if you are connecting Strapi and Hugo, the URL will be https://midas-installation.com/strapi/hugo.

Now go to the Headers and add the Authorization header with value Bearer {{insert API key}}, i.e. Bearer abcd-efgh-ijkl.

In the Events part it is recommended to select all checkboxes for Entry. Media is currently not supported.

Whole Strapi settings should look like this:

Creating archetypes

When creating archetypes for entries, you can use data from the Payload sent by the Provider. Most of the information will be stored in two maps: one named Entry (with entry data, like values of the fields from CMS), and second named Metadata with some generated data, like information if entry is published. Sample archetype for Strapi->Hugo relation may look like this:

---
title: "{{ index .Entry "Title" }}" # We read the title from one of the fields configured in CMS
date: {{ index .Metadata "createdAt" }} # We read the publication date from metadata
draft: {{ not (index .Metadata "published") }} # As well as the information if the post is published or not.
---

<h3>{{ index .Entry "Subtitle" }}</h3> <!-- We may for example include some subtitle -->
<div id="entry-content">
    {{ index .Entry "Content" }} <!-- And after all we include the main entry content -->
</div>

Feature requests? Bugs?

You are welcome to open an issue.

Contributing?

You are welcome to write new features, providers, receivers or deployment targets :)

License

Project is released under GNU GPLv3. For more information, see LICENSE file.

Author

Created and maintained by F4 Developer (Stanisław Kowański)

Documentation

Index

• Constants
• Variables
• func ApiKeyFromContext(ctx context.Context) string
• func CreateSlug(title string) string
• func ErrorCode(err error) string
• func ErrorMessage(err error) string
• func NewContextWithApiKey(ctx context.Context, key string) context.Context
• func NewContextWithSiteConfig(ctx context.Context, site *Site) context.Context
• type AWSDeploymentSettigs
• type Concurrent
• type ConcurrentList
• type Config
• type Deployment
• type DeploymentSettings
• type Error
  • func Errorf(code string, format string, args ...interface{}) *Error
  • func (e *Error) Error() string
• type ModelSettings
• type OutputSettings
• type Payload
• type Registry
• type RegistryService
• type RegistrySettings
• type SFTPDeploymentSettings
• type SanitizerService
• type Site
  • func SiteConfigFromContext(ctx context.Context) *Site
• type SiteService
• type StrapiWebhookEvents
  • func (event StrapiWebhookEvents) MarshalJSON() ([]byte, error)
  • func (event StrapiWebhookEvents) String() string
  • func (event *StrapiWebhookEvents) UnmarshalJSON(bytes []byte) error

Constants

const (
	ErrUnauthorized    = "unauthorized"
	ErrInternal        = "internal"
	ErrInvalid         = "invalid"
	ErrUnaccepted      = "unaccepted"
	ErrRegistry        = "registry"
	ErrSiteConfig      = "site config"
	ErrProcessNotFound = "process not found"
	ErrCancelled       = "process cancelled"
)

Variables

var (
	Commit  string
	Version string

	RegistryServices  map[string]func(site Site) RegistryService
	DeploymentTargets map[string]func(site Site, settings DeploymentSettings, isDraft bool) (Deployment, error)
	Sanitizer         SanitizerService
	Concurrents       ConcurrentList
)

var ReportError = func(ctx context.Context, err error, args ...interface{}) {
	log.Printf("error: %+v\n", err)
}

ReportError is used to notify external services of error.

Functions

func ApiKeyFromContext

func ApiKeyFromContext(ctx context.Context) string

ApiKeyFromContext returns current API key from context.

func CreateSlug

func CreateSlug(title string) string

CreateSlug generates an url-safe string from title (or any other string) to be used as post/page slug.

func ErrorCode

func ErrorCode(err error) string

ErrorCode unwraps an application error and returns its code. Non-application errors always return ErrInternal.

func ErrorMessage

func ErrorMessage(err error) string

ErrorMessage unwraps an application error and returns its message. Non-application errors always return "Internal error"

func NewContextWithApiKey

func NewContextWithApiKey(ctx context.Context, key string) context.Context

NewContextWithApiKey returns a new context with given API key.

func NewContextWithSiteConfig

func NewContextWithSiteConfig(ctx context.Context, site *Site) context.Context

NewContextWithSiteConfig returns a new context with given site config.

Types

type AWSDeploymentSettigs

type AWSDeploymentSettigs struct {
	BucketName             string `json:"bucketName"`
	Region                 string `json:"region"`
	AccessKey              string `json:"accessKey"`
	SecretKey              string `json:"secretKey"`
	S3Prefix               string `json:"s3Prefix"`
	CloudfrontDistribution string `json:"cloudfrontDistribution,omitempty"`
}

type Concurrent

type Concurrent interface {
	Stop()
	Site() Site
}

type ConcurrentList

type ConcurrentList interface {
	Add(concurrent Concurrent) error
	Has(name string) bool
	SafelyRemove(name string) error
	Remove(name string)
	Get(name string) (*Concurrent, error)
}

type Config

type Config struct {
	Domain       string          `json:"domain"`
	Addr         string          `json:"addr"`
	RollbarToken string          `json:"rollbarToken"`
	Sites        map[string]Site `json:"sites"` // [api key] => site
}

type Deployment

type Deployment interface {
	Deploy() error
}

type DeploymentSettings

type DeploymentSettings struct {
	Enabled bool                   `json:"enabled,default=false"`
	Target  string                 `json:"target"` // Can be: AWS, SFTP
	AWS     AWSDeploymentSettigs   `json:"aws,omitempty"`
	SFTP    SFTPDeploymentSettings `json:"sftp,omitempty"`
}

type Error

type Error struct {
	// Machine-readable error code
	Code string

	// Human-readable error message
	Message string
}

Error represents an application-specific error. App errors can be unwrapped to extract out the code & message.

Any non-application errors (such as a disk error) should be reported as en ErrInternal error, so the end-user should only see "internal error". These error details should be logged to the operator of application.

func Errorf

func Errorf(code string, format string, args ...interface{}) *Error

Errorf is a helper function to return an Error with a given code and formatted message.

func (*Error) Error

func (e *Error) Error() string

Error implements the error interface.

type ModelSettings

type ModelSettings struct {
	ArchetypePath string `json:"archetypePath,omitempty"`
	OutputDir     string `json:"outputDir,omitempty"`
	Fields        struct {
		Title *string   `json:"title,omitempty"`
		HTML  *[]string `json:"html,omitempty"`
	} `json:"fields"`
}

type OutputSettings

type OutputSettings struct {
	Build            string `json:"build,omitempty"`
	Draft            string `json:"draft,omitempty"`
	DraftEnvironment string `json:"draftEnvironment,omitempty"`
}

type Payload

type Payload interface {
	json.Unmarshaler
	json.Marshaler
	Event() string
	Metadata() map[string]interface{}
	Entry() map[string]interface{}
	SetEntry(map[string]interface{})
	Raw() interface{}
}

type Registry

type Registry map[string]string

Registry type is used to hold data from registries. It's structure is Id => Filename. So from this JSON:

{
  "1": "sample-post.html"
}

"1" would be a key and "sample-post.html" would be a value.

type RegistryService

type RegistryService interface {
	OpenStorage() error
	CloseStorage()
	CreateStorage() error
	RemoveStorage() error
	Flush() error
	CreateEntry(id, filename string) error
	ReadEntry(id string) (string, error)
	UpdateEntry(id, newFilename string) error
	DeleteEntry(id string) error
}

type RegistrySettings

type RegistrySettings struct {
	Type     string `json:"type"`
	Location string `json:"location"`
}

type SFTPDeploymentSettings

type SFTPDeploymentSettings struct {
	Host          string `json:"host"`
	Port          *int   `json:"port"`
	User          string `json:"user"`
	Method        string `json:"method"`
	Password      string `json:"password,omitempty"`
	Key           string `json:"key,omitempty"`
	KeyPassphrase string `json:"keyPassphrase,omitempty"`
	Path          string `json:"path"`
}

type SanitizerService

type SanitizerService interface {
	Sanitize(html string) string
}

type Site

type Site struct {
	SiteName string `json:"siteName"`
	Service  string `json:"service"`

	RootDir        string         `json:"rootDir"`
	OutputSettings OutputSettings `json:"outputSettings"`

	BuildDrafts bool   `json:"buildDrafts,default=false"`
	DraftsUrl   string `json:"draftsUrl"`

	Registry        RegistrySettings         `json:"registry"`
	CollectionTypes map[string]ModelSettings `json:"collectionTypes"`
	SingleTypes     map[string]ModelSettings `json:"singleTypes"`

	Deployment       DeploymentSettings `json:"deployment"`
	DraftsDeployment DeploymentSettings `json:"draftsDeployment"`
}

func SiteConfigFromContext

func SiteConfigFromContext(ctx context.Context) *Site

SiteConfigFromContext returns current config from context.

type SiteService

type SiteService interface {
	GetRegistryService() (RegistryService, error)
	BuildSite(useCache bool, log zerolog.Logger) error
	CreateEntry(payload Payload) (string, error)
	UpdateEntry(payload Payload) (string, error)
	DeleteEntry(payload Payload) (string, error)
	UpdateSingle(payload Payload) (string, error)
}

type StrapiWebhookEvents

type StrapiWebhookEvents int64

const (
	Undefined StrapiWebhookEvents = iota
	Create
	Update
	Delete
	Publish
	Unpublish
)

func (StrapiWebhookEvents) MarshalJSON

func (event StrapiWebhookEvents) MarshalJSON() ([]byte, error)

func (StrapiWebhookEvents) String

func (event StrapiWebhookEvents) String() string

func (*StrapiWebhookEvents) UnmarshalJSON

func (event *StrapiWebhookEvents) UnmarshalJSON(bytes []byte) error