tiddlybucket

README

tiddlybucket

This is my attempt at solving my specific use case where I want to run my TiddlyWiki5 instances with the tiddlers persisted in a Google Cloud or AWS bucket.

I like having each tiddler as a separate file, in the way that the TiddlyWiki5 server supports, so that I can create and edit from a terminal. Unfortunately, mounting a bucket to the filesystem incurred a large performance hit and scheduled synchronization tasks lead to race conditions and unintended overwrites.

This tool replicates the TiddlyWeb backend API and can read and write the tiddler files to a local directory like the canonical TiddlyWiki5 app. But, in addition, it can do the same with a given a Google Cloud Storage bucket or AWS S3 bucket.

grab a copy

already built binaries

how to install
go install git.sr.ht/~hokiegeek/tiddlybucket/cmd/tiddlybucket@latest
docker pull registry.gitlab.com/hokiegeek/tiddlybucket

^{note: a "latest" image is not published for the container images}

build it yourself

git clone https://git.sr.ht/~hokiegeek/tiddlybucket
cd tiddlybucket
make index
go install ./...

make index will clone a copy of TiddlyWiki5 and builds an index.html with with a custom tiddlywiki.info.

how to run

environment variables

variable	store type	notes
WIKI_LOCATION	all
PORT	n/a
DEBUG_LEVEL	n/a
GOOGLE_APPLICATION_CREDENTIALS	gs	required if using a Google Cloud Storage bucket
AWS_ACCESS_KEY_ID	s3	required if using an AWS S3 bucket
AWS_SECRET_ACCESS_KEY	s3	required if using an AWS S3 bucket
AWS_REGION	s3	required if using an AWS S3 bucket

command-line options

command-line options override their respective environment variables.

flag	option	default
-port	NUM	8080
-debug_level	trace|debug|info|warn|error|fatal	info
-credentials_file	PATH	none
-readers	comma-separated list of usernames	none
-writers	comma-separated list of usernames	none

positional arguments

Only one positional argument is expected. All others are ignored. The first positional argument is interpreted as being the location of the wiki files. A full URI is required (see below). The type of storage is determined from the URI scheme. The structure of the location is the same as with the TiddlyWiki5 server where it includes a directory named "tiddlers".

[gs|s3|file]://[WIKI_PATH]

This overrides the WIKI_LOCATION variable.

⚠️ This or WIKI_LOCATION must be provided or the server will panic upon startup.

examples

Google Cloud Storage bucket with access restrictions

env GOOGLE_APPLICATION_CREDENTIALS=/etc/secrets/ghammond.json \
    tiddlybucket \
    --credentials_file=personnel.csv --readers='(anon)' --writers='(authenticated)' \
    gs://sgc/sg-1/misson-logs

AWS S3 bucket with minimal logging

env AWS_ACCESS_KEY_ID=xxxxx \
    AWS_SECRET_ACCESS_KEY=uuuuu \
    AWS_REGION=us-west-1 \
    tiddlybucket --port=9876 --debug_level=warn \
    s3://sgc/known-races/furlings

local filesystem

tiddlybucket file:///Users/joneill/fishing

deployment suggestions

Google Cloud

Take a look at the terraform I am using for my main wiki. It runs tiddlybucket as a Google Cloud Run (kubernetes) service: https://git.sr.ht/~hokiegeek/wiki/tree/main/item/infrastructure/tf-cloudrun/main.tf

AWS

TODO

TiddlyWiki5 server feature parity

My goal with this project is to duplicate the functionality and features of the TiddlyWiki5 server that I use with my wiki. As such, not everything is supported.

feature	progress	notes
reading tiddler files from local disk	✅
writing tiddler files to local disk	✅
reading and writing meta tiddlers	✅	might need to revisit how text vs binary files are identified
TiddlyWeb API support	partial	the skinny list endpoint ignores the filter query
preloading tiddlers in store on HTML file	✅
serving $:/favicon.ico	✅
raw markup support	partial	wikified markup is not currently supported (see TODO #1)
basic auth	✅	the endpoint /login-basic behaves as the TW5 server
credentials file with readers and writers	✅
header credentials

contributing

A patchset is always welcomed!

If you want to submit a bug report or enhancement, the issue tracker is: https://todo.sr.ht/~hokiegeek/tiddlybucket

dependencies

There isn't much in the way of direct, third-party, dependencies. The tabel below lists the highlights.

package	use
github.com/spf13/viper	configuration management (env vars and command-line)
github.com/go-chi/chi/v5	a nicely idiomatic and minimal router
github.com/rs/zerolog	structured logging
cloud.google.com/go/storage	API to directly access a Google Cloud Storage bucket
github.com/aws/aws-sdk-go	API to directly access an AWS S3 bucket

vulnerability scanning

dependencies are scanned for vulnerabilities using Nancy on every CI build.

licensing

The index.html file that is generated upon build and distributed in the container releases is copyrighted to UnaMesa Association. The full license can be seen here: https://tiddlywiki.com/#License

All of the code within this project - aside from the generated index.html - is licensed under an MIT license. The full license can be seen here: https://git.sr.ht/~hokiegeek/tiddlybucket/tree/main/item/LICENSE

Documentation

Index

• Constants
• func ListenAndServe(addr string, insecureCreds Credentials, store TiddlerStore) error
• type Credentials
• type Tiddler
  • func (t *Tiddler) Bytes() []byte
  • func (t *Tiddler) Field(name string) string
  • func (t *Tiddler) Read(r io.Reader) error
• type TiddlerFile
  • func (t *TiddlerFile) Read(r io.Reader) error
  • func (t *TiddlerFile) Tiddler() Tiddler
  • func (t *TiddlerFile) Write(w io.Writer) error
• type TiddlerStore
  • func NewAwsS3Store(uri string) (TiddlerStore, error)
  • func NewFileStore(dir string) (TiddlerStore, error)
  • func NewGoogleBucketStore(uri string) (TiddlerStore, error)

Constants

const (
	AuthAnonUsername = "GUEST" // https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js#L91
)

Functions

func ListenAndServe

func ListenAndServe(addr string, insecureCreds Credentials, store TiddlerStore) error

Types

type Credentials

type Credentials struct {
	UserPasswordsClearText map[string]string
	Readers                []string
	Writers                []string
}

type Tiddler

type Tiddler map[string]interface{}

func (*Tiddler) Bytes

func (t *Tiddler) Bytes() []byte

func (*Tiddler) Field

func (t *Tiddler) Field(name string) string

func (*Tiddler) Read

func (t *Tiddler) Read(r io.Reader) error

type TiddlerFile

type TiddlerFile struct {
	// contains filtered or unexported fields
}

func (*TiddlerFile) Read

func (t *TiddlerFile) Read(r io.Reader) error

TODO?: https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js#L271

func (*TiddlerFile) Tiddler

func (t *TiddlerFile) Tiddler() Tiddler

func (*TiddlerFile) Write

func (t *TiddlerFile) Write(w io.Writer) error

type TiddlerStore

type TiddlerStore interface {
	ReadFile(path string) (io.ReadCloser, error)
	GetTiddler(title string) (Tiddler, error)
	GetAllTiddlers() ([]Tiddler, error)
	WriteTiddler(t Tiddler) error
	DeleteTiddler(title string) error
}

func NewAwsS3Store

func NewAwsS3Store(uri string) (TiddlerStore, error)

func NewFileStore

func NewFileStore(dir string) (TiddlerStore, error)

func NewGoogleBucketStore

func NewGoogleBucketStore(uri string) (TiddlerStore, error)