icarus

package module
v0.0.0-...-c4c1cbe Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 9, 2016 License: MIT Imports: 17 Imported by: 0

README

Icarus is a minimalistic blog platform for people who want to write Markdown or HTML documents in Git and have them served by a minimalistic server which still provides the basic nice-to-haves: trending content, full-text search and discovering similar content, while comfortably running on a small VPS.

It relies on quite a few opensource projects: Go, Redis, Bleve, Bootstrap 3, and many others.

Installation

First, you need to install some system dependencies, like Glide, and Redis. On OSX, that might look like:

brew install glide
brew install redis

Then you need to install further dependencies via Glide:

glide install

Then build or install the commands:

make build

or

make install

At which point the binaries will either be in cmd/icarus and cmd/icontent or will be in $GOPATH/bin/.

Configuring

Icarus is configured via a JSON configuration file, an example of which is available at config.json, and supporting these parameters:

{
  "server": {
    "loc": "127.0.0.1:8080",
    "proto": "http",
    "domain": "yourblog.com"
  },
  "rss": {
    "path": "/feeds/",
    "title": "Recent Pages"
  },
  "blog": {
    "name": "Your Blog",
    "results_per_page": 10,
    "pages_in_paginator": 10,
    "template_dir": "templates/",
    "static_dir": "static/"
  },
  "redis": {
    "loc": "localhost:6379"
  }
}

You'll certainly want to change server.domain and blog.name, and the easiest way to customize Icarus without changing the code is to supply your own templates in templates/ and your own static assets in static/ (you'll specify which files are used from static/ by customizing your templates).

By convention, you'll probably want to symlink your pages' static assets into static/ to allow the Go file server to serve your static assets, something along the lines of:

ln -s /Users/will/git/irrational_exuberance/static/ `pwd`/static/blog/

And you should be good to go.

Running

Once you've followed the Installation steps, you should be able to get things running via one of these options:

# if you did make install, recommended
$GOPATH/bin/icarus
# if you did make build
./cmd/icarus/icarus
# if you are making changes during development
make icarus

You can also pass in --config path/to/config.json if you're not running it from the icarus.git repository (or want to specify a different configuration file).

Adding Pages

Each article is either a Markdown or an HTML file (indicated via a trailing .md or .html respectively), but starting with a modified JSON blob:

"title": "This is my title",
"summary": "This is an exciting article about...",
"pub_date": 1184450111,
"slug": "a-unique-slug",
"tags": ["python", "programming"],
"draft": true,


Start writing your article here. Just make sure
you have an empty line after your JSON ends.
The trailing comma is optional.

The supported parameters are:

  • title is the human readable title for your page,
  • summary is the human readable description paragraph for a page,
  • pub_date is an optional timestamp for publishing date, defaults to time it is first sync'd,
  • slug is a unique URL component, such that /<slug>/ is the canonical URL for a page,
  • tags is a list of strings, for tags this page will be added to (tags are also used for calculating related/similar pages),
  • draft default to false and is optional, this governs if your page is included in analytics and the various article lists (e.g. a draft is only accessible if you type in its slug by hand, they are not discoverable).

From there you use the icontent tool to load the content:

$GOPATH/bin/icarus --config path/to/config.json blog/*.md
$GOPATH/bin/icarus --config path/to/config.json blog/*.html

And you're done.

If you want to unpublish a piece of content, the easiest solution right now is to mark "draft": true in the page's configuration and it will be removed from all indexes.

Conceivably we might want to add a iremove command at some point to truly remove content, or you could just use "draft": true to remove the indexes and then do redis-cli REM page.<post-slug> if you like living on the edge!

History

For reasons which are hard to explain, I've spent a lot of time over the past decade building mediocre blog platforms, and Icarus is the next in that glorious heritage.

Icarus is a Go reimagining of Sisyphus, which was my second personal blogging platform, inspired by the many, many mistakes I made in my first generation "Lifeflow" blog and also by ideas around real-time analytics and content suggestions/ranking that came from working at Digg.

For the third generation, looking to explore some additional ideas:

  1. Writing it in Go and avoiding a heavy-weight framework like Django.
  2. Try to make it actually usable by someone other than myself, mostly as an exercise and not because I anticipate much adoption.
  3. Keep using Bootstrap, maybe Bootstrap 4.
  4. Be faster than 60ms to load the front page. Haven't profiled this in a very long time, but I'd guess Redis lookups are responsible for most of that delay (~85 Redis lookups for the frontpage to load). So... let's try to hit ~5 ms, which probably means keeping everything in-memory (and also updating analytics out-of-band, which will be easy in Go). We can use Redis Pub-Sub to invalid the cache if we're running with more than one instance.
  5. Disqus comments are mostly just ads for me, and should either be removed entirely or replaced with something better / different.
  6. Move away from Python-only Whoosh for search.
  7. Use rrssb for better, simpler sharing to social sites.

Documentation

Overview

Configuration for Icarus instance.

Utilities for preprocessing Markdown from the Python style to the new, different style...

Specifically:

1. strip out [TOC] since there is no support here, 2. call out to pygments to render code blocks starting with :::<lang>

TODO: move this into its own module w/its own namespace

Handling templates and such.

Index

Constants

View Source 
const AnalyticsBackoff = "analytics.backoff.%v"
View Source 
const DaySeconds = 60 * 60 * 24
View Source 
const DirectReferrer = "DIRECT"
View Source 
const HistoricalReferrer = "imported from Google Analytics"
View Source 
const PageReferrers = "analytics.refer.%v"
View Source 
const PageString = "page.%v"
View Source 
const PageViewBonus = 60 * 60 * 24
View Source 
const PageViewBucket = "analytics.pv_bucket"
View Source 
const PageViewPageBucket = "analytics.pv_bucket.%v"
View Source 
const PageViews = "analytics.pv"
View Source 
const PageZsetByTime = "pages_by_time"
View Source 
const PageZsetByTrend = "pages_by_trend"
View Source 
const PagesInModules = 3

TODO: move this to Config

View Source 
const RateLimitPeriod = 60
View Source 
const Referrers = "analytics.refer"
View Source 
const SimilarPagesByTrend = "similar_pages.%v"
View Source 
const SimilarPagesExpire = 60 * 60 * 24
View Source 
const TagPagesZsetByTime = "tag_pages_by_time.%v"
View Source 
const TagPagesZsetByTrend = "tag_pages_by_trend.%v"
View Source 
const TagZsetByPages = "tags_by_pages"
View Source 
const TagZsetByTime = "tags_by_times"
View Source 
const UserAgents = "analytics.useragent"

Variables

View Source 
var BotAgents = []string{
	"-",
	"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)",
	"mozilla/5.0 (compatible; yahoo! slurp; http://help.yahoo.com/help/us/ysearch/slurp)",
	"disqus/1.0",
	"sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)",
	"baiduspider+(+http://www.baidu.com/search/spider.htm)",
	"baiduspider+(+http://www.baidu.jp/spider/)",
	"ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)",
	"java/1.6.0_24",
	"python-urllib/2.6",
	"ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)",
	"mozilla/5.0 (compatible; topblogsInfo/2.0; +topblogsinfo@gmail.com)",
	"mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
}
View Source 
var FilteredRefs = []string{
	"www.google.com",
	"lethain.com",
	"dev.lethain.com",
	"DIRECT",
	HistoricalReferrer,
}

Functions

func BuildAtomFeed 

func BuildAtomFeed(cfg *Config, ps []*Page) (*feeds.Feed, error)

func ConfigRedis 

func ConfigRedis(cfg *Config) error

func ConfigSearch 

func ConfigSearch(cfg *Config) error

func ConfigTemplates 

func ConfigTemplates(cfg *Config) error

func CurrentTimestamp 

func CurrentTimestamp() int64

func GetIP 

func GetIP(r *http.Request) string

func GetRedisClient 

func GetRedisClient() (*redis.Client, error)

func IndexPage 

func IndexPage(p *Page) error

func IndexPages 

func IndexPages(pgs []*Page) error

func IsRateLimited 

func IsRateLimited(key string) bool

func PagesInList 

func PagesInList(list string) (int, error)

func PutRedisClient 

func PutRedisClient(rc *redis.Client)

func Referrer 

func Referrer(r *http.Request) string

func RegisterPage 

func RegisterPage(p *Page) error

func RegisterPageTag 

func RegisterPageTag(p *Page, tag string) error

func ReindexAll 

func ReindexAll() error
func Search(qs string) ([]string, error)

func Serve 

func Serve(cfg *Config)

func ShouldIgnore 

func ShouldIgnore(p *Page, r *http.Request) bool

func SlugsForList 

func SlugsForList(list string, offset int, count int, reverse bool) ([]string, error)

func Track 

func Track(p *Page, r *http.Request) error

func UnindexPage 

func UnindexPage(p *Page) error

func UnregisterPage 

func UnregisterPage(p *Page) error

func UnregisterPageTag 

func UnregisterPageTag(p *Page, tag string) error

Types

type BlogConfig 

type BlogConfig struct {
	Name             string
	ResultsPerPage   int    `json:"results_per_page"`
	PagesInPaginator int    `json:"pages_in_paginator"`
	TemplateDir      string `json:"template_dir"`
	StaticDir        string `json:"static_dir"`
}

.BlogName -> .Blog.Name .ListCount -> .Blog.ResultsPerPage .NumPages -> .Blog.PagesInPaginator

type Config 

type Config struct {
	Server ServerConfig
	RSS    RSSConfig
	Blog   BlogConfig
	Redis  RedisConfig
}

func NewConfigFromFile 

func NewConfigFromFile(path string) (*Config, error)

Build a new configuration file from disk.

func (*Config) BaseURL 

func (cfg *Config) BaseURL() string

type NoSuchPagesError 

type NoSuchPagesError struct {
	Slugs []string
	// contains filtered or unexported fields
}

func (*NoSuchPagesError) Error 

func (e *NoSuchPagesError) Error() string

type Page 

type Page struct {
	Slug        string   `json:"slug"`
	Tags        []string `json:"tags"`
	Title       string   `json:"title"`
	Summary     string   `json:"summary"`
	Content     string   `json:"html"`
	Draft       bool     `json:"draft"`
	PubDateStr  int64    `json:"pub_date"`
	EditDateStr int64    `json:"edit_date"`
}

func PageFromRedis 

func PageFromRedis(slug string) (*Page, error)

Retrieve one page from Redis.

func PagesForList 

func PagesForList(list string, offset int, count int, reverse bool) ([]*Page, error)

func PagesFromRedis 

func PagesFromRedis(slugs []string) ([]*Page, error)

Retrieve a list of slugs from Redis.

func ReadHeaders 

func ReadHeaders(content string) (*Page, string, error)

Create a new Page with the headers stripped out of it.

Headers look like this:

"tags": ["python", "redis"], "title": "Storing Bounded Timeboxes in Redis", "summary": "This is the summary...", "slug": "storing-bounded-timeboxes-in-redis",

Including a \n\n (outside of string) to designate the end of the section. In general, you can describe the header as a JSON dictionary without the opening or closing {}s.

func RecentPages 

func RecentPages(offset int, count int) ([]*Page, error)

func Render 

func Render(filename string, content string) (*Page, error)

func RenderHTML 

func RenderHTML(content string) (*Page, error)

func RenderMarkdown 

func RenderMarkdown(content string) (*Page, error)

func SimilarPages 

func SimilarPages(p *Page, offset int, count int) ([]*Page, error)

func Surrounding 

func Surrounding(p *Page, num int, reverse bool) ([]*Page, error)

Get up to N preceeding or following pages.

func TrendingPages 

func TrendingPages(offset int, count int) ([]*Page, error)

func (*Page) EditDate 

func (p *Page) EditDate() time.Time

func (*Page) EnsureEditDate 

func (p *Page) EnsureEditDate()

func (*Page) EnsurePubDate 

func (p *Page) EnsurePubDate()

func (*Page) InitEditDate 

func (p *Page) InitEditDate()

func (*Page) InitPubDate 

func (p *Page) InitPubDate()

func (*Page) Key 

func (p *Page) Key() string

Generate the Redis key for this page.

func (*Page) PubDate 

func (p *Page) PubDate() time.Time

func (*Page) Sync 

func (p *Page) Sync() error

Synchronize this page to Redis.

type PageOpt 

type PageOpt struct {
	Num      int
	Offset   int
	Selected bool
}

type Paginator 

type Paginator struct {
	Pages      []PageOpt
	NextOffset int
	PrevOffset int
	HasNext    bool
	HasPrev    bool
	Show       bool
}

func NewPaginator 

func NewPaginator(offset int, total int, pageSize int, numPages int) Paginator

Create a new Paginator.

offset is the current offset (in items). total is the total number of items. pageSize is the number of items per page. numPages is the number of pages to display

type RSSConfig 

type RSSConfig struct {
	Path  string
	Title string
}

type RedisConfig 

type RedisConfig struct {
	Loc      string
	Proto    string
	PoolSize int
}

type ServerConfig 

type ServerConfig struct {
	Loc    string
	Proto  string
	Domain string
}

replacing: 

NetLoc      string `json:"netloc"`
DomainUrl   string `json:"domain"`

type Tag 

type Tag struct {
	Slug  string
	Count int
}

func GetAllTags 

func GetAllTags() ([]Tag, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
icarus
icarus-index
icontent

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.