supersense

README

Supersense

Supersense is a simple and customizable event board. The main goal of supersense is given an open source alternative to the social walls or multiple source event trackers. At the moment, supersense implements only two sources: Twitter & Github, but the architecture was development in a scalable and extendable way.

Technology

Supersense was created with a micro-service paradigm in mind, is a stateless service easy to horizontal scale (but also can have persistence and to be an stateful service). The project has two principal services called: Core and Observer, each one is distributed using docker images and published in the DockerHub with the tags minskylab/supersense and minskylab/supersense-observer respectively.

About the stack

Core

Supersense Core was written in Go and uses GraphQL as API layer to communicate with the observer or any other custom API consumer, the main goal of core is provide a powerful and realtime API to broadcast all the events coming from the configurase sources (Twitter, Github, etc...).

Observer

Supersense Observer was build with Typescript and React, and all the source are bundled* using snowpack a interesting modern technology that use a modern feature called ESM (ES Modules) to distribute files in a unbundled mode, hence we have very fast development flow.

* Snowpack offers two ways to use it, with its unbundled proposal using ESM and another where you can use the classical bundlers (like Webpack, Parcel).

Architecture

Getting Started

Write a .env config file like:

# .env file
SS_GRAPHQL_PLAYGROUND=true

SS_TWITTER_CONSUMER_KEY=<YOUR_TWITTER_CONSUMER_KEY>
SS_TWITTER_CONSUMER_SECRET=<YOUR_TWITTER_CONSUMER_SECRET_KEY>
SS_TWITTER_ACCESS_TOKEN=<YOUR_TWITTER_ACCESS_TOKEN_KEY>
SS_TWITTER_ACCESS_SECRET=<YOUR_TWITTER_ACCESS_SECRET_KEY>
SS_TWITTER_QUERY="#peru"

SS_GITHUB_TOKEN=<GITHUB_TOKEN>
SS_GITHUB_REPOS=minskylab/supersense,minskylab/figport,minskylab/base

Supersense core is delivery using docker hub, to run only execute:

$ docker run -p 8080:8080 --env-file .env minskylab/supersense

Observer:

$ TODO

// TODO

// TODO

Configuration

Currently supersense only accepts one way to config the service parameters: the environment variables. Below you can see all of this variables.

Environment Variable Name	Default Value	Description
SS_PORT	4000
SS_TWITTER_CONSUMER_KEY	-
SS_TWITTER_CONSUMER_SECRET	-
SS_TWITTER_ACCESS_TOKEN	-
SS_TWITTER_ACCESS_SECRET	-
SS_TWITTER_QUERY	-
SS_GITHUB_TOKEN	-
SS_GITHUB_REPOS	-
SS_DUMMY_PERIOD	-
SS_DUMMY_MESSAGE	-
SS_SPOKESMAN	false
SS_SPOKESMAN_NAME	-
SS_SPOKESMAN_USERNAME	-
SS_SPOKESMAN_EMAIL	-
SS_PERSISTENCE	false
SS_PERSISTENCE_BOLTDB_FILEPATH	-
SS_PERSISTENCE_REDIS_ADDRESS	-
SS_PERSISTENCE_REDIS_PASSWORD	-
SS_GRAPHQL_PLAYGROUND	false

// TODO

• More easy observer configuration
• Twitter Images
• Github project summary
• Github open issues (extends the event demux)
• Deployment Instructions

Deploy with Helm

Before to deploy in your k8s cluster you need declare a secret with your configuration. You need to define a .env file with your configuration and use the following command to create a kubernetes generic secret.

$ kubectl create secret generic <your-secret-name> --from-env-file=.env

Then, now add the minsky charts repository and install supersense setting your secretRef.

$ helm repo add minsky https://charts.minsky.cc
$ helm repo update # optional
$ helm install minsky/supersense --set config.secretRef=<your-secret-name>

Documentation

Index

• type Entities
• type Event
• type ID
• type MediaEntity
• type Mux
  • func NewMux(sources ...Source) (*Mux, error)
  • func (m *Mux) AddNewSource(s Source)
  • func (m *Mux) Register(pipeline chan *Event, done <-chan struct{}, filteredSources ...string)
  • func (m *Mux) RunAllSources() error
• type Person
• type Source
• type URLEntity

Types

type Entities

type Entities struct {
	Tags  []string      `json:"tags"`
	Media []MediaEntity `json:"media"`
	Urls  []URLEntity   `json:"urls"`
}

Entities saves three types of entities

type Event

type Event struct {
	ID         string    `json:"id"`
	Title      string    `json:"title"`
	Entities   Entities  `json:"entities"`
	ShareURL   string    `json:"shareURL"`
	CreatedAt  time.Time `json:"createdAt"`
	EmittedAt  time.Time `json:"emittedAt"`
	Message    string    `json:"message"`
	Actor      Person    `json:"actor"`
	SourceID   string    `json:"sourceId"`
	SourceName string    `json:"sourceName"`
	EventKind  string    `json:"eventKind"`
	// New
	Labels []string `json:"labels"`
}

Event describes a simple event from a source

type ID

type ID string

ID is a string because we're use UUIDv4 as ID

type MediaEntity

type MediaEntity struct {
	URL  string `json:"url"`
	Type string `json:"type"`
}

MediaEntity represents a media in the event (e.g. Photos, Videos, Files)

type Mux

type Mux struct {
	// contains filtered or unexported fields
}

Mux is a necessary struct to join different sources

func NewMux

func NewMux(sources ...Source) (*Mux, error)

NewMux returns a new mux to use as a mani pipeline for all your event sources

func (*Mux) AddNewSource

func (m *Mux) AddNewSource(s Source)

AddNewSource exports this function to public

func (*Mux) Register

func (m *Mux) Register(pipeline chan *Event, done <-chan struct{}, filteredSources ...string)

Register attach a new channel to the pipes list.

func (*Mux) RunAllSources

func (m *Mux) RunAllSources() error

RunAllSources run all the sources at the same time

type Person

type Person struct {
	Name       string  `json:"name"`
	Photo      string  `json:"photo"`
	Owner      string  `json:"owner"`
	Email      *string `json:"email"`
	ProfileURL *string `json:"profileURL"`
	Username   *string `json:"username"`
}

Person is a struct to identify a person who post any on supersense

type Source

type Source interface {
	Run() error
	Identify(string) bool // That's only a temporal way to identify a source
	Dispose()
	Pipeline() <-chan Event
}

Source is a new Event emitter

type URLEntity

type URLEntity struct {
	URL        string `json:"url"`
	DisplayURL string `json:"display_url"`
}

URLEntity wrap a simple URL from the event message