frogpond

README

Frogpond

Frogpond is a simple, INSECURE peer-to-peer key value store and host discovery service. I use it on all my home computers to find each other and share configuration and a small amount of data.

Frogpond is a library that can be used in other applications, and there is also a stand-alone server and client that can connect to the embedded library or work entirely on their own.

There is a simple demonstration of the library used in the "trayheadless" program, which keeps my Raspberry Pis and other small computers in contact with my desktop.

Insecure

Frogpond is not secure. It is not encrypted, and it does not authenticate. It is not intended for use on the Internet. It is intended for use on a local network, where the only people who can see the traffic are people you trust.

Do not, under any circumstances, use Frogpond on the Internet.

Installation

export GO111MODULE=auto
go get github.com/donomii/racketprogs/frogpond 
go install github.com/donomii/racketprogs/frogpond 

Use

Frogpond can be used as a library, or as a stand-alone server and client.

For standalone, run the server one one machine with

./server --name "Server 1"

and on another

./server --peer xxx.xxx.xxx.xxx --name "Server 2"

where xxx.xxx.xxx.xxx is the IP address of the first server. You can also use the hostname of the first server.

Then run the client on either machine

./client add testkey testvalue
./client get testkey

move to the second machine and run

./client get testkey

and you should see the value.

If you don't see your value on the second machine, try running

./client ips

to list the known peers.

The complete list of commands for the client:

ips               - Print the ips of known frogpond servers
dump              - Dump all data from the frogpond data pool
add [key] [value] - Add a key/svalue pair to the frogpond data pool
delete [key]      - Delete a key/value from the frogpond data pool
get [key]         - Get a value from the frogpond data pool

Library Use

The best example is the frogpond stand alone server. Mainly you have to run StartServer() and then UpdatePeers() and UpdatePeersData() periodically.

Security

Frogpond communicates over HTTP, and does not authenticate or encrypt. It is not secure. It is intended for use on a local network, where the only people who can see the traffic are people you trust.

Do not use Frogpond on the Internet.

Design

Frogpond communicates with other Frogpond servers using HTTP. It uses a simple REST API to add, delete, and get key/value pairs. It also uses a simple REST API to get the IP addresses of other Frogpond servers.

The APIs for P2P and data transfer are separate. This allows Frogpond to be used as a simple host discovery service, and/or as a simple key/value store.

All values are kept in memory, there is no persistence.

Every time you modify the data pool, Frogpond updates its peers. This means that if you add a key/value pair, it will be available to all peers within a few seconds.

Each Frogpond server regularly checks in with its peers, by default every 50s. This allows updates to move through NATs and firewalls, but with a larger delay than usual.

Documentation

Index

• Variables
• func AddHost(h *HostService)
• func AddHosts(hs []*HostService)
• func AnnounceAll()
• func AnnounceSelf(ip, port string)
• func AppendHosts(hs []*HostService)
• func ArpScan()
• func CidrHosts(cidr string) ([]string, error)
• func FormatHttpIpPort(ip, port string) string
• func FormatHttpIpPort_i(ip string, port uint) string
• func FuckIpv6(ip string) string
• func Hosts2Json() []byte
• func ScanAllHostsPublicInfo()
• func ScanC()
• func ScanConfig()
• func ScanHostPublicInfo(host *HostService)
• func ScanPort(ip string, port uint, timeout time.Duration) bool
• func SendPeer(host *HostService)
• func StartServer(publicport uint)
• func Ulimit() int64
• func UpdatePeers()
• func UpdatePeersData()
• type Config
• type DataPoint
• type DataPoolList
  • func DataMap2DataList(dm DataPoolMap) DataPoolList
  • func (a DataPoolList) Len() int
  • func (a DataPoolList) Less(i, j int) bool
  • func (a DataPoolList) Swap(i, j int)
• type DataPoolMap
  • func DataList2DataMap(dl DataPoolList) DataPoolMap
• type HostService
  • func AllHosts() []*HostService
  • func ScanNetwork(cidr string, ports []uint) (out []*HostService)
• type HostServiceList
  • func (a HostServiceList) Len() int
  • func (a HostServiceList) Less(i, j int) bool
  • func (a HostServiceList) Swap(i, j int)
• type InfoStruct
• type Node
  • func NewNode() *Node
  • func (n *Node) AppendDataPoint(dataPoint DataPoint) []DataPoint
  • func (n *Node) AppendDataPoints(dataPoints []DataPoint) []DataPoint
  • func (n *Node) DeleteAllMatchingPrefix(keyStr string) []DataPoint
  • func (n *Node) DeleteDataPoint(keyStr string) []DataPoint
  • func (n *Node) DeleteDataPointWithPrefix(prefix, key string) []DataPoint
  • func (n *Node) GetAllMatchingPrefix(keyStr string) []DataPoint
  • func (n *Node) GetDataPoint(keyStr string) DataPoint
  • func (n *Node) GetDataPointWithPrefix(prefix, key string) DataPoint
  • func (n *Node) JsonDump() []byte
  • func (n *Node) SetDataPoint(key string, val []byte) []DataPoint
  • func (n *Node) SetDataPointWithPrefix(prefix, key string, val []byte) []DataPoint
  • func (n *Node) SetDataPointWithPrefix_iface(prefix, key string, val interface{}) []DataPoint
  • func (n *Node) SetDataPointWithPrefix_str(prefix, key string, val string) []DataPoint
• type PortScanner
  • func (ps *PortScanner) ScanList(timeout time.Duration, ports []uint) (out []uint)
  • func (ps *PortScanner) Start(startPort, endPort uint, timeout time.Duration)
• type Service

Variables

var AppendHostsLock = &sync.Mutex{}

var GlobalScanSemaphore = semaphore.NewWeighted(5)

var HostListLock = sync.Mutex{}

var HostsMap map[string]*HostService

var PortsToScan = []uint{16002}

Functions

func AddHost

func AddHost(h *HostService)

func AddHosts

func AddHosts(hs []*HostService)

func AnnounceAll

func AnnounceAll()

Send our details to all known peers

func AnnounceSelf

func AnnounceSelf(ip, port string)

Send our data to one peer

func AppendHosts

func AppendHosts(hs []*HostService)

func ArpScan

func ArpScan()

func CidrHosts

func CidrHosts(cidr string) ([]string, error)

func FormatHttpIpPort

func FormatHttpIpPort(ip, port string) string

Take an ip, port and return a url without trailing slash

func FormatHttpIpPort_i

func FormatHttpIpPort_i(ip string, port uint) string

As for FormatHttpIpPort, but the port can be a uint

func FuckIpv6

func FuckIpv6(ip string) string

Attempt to retrieve an ipv4 or ipv6 address from a string

func Hosts2Json

func Hosts2Json() []byte

func ScanAllHostsPublicInfo

func ScanAllHostsPublicInfo()

func ScanC

func ScanC()

func ScanConfig

func ScanConfig()

func ScanHostPublicInfo

func ScanHostPublicInfo(host *HostService)

func ScanPort

func ScanPort(ip string, port uint, timeout time.Duration) bool

func SendPeer

func SendPeer(host *HostService)

Send our host list to a peer

func StartServer

func StartServer(publicport uint)

func Ulimit

func Ulimit() int64

func UpdatePeers

func UpdatePeers()

Send our host list to all known peers, update our host list based on their response

func UpdatePeersData

func UpdatePeersData()

Send our frogpond data to another peer

Types

type Config

type Config struct {
	HttpPort           uint
	StartPagePort      uint
	Name               string
	MaxUploadSize      uint
	Networks           []string
	KnownPeers         []string
	ArpCheckInterval   int
	PeerUpdateInterval int
}

var Configuration Config

type DataPoint

type DataPoint struct {
	Key     []byte
	Value   []byte
	Name    string
	Updated time.Time
	Deleted bool
}

type DataPoolList

type DataPoolList []DataPoint

func DataMap2DataList

func DataMap2DataList(dm DataPoolMap) DataPoolList

Convert a map of data points to a list of data points

func (DataPoolList) Len

func (a DataPoolList) Len() int

func (DataPoolList) Less

func (a DataPoolList) Less(i, j int) bool

func (DataPoolList) Swap

func (a DataPoolList) Swap(i, j int)

type DataPoolMap

type DataPoolMap map[string]DataPoint

func DataList2DataMap

func DataList2DataMap(dl DataPoolList) DataPoolMap

Convert a list of data points to a map of data points

type HostService

type HostService struct {
	GUID     string
	Ip       string
	Ports    []uint
	Port     uint
	Services []Service
	Name     string
	LastSeen time.Time
	Path     string
}

var ThisHost *HostService

func AllHosts

func AllHosts() []*HostService

func ScanNetwork

func ScanNetwork(cidr string, ports []uint) (out []*HostService)

type HostServiceList

type HostServiceList []*HostService

func (HostServiceList) Len

func (a HostServiceList) Len() int

func (HostServiceList) Less

func (a HostServiceList) Less(i, j int) bool

func (HostServiceList) Swap

func (a HostServiceList) Swap(i, j int)

type InfoStruct

type InfoStruct struct {
	GUID     string
	Name     string
	Services []Service
}

var Info InfoStruct

type Node

type Node struct {
	DataPoolLock sync.Mutex
	DataPool     DataPoolMap
}

var ThisNode *Node

func NewNode

func NewNode() *Node

Create a new frogpond node

func (*Node) AppendDataPoint

func (n *Node) AppendDataPoint(dataPoint DataPoint) []DataPoint

Add or update a single data point to the data pool

func (*Node) AppendDataPoints

func (n *Node) AppendDataPoints(dataPoints []DataPoint) []DataPoint

Add or update a list of data points to the data pool

func (*Node) DeleteAllMatchingPrefix

func (n *Node) DeleteAllMatchingPrefix(keyStr string) []DataPoint

func (*Node) DeleteDataPoint

func (n *Node) DeleteDataPoint(keyStr string) []DataPoint

func (*Node) DeleteDataPointWithPrefix

func (n *Node) DeleteDataPointWithPrefix(prefix, key string) []DataPoint

func (*Node) GetAllMatchingPrefix

func (n *Node) GetAllMatchingPrefix(keyStr string) []DataPoint

func (*Node) GetDataPoint

func (n *Node) GetDataPoint(keyStr string) DataPoint

Get a single data point

func (*Node) GetDataPointWithPrefix

func (n *Node) GetDataPointWithPrefix(prefix, key string) DataPoint

Get a single data point with a prefix, e.g. "/foo/" and "bar" becomes "/foo/bar"

func (*Node) JsonDump

func (n *Node) JsonDump() []byte

Dump the entire data pool as a json array

func (*Node) SetDataPoint

func (n *Node) SetDataPoint(key string, val []byte) []DataPoint

Set a single data point

func (*Node) SetDataPointWithPrefix

func (n *Node) SetDataPointWithPrefix(prefix, key string, val []byte) []DataPoint

Set a single data point with a prefix, e.g. "/foo/" and "bar" becomes "/foo/bar"

func (*Node) SetDataPointWithPrefix_iface

func (n *Node) SetDataPointWithPrefix_iface(prefix, key string, val interface{}) []DataPoint

Set a single data point with a prefix, e.g. "/foo/" and "bar" becomes "/foo/bar"

func (*Node) SetDataPointWithPrefix_str

func (n *Node) SetDataPointWithPrefix_str(prefix, key string, val string) []DataPoint

Set a single data point with a prefix, e.g. "/foo/" and "bar" becomes "/foo/bar"

type PortScanner

type PortScanner struct {
	// contains filtered or unexported fields
}

func (*PortScanner) ScanList

func (ps *PortScanner) ScanList(timeout time.Duration, ports []uint) (out []uint)

func (*PortScanner) Start

func (ps *PortScanner) Start(startPort, endPort uint, timeout time.Duration)

type Service

type Service struct {
	Name        string
	Ip          string
	Port        int
	Protocol    string
	Description string
	Global      bool
	Path        string
}