zif

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 7, 2017 License: AGPL-3.0 Imports: 33 Imported by: 0

README

Zif

Whitepaper

A distributed torrent sharing and indexing network.

NOTE: I'd prefer if people please refrain from putting this on Hacker News/Reddit/lobste.rs/etc for the moment, it's still buggy and messy and not yet how I want it to be

Check out the develop branch for the latest work!

Zif makes it easy to share files, without having any bandwidth or storage costs. It uses Bittorrent for file transfers, but provides a decentralised index for discovering new content.

What even is this?

  • a queryable database of torrents and metadata
  • a p2p network for the distribution of torrent info hashes and metadata

Peer discovery is done via an implementation of Kademlia, a DHT similar to the one Bittorrent uses. Peers are assigned addresses similar to Bitcoin addresses, except using Ed25519 and SHA3.

Sounds cool, when can I use it?

Sometime in the future. As of the time of writing, I have written a DHT for peer resolution, a database system for post storage/indexing, and a protocol for remote searching of posts and mirroring of peer databases. There's also a graphical client in the works, see the ui folder!

It's actually relatively usable at the moment, just needs more testing - the UI also needs work for it to be properly functional.

Documentation

Index

Constants

View Source
const AnnounceFrequency = time.Minute * 30
View Source
const HeartbeatFrequency = time.Second * 30
View Source
const MaxSearchLength = 256
View Source
const ResolveListSize = 1
View Source
const SeedSearchFrequency = time.Minute * 5
View Source
const TimeBeforeReExplore = 60 * 60

Variables

View Source
var (
	PeerUnreachable  = errors.New("Peer could not be reached")
	PeerDisconnected = errors.New("Peer has disconnected")
	RecursionLimit   = errors.New("Recursion limit reached, peer cannot be resolved")
)

Functions

func CreateNetMap

func CreateNetMap(entry dht.Entry, db *dht.DHT, currentNodes map[string]bool, currentLinks map[string]bool) ([]MapNode, []MapLink)

This takes a node to start with, and recurses through all seeds/seeding

func ExternalIp

func ExternalIp() string

TODO: Make this check using UpNp/NAT_PMP first, then query services.

func SetupZifTorService

func SetupZifTorService(port, tor int, cookie string) (*torc.TorControl, string, error)

Types

type CommandAddMeta

type CommandAddMeta struct {
	CommandMeta
	Value string `json:"value"`
}

type CommandAddPost

type CommandAddPost struct {
	data.Post
	Index bool
}

type CommandAddressEncode

type CommandAddressEncode struct {
	Raw []byte `json:"raw"`
}

type CommandAnnounce

type CommandAnnounce CommandPeer

type CommandBootstrap

type CommandBootstrap CommandPeer

type CommandFile

type CommandFile struct {
	File string `json:"file"`
}

type CommandGetMeta

type CommandGetMeta CommandMeta

type CommandLocalGet

type CommandLocalGet struct {
	Key string `json:"key"`
}

type CommandLocalSet

type CommandLocalSet struct {
	Key   string `json:"key"`
	Value string `json:"key"`
}

Used for setting values in the localpeer entry

type CommandMeta

type CommandMeta struct {
	PId int `json:"pid"`
}

type CommandMirror

type CommandMirror CommandPeer

type CommandMirrorProgress

type CommandMirrorProgress CommandPeer

type CommandNetMap

type CommandNetMap struct {
	Address string
}

type CommandPeer

type CommandPeer struct {
	Address string `json:"address"`
}

type CommandPeerIndex

type CommandPeerIndex struct {
	CommandPeer
	Since int `json:"since"`
}

type CommandPeerPopular

type CommandPeerPopular CommandPeerRecent

type CommandPeerRecent

type CommandPeerRecent struct {
	CommandPeer
	Page int `json:"page"`
}

type CommandPeerSearch

type CommandPeerSearch CommandRSearch

type CommandPeers

type CommandPeers interface{}

type CommandPing

type CommandPing CommandPeer

type CommandRSearch

type CommandRSearch struct {
	CommandPeer
	Query string `json:"query"`
	Page  int    `json:"page"`
}

type CommandRebuildCollection

type CommandRebuildCollection interface{}

type CommandRequestAddPeer

type CommandRequestAddPeer struct {
	// The peer to send the request to
	Remote string `json:"remote"`
	// The peer we wish to be registered as a peer for
	Peer string `json:"peer"`
}

type CommandResolve

type CommandResolve CommandPeer

type CommandResult

type CommandResult struct {
	IsOK   bool        `json:"status"`
	Result interface{} `json:"value"`
	Error  error       `json:"err"`
}

func (*CommandResult) WriteJSON

func (cr *CommandResult) WriteJSON(w io.Writer)

type CommandSaveCollection

type CommandSaveCollection interface{}

type CommandSaveRoutingTable

type CommandSaveRoutingTable interface{}

type CommandSearchEntry

type CommandSearchEntry struct {
	Name string
	Desc string
	Page int
}

type CommandSelfIndex

type CommandSelfIndex struct {
	Since int `json:"since"`
}

type CommandSelfPopular

type CommandSelfPopular CommandSelfRecent

type CommandSelfRecent

type CommandSelfRecent struct {
	Page int `json:"page"`
}

type CommandSelfSearch

type CommandSelfSearch struct {
	CommandSuggest
	Page int `json:"page"`
}

type CommandServer

type CommandServer struct {
	LocalPeer *LocalPeer

	// Piece count for ongoing mirrors
	MirrorProgress cmap.ConcurrentMap
}

func NewCommandServer

func NewCommandServer(lp *LocalPeer) *CommandServer

func (*CommandServer) AddMeta

func (cs *CommandServer) AddMeta(cam CommandAddMeta) CommandResult

func (*CommandServer) AddPost

func (cs *CommandServer) AddPost(ap CommandAddPost) CommandResult

func (*CommandServer) AddressEncode

func (cs *CommandServer) AddressEncode(ce CommandAddressEncode) CommandResult

func (*CommandServer) Announce

func (*CommandServer) Bootstrap

func (cs *CommandServer) Bootstrap(cb CommandBootstrap) CommandResult

func (*CommandServer) EntrySearch

func (cs *CommandServer) EntrySearch(ps CommandSearchEntry) CommandResult

func (*CommandServer) Explore

func (cs *CommandServer) Explore() CommandResult

func (*CommandServer) GetMirrorProgress

func (cs *CommandServer) GetMirrorProgress(cmp CommandMirrorProgress) CommandResult

func (*CommandServer) LocalGet

func (cs *CommandServer) LocalGet(clg CommandLocalGet) CommandResult

func (*CommandServer) LocalSet

func (cs *CommandServer) LocalSet(cls CommandLocalSet) CommandResult

Set a value in the localpeer entry

func (*CommandServer) MemProfile

func (cs *CommandServer) MemProfile(cf CommandFile) CommandResult

func (*CommandServer) Mirror

func (cs *CommandServer) Mirror(cm CommandMirror) CommandResult

func (*CommandServer) NetMap

func (cs *CommandServer) NetMap(cnm CommandNetMap) CommandResult

func (*CommandServer) PeerIndex

func (cs *CommandServer) PeerIndex(ci CommandPeerIndex) CommandResult

func (*CommandServer) PeerPopular

func (cs *CommandServer) PeerPopular(pp CommandPeerPopular) CommandResult

func (*CommandServer) PeerRecent

func (cs *CommandServer) PeerRecent(pr CommandPeerRecent) CommandResult

func (*CommandServer) PeerSearch

func (cs *CommandServer) PeerSearch(ps CommandPeerSearch) CommandResult

func (*CommandServer) Peers

func (*CommandServer) Ping

func (*CommandServer) RSearch

func (cs *CommandServer) RSearch(rs CommandRSearch) CommandResult

func (*CommandServer) RebuildCollection

func (cs *CommandServer) RebuildCollection(crc CommandRebuildCollection) CommandResult

func (*CommandServer) RequestAddPeer

func (cs *CommandServer) RequestAddPeer(crap CommandRequestAddPeer) CommandResult

func (*CommandServer) Resolve

func (cs *CommandServer) Resolve(cr CommandResolve) CommandResult

func (*CommandServer) SaveCollection

func (cs *CommandServer) SaveCollection(csc CommandSaveCollection) CommandResult

func (*CommandServer) SelfIndex

func (cs *CommandServer) SelfIndex(ci CommandSelfIndex) CommandResult

func (*CommandServer) SelfPopular

func (cs *CommandServer) SelfPopular(cp CommandSelfPopular) CommandResult

func (*CommandServer) SelfRecent

func (cs *CommandServer) SelfRecent(cr CommandSelfRecent) CommandResult

func (*CommandServer) SelfSearch

func (cs *CommandServer) SelfSearch(css CommandSelfSearch) CommandResult

func (*CommandServer) SelfSuggest

func (cs *CommandServer) SelfSuggest(css CommandSuggest) CommandResult

func (*CommandServer) SetSeedLeech

func (cs *CommandServer) SetSeedLeech(csl CommandSetSeedLeech) CommandResult

func (*CommandServer) StartCpuProfile

func (cs *CommandServer) StartCpuProfile(cf CommandFile) CommandResult

func (*CommandServer) StopCpuProfile

func (cs *CommandServer) StopCpuProfile() CommandResult

type CommandSetSeedLeech

type CommandSetSeedLeech struct {
	Id       uint
	Seeders  uint
	Leechers uint
}

type CommandSuggest

type CommandSuggest struct {
	Query string `json:"query"`
}

type HttpServer

type HttpServer struct {
	CommandServer *CommandServer
}

func (*HttpServer) AddMeta

func (hs *HttpServer) AddMeta(w http.ResponseWriter, r *http.Request)

func (*HttpServer) AddPost

func (hs *HttpServer) AddPost(w http.ResponseWriter, r *http.Request)

func (*HttpServer) AddressEncode

func (hs *HttpServer) AddressEncode(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Announce

func (hs *HttpServer) Announce(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Bootstrap

func (hs *HttpServer) Bootstrap(w http.ResponseWriter, r *http.Request)

func (*HttpServer) CpuProfile

func (hs *HttpServer) CpuProfile(w http.ResponseWriter, r *http.Request)

func (*HttpServer) FtsIndex

func (hs *HttpServer) FtsIndex(w http.ResponseWriter, r *http.Request)

func (*HttpServer) IndexHandler

func (hs *HttpServer) IndexHandler(w http.ResponseWriter, r *http.Request)

func (*HttpServer) ListenHttp

func (hs *HttpServer) ListenHttp(addr string)

func (*HttpServer) MemProfile

func (hs *HttpServer) MemProfile(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Mirror

func (hs *HttpServer) Mirror(w http.ResponseWriter, r *http.Request)

func (*HttpServer) MirrorProgress

func (hs *HttpServer) MirrorProgress(w http.ResponseWriter, r *http.Request)

func (*HttpServer) NetMap

func (hs *HttpServer) NetMap(w http.ResponseWriter, r *http.Request)

func (*HttpServer) PeerFtsIndex

func (hs *HttpServer) PeerFtsIndex(w http.ResponseWriter, r *http.Request)

func (*HttpServer) PeerRSearch

func (hs *HttpServer) PeerRSearch(w http.ResponseWriter, r *http.Request)

func (*HttpServer) PeerSearch

func (hs *HttpServer) PeerSearch(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Peers

func (hs *HttpServer) Peers(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Ping

func (hs *HttpServer) Ping(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Popular

func (hs *HttpServer) Popular(w http.ResponseWriter, r *http.Request)

func (*HttpServer) RebuildCollection

func (hs *HttpServer) RebuildCollection(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Recent

func (hs *HttpServer) Recent(w http.ResponseWriter, r *http.Request)

func (*HttpServer) RequestAddPeer

func (hs *HttpServer) RequestAddPeer(w http.ResponseWriter, r *http.Request)

func (*HttpServer) Resolve

func (hs *HttpServer) Resolve(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SaveCollection

func (hs *HttpServer) SaveCollection(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SearchEntry

func (hs *HttpServer) SearchEntry(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SelfExplore

func (hs *HttpServer) SelfExplore(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SelfGet

func (hs *HttpServer) SelfGet(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SelfPopular

func (hs *HttpServer) SelfPopular(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SelfRecent

func (hs *HttpServer) SelfRecent(w http.ResponseWriter, r *http.Request)

TODO: SelfSuggest after merge

func (*HttpServer) SelfSearch

func (hs *HttpServer) SelfSearch(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SelfSet

func (hs *HttpServer) SelfSet(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SelfSuggest

func (hs *HttpServer) SelfSuggest(w http.ResponseWriter, r *http.Request)

func (*HttpServer) SetSeedLeech

func (hs *HttpServer) SetSeedLeech(w http.ResponseWriter, r *http.Request)

type LocalPeer

type LocalPeer struct {
	Peer
	Entry         *dht.Entry
	DHT           *dht.DHT
	Server        *proto.Server
	Collection    *data.Collection
	Database      *data.Database
	PublicAddress string
	// These are the databases of all of the peers that we have mirrored.
	Databases   cmap.ConcurrentMap
	Collections cmap.ConcurrentMap

	SearchProvider *data.SearchProvider
	// contains filtered or unexported fields
}

func (*LocalPeer) AddEntry

func (lp *LocalPeer) AddEntry(entry dht.Entry) error

func (*LocalPeer) AddPost

func (lp *LocalPeer) AddPost(p data.Post, store bool) (int64, error)

func (*LocalPeer) AddSeeding

func (lp *LocalPeer) AddSeeding(entry dht.Entry) error

func (*LocalPeer) Close

func (lp *LocalPeer) Close()

func (*LocalPeer) ConnectPeer

func (lp *LocalPeer) ConnectPeer(addr dht.Address) (*Peer, *dht.Entry, error)

func (*LocalPeer) ConnectPeerDirect

func (lp *LocalPeer) ConnectPeerDirect(addr string) (*Peer, error)

func (*LocalPeer) GenerateKey

func (lp *LocalPeer) GenerateKey()

Generate a ed25519 keypair.

func (*LocalPeer) GetNetworkPeer

func (lp *LocalPeer) GetNetworkPeer(addr dht.Address) proto.NetworkPeer

func (*LocalPeer) GetPeer

func (lp *LocalPeer) GetPeer(addr dht.Address) *Peer

func (*LocalPeer) GetSocksPort

func (lp *LocalPeer) GetSocksPort() int

func (*LocalPeer) HandleAddPeer

func (lp *LocalPeer) HandleAddPeer(msg *proto.Message) error

func (*LocalPeer) HandleAnnounce

func (lp *LocalPeer) HandleAnnounce(msg *proto.Message) error

func (*LocalPeer) HandleCloseConnection

func (lp *LocalPeer) HandleCloseConnection(addr *dht.Address)

func (*LocalPeer) HandleFindClosest

func (lp *LocalPeer) HandleFindClosest(msg *proto.Message) error

func (*LocalPeer) HandleHandshake

func (lp *LocalPeer) HandleHandshake(header proto.ConnHeader) (proto.NetworkPeer, error)

func (*LocalPeer) HandleHashList

func (lp *LocalPeer) HandleHashList(msg *proto.Message) error

func (*LocalPeer) HandlePiece

func (lp *LocalPeer) HandlePiece(msg *proto.Message) error

func (*LocalPeer) HandlePopular

func (lp *LocalPeer) HandlePopular(msg *proto.Message) error

func (*LocalPeer) HandleQuery

func (lp *LocalPeer) HandleQuery(msg *proto.Message) error

Querying peer sends a Zif address This peer will respond with a list of the k closest peers, ordered by distance. The top peer may well be the one that is being queried for :)

func (*LocalPeer) HandleRecent

func (lp *LocalPeer) HandleRecent(msg *proto.Message) error

func (*LocalPeer) HandleSearch

func (lp *LocalPeer) HandleSearch(msg *proto.Message) error

func (*LocalPeer) Listen

func (lp *LocalPeer) Listen(addr string)

Pass the address to listen on. This is for the Zif connection.

func (*LocalPeer) ListenStream

func (lp *LocalPeer) ListenStream(peer *Peer)

func (*LocalPeer) LoadEntry

func (lp *LocalPeer) LoadEntry() error

func (*LocalPeer) PeerCount

func (lp *LocalPeer) PeerCount() int

convenience methods

func (*LocalPeer) Peers

func (lp *LocalPeer) Peers() map[string]*Peer

func (*LocalPeer) QueryEntry

func (lp *LocalPeer) QueryEntry(addr dht.Address) (*dht.Entry, error)

func (*LocalPeer) QuerySelf

func (lp *LocalPeer) QuerySelf()

func (*LocalPeer) ReadKey

func (lp *LocalPeer) ReadKey() error

Read the private key from file. This is the "identity.dat" file. The public key is also then generated from the private key.

func (*LocalPeer) Resolve

func (lp *LocalPeer) Resolve(addr dht.Address) (*dht.Entry, error)

func (*LocalPeer) SaveEntry

func (lp *LocalPeer) SaveEntry() error

func (*LocalPeer) SetNetworkPeer

func (lp *LocalPeer) SetNetworkPeer(p proto.NetworkPeer)

func (*LocalPeer) SetPeer

func (lp *LocalPeer) SetPeer(p *Peer)

func (*LocalPeer) SetSocks

func (lp *LocalPeer) SetSocks(on bool)

func (*LocalPeer) SetSocksPort

func (lp *LocalPeer) SetSocksPort(port int)

func (*LocalPeer) Setup

func (lp *LocalPeer) Setup()

func (*LocalPeer) Sign

func (lp *LocalPeer) Sign(msg []byte) []byte

Sign any bytes.

func (*LocalPeer) SignEntry

func (lp *LocalPeer) SignEntry()

func (*LocalPeer) StartExploring

func (lp *LocalPeer) StartExploring() error

func (*LocalPeer) WriteKey

func (lp *LocalPeer) WriteKey() error

Writes the private key to a file, in this way persisting your identity - all the other addresses can be generated from this, no need to save them. By default this file is "identity.dat"

type MapLink struct {
	Source string `json:"source"`
	Target string `json:"target"`
}

type MapNode

type MapNode struct {
	// the address is treated like an id
	Address string `json:"id"`
	Name    string `json:"name"`
}

type Peer

type Peer struct {
	// contains filtered or unexported fields
}

func (*Peer) AddStream

func (p *Peer) AddStream(conn net.Conn)

func (*Peer) Address

func (p *Peer) Address() *dht.Address

func (*Peer) Announce

func (p *Peer) Announce(lp *LocalPeer) error

func (*Peer) Bootstrap

func (p *Peer) Bootstrap(d *dht.DHT) error

func (*Peer) CloseStreams

func (p *Peer) CloseStreams()

func (*Peer) Connect

func (p *Peer) Connect(addr string, lp *LocalPeer) error

func (*Peer) ConnectClient

func (p *Peer) ConnectClient(lp *LocalPeer) (*yamux.Session, error)

func (*Peer) ConnectServer

func (p *Peer) ConnectServer() (*yamux.Session, error)

func (*Peer) EAddress

func (p *Peer) EAddress() common.Encoder

func (*Peer) Entry

func (p *Peer) Entry() (*dht.Entry, error)

func (*Peer) FindClosest

func (p *Peer) FindClosest(address dht.Address) ([]common.Verifier, error)

func (*Peer) GetCapabilities

func (p *Peer) GetCapabilities() *proto.MessageCapabilities

func (*Peer) GetEntry

func (p *Peer) GetEntry() (*dht.Entry, error)

func (*Peer) GetStream

func (p *Peer) GetStream(conn net.Conn) *proto.Client

func (*Peer) Mirror

func (p *Peer) Mirror(db *data.Database, lp dht.Address, onPiece chan int) error

func (*Peer) NewMessage

func (p *Peer) NewMessage(header string) *proto.Message

func (*Peer) OpenStream

func (p *Peer) OpenStream() (*proto.Client, error)

func (*Peer) Ping

func (p *Peer) Ping(timeOut time.Duration) (time.Duration, error)

func (*Peer) Popular

func (p *Peer) Popular(page int) ([]*data.Post, error)

func (*Peer) PublicKey

func (p *Peer) PublicKey() []byte

func (*Peer) Query

func (p *Peer) Query(address dht.Address) (common.Verifier, error)

func (*Peer) Recent

func (p *Peer) Recent(page int) ([]*data.Post, error)

func (*Peer) RemoveStream

func (p *Peer) RemoveStream(conn net.Conn)

func (*Peer) RequestAddPeer

func (p *Peer) RequestAddPeer(entry dht.Entry) error

func (*Peer) Search

func (p *Peer) Search(search string, page int) (*data.SearchResult, error)

asks a peer to query its database and return the results

func (*Peer) Session

func (p *Peer) Session() *yamux.Session

func (*Peer) SetCapabilities

func (p *Peer) SetCapabilities(caps proto.MessageCapabilities)

func (*Peer) SetTCP

func (p *Peer) SetTCP(header proto.ConnHeader)

func (*Peer) Streams

func (p *Peer) Streams() *proto.StreamManager

func (*Peer) Terminate

func (p *Peer) Terminate()

func (*Peer) UpdateSeen

func (p *Peer) UpdateSeen()

type PeerManager

type PeerManager struct {
	// contains filtered or unexported fields
}

handles peer connections

func NewPeerManager

func NewPeerManager(lp *LocalPeer) *PeerManager

func (*PeerManager) AddSeedManager

func (pm *PeerManager) AddSeedManager(addr dht.Address) error

func (*PeerManager) ConnectPeer

func (pm *PeerManager) ConnectPeer(addr dht.Address) (*Peer, *dht.Entry, error)

Resolved a Zif address into an entry, connects to the peer at the PublicAddress in the Entry, then return it. The peer is also stored in a map.

func (*PeerManager) ConnectPeerDirect

func (pm *PeerManager) ConnectPeerDirect(addr string) (*Peer, error)

Given a direct address, for instance an IP or domain, connect to the peer there. This can be used for something like bootstrapping, or for something like connecting to a peer whose Zif address we have just resolved.

func (*PeerManager) Count

func (pm *PeerManager) Count() int

func (*PeerManager) GetPeer

func (pm *PeerManager) GetPeer(addr dht.Address) *Peer

func (*PeerManager) HandleCloseConnection

func (pm *PeerManager) HandleCloseConnection(addr *dht.Address)

func (*PeerManager) LoadSeeds

func (pm *PeerManager) LoadSeeds() error

func (*PeerManager) Peers

func (pm *PeerManager) Peers() map[string]*Peer

func (*PeerManager) Resolve

func (pm *PeerManager) Resolve(addr dht.Address) (*dht.Entry, error)

Resolves a Zif address into an entry. Hopefully we already have the entry, in which case it's just loaded from disk. Otherwise, recursive network queries are made to try and find it.

func (*PeerManager) SetPeer

func (pm *PeerManager) SetPeer(p *Peer)

type SeedManager

type SeedManager struct {
	Close chan bool
	// contains filtered or unexported fields
}

func NewSeedManager

func NewSeedManager(track dht.Address, lp *LocalPeer) (*SeedManager, error)

Creates a new seed manager, given an address to track seeds for and the localpeer.

func (*SeedManager) Start

func (sm *SeedManager) Start()

Start looking for seeds

Directories

Path Synopsis
cmd
a few network helpers
a few network helpers

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL