store

package
v0.0.0-...-da267cc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 10, 2021 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package store provides the underlying interfaces and glue for the backend storage drivers.

We define 3 distinct interfaces to allow for flexibility in storage options. Store is meant as a persistent storage backend for user data which is backed to permanent storage StoreI is meant as a persistent storage backend for torrent data which is backed to permanent storage PeerStore is meant as a cache to store ephemeral peer/swarm data, it does not need to be backed by persistent storage, but the option is there if desired.

NOTE defer calls should not be used anywhere in the store packages to reduce as much overhead as possible.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddDriver 

func AddDriver(name string, driver Driver)

AddDriver will register a new driver able to instantiate a StoreI

func InfoHashFromBytes 

func InfoHashFromBytes(infoHash *InfoHash, b []byte) error

InfoHashFromBytes returns a binary infohash from a byte array

func InfoHashFromHex 

func InfoHashFromHex(infoHash *InfoHash, h string) error

InfoHashFromHex returns a binary infohash from a byte array

func InfoHashFromString 

func InfoHashFromString(infoHash *InfoHash, s string) error

InfoHashFromString returns a binary infohash from the info string

func PeerHashFromHex 

func PeerHashFromHex(peerHash *PeerHash, h string) error

PeerHashFromHex returns a binary infohash from a byte array

func TestStore 

func TestStore(t *testing.T, s Store)

TestTorrentStore tests the interface implementation

Types

type AnnounceHist 

type AnnounceHist struct {
	Downloaded uint64
	Uploaded   uint64
	Timestamp  time.Time
}

type BTClient 

type BTClient struct {
	Name     string
	Major    int
	Minor    int
	Patch    int
	SubPatch int
}

func ClientString 

func ClientString(peerID PeerID) BTClient

ClientString transforms the peer id into a client and version description

Shadow Style: Each character in the version string represents a number from 0 to 63. '0'=0, ..., '9'=9, 'A'=10, ..., 'Z'=35, 'a'=36, ..., 'z'=61, '.'=62, '-'=63.

func (BTClient) String 

func (b BTClient) String() string

type Driver 

type Driver interface {
	// New instantiates a new StoreI
	New(config config.StoreConfig) (Store, error)
}

Driver provides a interface to enable registration of StoreI drivers

type InfoHash 

type InfoHash [20]byte

InfoHash is a unique 20byte identifier for a torrent

func (InfoHash) Bytes 

func (ih InfoHash) Bytes() []byte

Bytes returns the raw bytes of the info_hash. This is primarily useful for inserting to SQL stores since they have trouble with the sized variant

func (*InfoHash) RawString 

func (ih *InfoHash) RawString() string

RawString returns a 20-byte string of the raw bytes of the ID.

func (*InfoHash) Scan 

func (ih *InfoHash) Scan(v interface{}) error

Scan implements the sql.Scanner interface for conversion to our custom type

func (InfoHash) String 

func (ih InfoHash) String() string

String implements fmt.Stringer, returning the base16 encoded PeerID.

func (InfoHash) URLEncode 

func (ih InfoHash) URLEncode() string

URLEncode returns the peer id suitably encoded for a URL

func (*InfoHash) Value 

func (ih *InfoHash) Value() (driver.Value, error)

Value implements the database.Valuer interface

type Peer 

type Peer struct {
	// Total amount uploaded as reported by client
	Uploaded uint64 `db:"total_uploaded" redis:"total_uploaded" json:"total_uploaded"`
	// Total amount downloaded as reported by client
	Downloaded uint64 `db:"total_downloaded" redis:"total_downloaded" json:"total_downloaded"`
	// Clients reported bytes left of the download
	Left uint32 `db:"total_left" redis:"total_left" json:"total_left"`
	// Total active swarm participation time
	TotalTime time.Duration `db:"total_time" redis:"total_time" json:"total_time"`
	// Current speed up, bytes/sec
	SpeedUP uint32 `db:"speed_up" redis:"speed_up" json:"speed_up"`
	// Current speed dn, bytes/sec
	SpeedDN uint32 `db:"speed_dn" redis:"speed_dn" json:"speed_dn"`
	// Max recorded up speed, bytes/sec
	SpeedUPMax uint32 `db:"speed_up_max"  redis:"speed_up_max" json:"speed_up_max"`
	// Max recorded dn speed, bytes/sec
	SpeedDNMax uint32 `db:"speed_dn_max" redis:"speed_dn_max" json:"speed_dn_max"`
	// Clients IPv4 Address detected automatically, does not use client supplied value
	IP   net.IP `db:"addr_ip" redis:"addr_ip" json:"addr_ip"`
	IPv6 bool   `db:"ipv6" json:"ipv6"`
	// Clients reported port
	Port uint16 `db:"addr_port" redis:"addr_port" json:"addr_port"`
	// Total number of announces the peer has made
	Announces uint32 `db:"total_announces" redis:"total_announces" json:"total_announces"`
	// Last announce timestamp
	AnnounceLast time.Time `db:"announce_last" redis:"announce_last" json:"announce_last"`
	// First announce timestamp
	AnnounceFirst time.Time `db:"announce_first" redis:"announce_first" json:"announce_first"`
	// Peer id, reported by client. Must have white-listed prefix
	PeerID      PeerID      `db:"peer_id" redis:"peer_id" json:"peer_id"`
	Location    geo.LatLong `db:"location" redis:"location" json:"location"`
	CountryCode string      `db:"country_code" json:"country_code"`
	ASN         uint32      `db:"asn" json:"asn"`
	AS          string      `db:"as_name" json:"as_name"`
	UserID      uint32      `db:"user_id" redis:"user_id" json:"user_id"`
	// Client is the user-agent header sent
	Client string `db:"client" json:"client"`
	// TODO Do we actually care about these times? Announce times likely enough
	//CreatedOn time.Time `db:"created_on" redis:"created_on" json:"created_on"`
	//UpdatedOn time.Time `db:"updated_on" redis:"updated_on" json:"updated_on"`
	CryptoLevel consts.CryptoLevel `db:"crypto_level" json:"crypto_level"`
	Paused      bool
}

Peer represents a single unique peer in a swarm

func GenerateTestPeer 

func GenerateTestPeer() *Peer

GenerateTestPeer creates a peer using fake data for the provided user. Used for testing.

func NewPeer 

func NewPeer(userID uint32, peerID PeerID, ip net.IP, port uint16) *Peer

NewPeer create a new peer instance for inserting into a swarm

func (*Peer) Expired 

func (peer *Peer) Expired() bool

Expired checks if the peer last lost contact with us TODO remove hard coded expiration time

func (*Peer) IsNew 

func (peer *Peer) IsNew() bool

IsNew checks if the peer is making its first announce request

func (*Peer) Valid 

func (peer *Peer) Valid() bool

Valid returns true if the peer data meets the minimum requirements to participate in swarms

type PeerHash 

type PeerHash [40]byte

PeerHash is a merger of the infohash and peer_id, used for simpler map lookups

func NewPeerHash 

func NewPeerHash(ih InfoHash, pid PeerID) PeerHash

NewPeerHash created a new PeerHash from the existing infohash and peer_id

func (PeerHash) InfoHash 

func (ph PeerHash) InfoHash() InfoHash

InfoHash returns the first 20 bytes of the data

func (PeerHash) PeerID 

func (ph PeerHash) PeerID() PeerID

PeerID returns the last 20 bytes of the data

func (PeerHash) String 

func (ph PeerHash) String() string

String implements fmt.Stringer, returning the base16 encoded PeerID.

type PeerID 

type PeerID [20]byte

PeerID is the client supplied unique identifier for a peer

func PeerIDFromString 

func PeerIDFromString(s string) PeerID

PeerIDFromString translates a string into a binary PeerID

func (PeerID) Bytes 

func (p PeerID) Bytes() []byte

Bytes returns the raw bytes of the peer_id. This is primarily useful for inserting to SQL stores since they have trouble with the sized variant

func (PeerID) RawString 

func (p PeerID) RawString() string

RawString returns a 20-byte string of the raw bytes of the ID.

func (*PeerID) Scan 

func (p *PeerID) Scan(v interface{}) error

Scan implements the sql.Scanner interface for conversion to our custom type

func (PeerID) String 

func (p PeerID) String() string

String implements fmt.Stringer, returning the base16 encoded PeerID.

func (PeerID) URLEncode 

func (p PeerID) URLEncode() string

URLEncode returns the peer id suitably encoded for a URL

func (*PeerID) Value 

func (p *PeerID) Value() (driver.Value, error)

Value implements the driver.Valuer interface

type PeerStats 

type PeerStats struct {
	Left   uint32
	Hist   []AnnounceHist
	Paused bool
}

PeerStats is any info to batch peer updates

func (*PeerStats) Totals 

func (ps *PeerStats) Totals() PeerSummary

type PeerSummary 

type PeerSummary struct {
	TotalUp    uint64
	TotalDn    uint64
	SpeedUp    uint64
	SpeedDn    uint64
	SpeedUpMax uint64
	SpeedDnMax uint64
	LastAnn    time.Time
}

type Peers 

type Peers map[PeerID]*Peer

type Role 

type Role struct {
	RoleID          uint32    `json:"role_id" db:"role_id"`
	RemoteID        uint64    `json:"remote_id" db:"remote_id"`
	RoleName        string    `json:"role_name" db:"role_name"`
	Priority        int32     `json:"priority" db:"priority"`
	MultiUp         float64   `json:"multi_up" db:"multi_up"`
	MultiDown       float64   `json:"multi_down" db:"multi_down"`
	DownloadEnabled bool      `json:"download_enabled" db:"download_enabled"`
	UploadEnabled   bool      `json:"upload_enabled" db:"upload_enabled"`
	CreatedOn       time.Time `json:"created_on" db:"created_on"`
	UpdatedOn       time.Time `json:"updated_on" db:"updated_on"`
}

func GenerateTestRole 

func GenerateTestRole() Role

GenerateTestRole creates a role using fake data. Used for testing.

func (Role) Log 

func (r Role) Log() *log.Entry

type Roles 

type Roles map[uint32]*Role

Roles is a map of roles by role_id

func (Roles) Get 

func (r Roles) Get(roleID uint32) *Role

Remove removes a users from a Users slice

type Store 

type Store interface {
	Users() (Users, error)
	// UserAdd will add a new user to the backing store
	UserAdd(u *User) error
	// UserGetByPasskey returns a user matching the passkey
	UserGetByPasskey(passkey string) (*User, error)
	// UserGetByID returns a user matching the userId
	UserGetByID(userID uint32) (*User, error)
	// UserDelete removes a user from the backing store
	UserDelete(user *User) error
	// UserSave is used to change a known user
	UserSave(user *User) error
	// Close will cleanup and close the underlying storage driver if necessary
	UserSync(b []*User) error

	// Roles fetches all known groups
	Roles() (Roles, error)
	// Roles fetches all known groups
	RoleByID(roleID uint32) (*Role, error)
	// RoleAdd adds a new role to the system
	RoleAdd(role *Role) error
	// RoleDelete permanently deletes a role from the system
	RoleDelete(roleID uint32) error
	// RoleSave commits the role to persistent store
	RoleSave(role *Role) error

	// Torrents returns all torrents in the store
	Torrents() (Torrents, error)
	// TorrentAdd adds a new torrent to the backing store
	TorrentAdd(t *Torrent) error
	// TorrentDelete will mark a torrent as deleted in the backing store.
	// If dropRow is true, it will permanently remove the torrent from the store
	TorrentDelete(ih InfoHash, dropRow bool) error
	// TorrentGet returns the Torrent matching the infohash
	TorrentGet(hash InfoHash, deletedOk bool) (*Torrent, error)
	// TorrentSave will update certain parameters within the torrent
	TorrentSave(torrent *Torrent) error
	// TorrentSync batch updates the backing store with the new TorrentStats provided
	TorrentSync(b []*Torrent) error

	// WhiteListDelete removes a client from the global whitelist
	WhiteListDelete(client *WhiteListClient) error
	// WhiteListAdd will insert a new client prefix into the allowed clients list
	WhiteListAdd(client *WhiteListClient) error
	// WhiteListGetAll fetches all known whitelisted clients
	WhiteListGetAll() ([]*WhiteListClient, error)

	// Migrate
	Migrate() error

	// Conn returns the underlying connection, if any
	Conn() interface{}
	// Name returns the name of the data store type
	Name() string
	// Close will cleanup and close the underlying storage driver if necessary
	Close() error
}

Store defines a interface used to retrieve user data from a backing store. These should be cached indefinitely, we treat any known user as allowed to connect. To disable a user they MUST be deleted from the active user cache

func NewStore 

func NewStore(config config.StoreConfig) (Store, error)

NewStore will attempt to initialize a StoreI using the driver name provided

type Swarm 

type Swarm struct {
	Peers
	Seeders  int
	Leechers int
	*sync.RWMutex
}

Swarm is a set of users participating in a torrent

func NewSwarm 

func NewSwarm() *Swarm

NewSwarm instantiates a new swarm

func (Swarm) Add 

func (s Swarm) Add(p *Peer)

Add inserts a new peer into the swarm

func (Swarm) Get 

func (s Swarm) Get(peerID PeerID) (*Peer, error)

Get will copy a peer into the peer pointer passed in if it exists.

func (Swarm) GetN 

func (s Swarm) GetN(n int) ([]*Peer, error)

Get will copy a peer into the peer pointer passed in if it exists.

func (Swarm) ReapExpired 

func (s Swarm) ReapExpired(infoHash InfoHash) []PeerHash

ReapExpired will delete any peers from the swarm that are considered expired

func (Swarm) Remove 

func (s Swarm) Remove(p PeerID)

Remove removes a peer from a slice

func (Swarm) UpdatePeer 

func (s Swarm) UpdatePeer(peerID PeerID, stats PeerStats) (*Peer, bool)

UpdatePeer will update a swarm member with new stats

type Torrent 

type Torrent struct {
	InfoHash InfoHash `db:"info_hash" json:"info_hash"`
	Snatches uint32   `db:"total_completed" json:"total_completed"`
	// This is stored as MB to reduce storage costs
	Uploaded uint64 `db:"total_uploaded" json:"total_uploaded"`
	// This is stored as MB to reduce storage costs
	Downloaded uint64 `db:"total_downloaded" json:"total_downloaded"`
	// This is stored as MB to reduce storage costs. Totals without multipliers added
	UploadedReal uint64 `db:"total_uploaded_real" json:"total_uploaded_real"`
	// This is stored as MB to reduce storage costs.  Totals without multipliers added
	DownloadedReal uint64 `db:"total_downloaded_real" json:"total_downloaded_real"`
	IsDeleted      bool   `db:"is_deleted" json:"is_deleted"`
	// When you have a message to pass to a client set enabled = false and set the reason message.
	// If IsDeleted is true, then nothing will be returned to the client
	IsEnabled bool `db:"is_enabled" json:"is_enabled"`
	// Reason when set will return a message to the torrent client
	Reason string `db:"reason" json:"reason"`
	// Upload multiplier added to the users totals
	MultiUp float64 `db:"multi_up" json:"multi_up"`
	// Download multiplier added to the users totals
	// 0 denotes freeleech status
	MultiDn   float64   `db:"multi_dn" json:"multi_dn"`
	Announces uint64    `db:"announces" json:"announces"`
	Seeders   uint32    `db:"seeders" json:"seeders"`
	Leechers  uint32    `db:"leechers" json:"leechers"`
	Title     string    `db:"title" json:"title"`
	CreatedOn time.Time `db:"created_on" json:"created_on"`
	UpdatedOn time.Time `db:"updated_on" json:"updated_on"`

	Peers *Swarm `db:"-" json:"peers"`

	// Keeps track of how often the values have been changes
	// TODO Items with the most writes will get written to soonest
	Writes uint32 `db:"-" json:"-"`
}

Torrent is the core struct for our torrent being tracked

func GenerateTestTorrent 

func GenerateTestTorrent() Torrent

GenerateTestTorrent creates a torrent using fake data. Used for testing.

func NewTorrent 

func NewTorrent(ih InfoHash) Torrent

NewTorrent allocates and returns a new Torrent instance pointer with all the minimum value required to operated in place

func (*Torrent) Log 

func (t *Torrent) Log() *log.Entry

type TorrentStats 

type TorrentStats struct {
	Seeders    uint32 `json:"seeders"`
	Leechers   uint32 `json:"leechers"`
	Snatches   uint32 `json:"snatches"`
	Uploaded   uint64 `json:"uploaded"`
	Downloaded uint64 `json:"downloaded"`
	Announces  uint64 `json:"announces"`
}

TorrentStats is used to relay info stats for a torrent around. It contains rolled up stats from peer info as well as the normal torrent stats.

type TorrentUpdate 

type TorrentUpdate struct {
	Keys        []string
	ReleaseName string  `json:"release_name"`
	IsDeleted   bool    `json:"is_deleted"`
	IsEnabled   bool    `json:"is_enabled"`
	Reason      string  `json:"reason"`
	MultiUp     float64 `json:"multi_up"`
	MultiDn     float64 `json:"multi_dn"`
}

type Torrents 

type Torrents map[InfoHash]*Torrent

Torrents is a basic type alias for multiple torrents

type UpdateState 

type UpdateState struct {
	InfoHash InfoHash
	PeerID   PeerID
	Passkey  string
	// Total amount uploaded as reported by client
	Uploaded uint64
	// Total amount downloaded as reported by client
	Downloaded uint64
	// Clients reported bytes left of the download
	Left uint32
	// Timestamp is the time the new stats were announced
	Timestamp time.Time
	Event     consts.AnnounceType
	Paused    bool
}

UpdateState is used to store temporary data used for batch updates

type User 

type User struct {
	UserID          uint32    `db:"user_id" json:"user_id"`
	RoleID          uint32    `json:"role_id" db:"role_id"`
	RemoteID        uint64    `json:"remote_id" db:"remote_id"`
	UserName        string    `db:"user_name" json:"user_name"`
	Passkey         string    `db:"passkey" json:"passkey"`
	IsDeleted       bool      `db:"is_deleted" json:"is_deleted"`
	DownloadEnabled bool      `db:"download_enabled" json:"download_enabled"`
	Downloaded      uint64    `db:"downloaded" json:"downloaded"`
	Uploaded        uint64    `db:"uploaded" json:"uploaded"`
	Announces       uint32    `db:"announces" json:"announces"`
	CreatedOn       time.Time `db:"created_on" json:"created_on"`
	UpdatedOn       time.Time `db:"updated_on" json:"updated_on"`
	Role            *Role     `json:"role" db:"-"`

	// Keeps track of how often the values have been changes
	// TODO Items with the most writes will get written to soonest
	Writes uint32 `db:"-" json:"-"`
}

User defines a basic user known to the tracker All users are considered enabled if they exist. You must remove them from the backing store to ensure they cannot access any resources

func GenerateTestUser 

func GenerateTestUser() User

GenerateTestUser creates a peer using fake data. Used for testing.

func (User) Log 

func (u User) Log() *log.Entry

func (User) Valid 

func (u User) Valid() bool

Valid performs basic validation of the user info ensuring we have the minimum required data to be considered valid by the tracker

type UserStats 

type UserStats struct {
	Uploaded   uint64
	Downloaded uint64
	Announces  uint32
}

UserStats is any info we want to batch update for a user

type Users 

type Users map[string]*User

Users is a slice of known users

func (Users) Remove 

func (users Users) Remove(p *User)

Remove removes a users from a Users slice

type WhiteList 

type WhiteList map[string]*WhiteListClient

WhiteList is a map of whitelisted clients by 8 chars of client prefix

type WhiteListClient 

type WhiteListClient struct {
	ClientPrefix string `db:"client_prefix" json:"client_prefix"`
	ClientName   string `db:"client_name" json:"client_name"`
}

WhiteListClient defines a whitelisted bittorrent client allowed to participate in swarms. This is not a foolproof solution as its fairly trivial for a motivated attacker to fake this.

func (WhiteListClient) Match 

func (wl WhiteListClient) Match(client string) bool

Match returns true if the client matches this prefix

Source Files

View all Source files

Directories

Path Synopsis
Package mysql provides mysql/mariadb backed persistent storage
 Package mysql provides mysql/mariadb backed persistent storage
Package postgres provides the backing store for postgresql TODO create domains for the uint types, eg: create domain uint64 as numeric(20,0);
 Package postgres provides the backing store for postgresql TODO create domains for the uint types, eg: create domain uint64 as numeric(20,0);

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.