privtrak

README

chihaya-privtrak-middleware

A private tracker middleware for the chihaya BitTorrent tracker.

go get -t -u github.com/mrd0ll4r/chihaya-privtrak-middleware

Features

• In-memory storage of peer stats
• Versatile user ID handling (anything unique that fits into 16 bytes works)
• Generates deltas for user/infohash pairs
• Batches deltas, hands them off to a DeltaHandler, for further processing
• Purges stale peer stats
• Built for concurrency

Non-Features

• Doesn't check whether a user is allowed to announce. Write your own middleware for that, don't try to build one monolithic private tracker middleware.
• Doesn't check whether an infohash/client/... is whitelisted. Same as above.

How to use this

the whole thing is in early stages, beware

Roughly, you need to do something like this:

1. Implement a UserIdentifier to derive unique user IDs from requests. You could use HTTP/UDP optional parameters for this, or customized routes, or ... Just make sure the process is fast and reliable. Do not throw database queries in here, you WILL kill your tracker and/or DB. A sane way of doing this would be to issue 16-byte passkeys and just use them as user IDs.
2. Implement a DeltaHandler to handle the deltas. You'd probably write these to a DB, or reprocess them first. They come in batches for a reason - don't insert them into a DB one by one. Before throwing them into a DB, you might want to
   • look up a user's numeric ID
   • look up the torrents numeric ID
   • (aggregate, possibly)
   • push them into an event processing system, for cool continuous queries
   • produce prometheus stats?
3. Compile this middleware into your chihaya instance by adding a few lines in the cmd/chihay/config.go file.
4. Adjust your config accordingly.

Expected issues

• Should roughly double the memory usage of a tracker running optmem. It'd be elegant to merge the two somehow, but I don't see that happening soon.
• Must run as a PreHook to throttle tracker throughput in case of a slow DeltaHandler. Otherwise we just get a goroutine explosion.

Future plans

• Add an interface to deal with backup/restore functionality Basically, it needs to:

  1. Restore everything on startup
  2. Be notified whenever we remove a peer due to GC (maybe also stopped peers? What about partial seeds? What about paused downloads?)
  3. Figure out if we know a user/infohash combination before we create it from scratch

• Possibly include IP/PeerID in the deltas? The DeltaHandler can figure out whether to remove these due to special user's protection.

License

MIT, see LICENSE

Documentation

Index

• Variables
• func New(provided Config, identifier UserIdentifier, handler DeltaHandler) (middleware.Hook, error)
• type Config
  • func (cfg Config) LogFields() log.Fields
• type DeltaHandler
• type ID
• type StatDelta
  • func (s StatDelta) LogFields() log.Fields
• type UserIdentifier

Variables

var (
	// PromGCDurationMilliseconds is a histogram used by storage to record the
	// durations of execution time required for removing expired peers.
	PromGCDurationMilliseconds = prometheus.NewHistogram(prometheus.HistogramOpts{
		Name:    "chihaya_privtrak_gc_duration_milliseconds",
		Help:    "The time it takes to perform privtrak garbage collection",
		Buckets: prometheus.ExponentialBuckets(9.375, 2, 10),
	})

	// PromUsersCount is a gauge used to hold the current total amount of
	// unique swarms being tracked by a storage.
	PromUsersCount = prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "chihaya_privtrak_users_count",
		Help: "The number of users tracked",
	})
)

var ErrClosing = errors.New("privtrak middleware closing")

ErrClosing is returned on an announce if the middleware is shut down.

Functions

func New

func New(provided Config, identifier UserIdentifier, handler DeltaHandler) (middleware.Hook, error)

New constructs a new instance of the privtrak middleware.

Types

type Config

type Config struct {
	BatchSize                   uint          `yaml:"batch_size"`
	ShardCount                  uint          `yaml:"shard_count"`
	GCInterval                  time.Duration `yaml:"gc_interval"`
	PrometheusReportingInterval time.Duration `yaml:"prometheus_reporting_interval"`
	PeerLifetime                time.Duration `yaml:"peer_lifetime"`
}

A Config holds the configuration for the privtrak middleware.

func (Config) LogFields

func (cfg Config) LogFields() log.Fields

LogFields renders the current config as a set of Logrus fields.

type DeltaHandler

type DeltaHandler interface {
	// HandleDeltas handles a batch of deltas.
	// This operation is assumed to be atomic:
	//   if no error is returned, the deltas are assumed to be persisted and
	//     thus deleted from this middleware
	//   if an error is returned, the deltas are kept in the middleware and the
	//     error will be returned on the announce route. We will attempt to
	//     flush the deltas again next time, or during routine garbage
	//     collection.
	HandleDeltas([]StatDelta) error
}

A DeltaHandler handles batches of stat-deltas.

type ID

type ID [16]byte

An ID uniquely identifies a user.

type StatDelta

type StatDelta struct {
	User      ID
	InfoHash  bittorrent.InfoHash
	DeltaUp   int64
	DeltaDown int64
	Event     bittorrent.Event
	Reported  time.Time
}

A StatDelta is a delta for a user's stats. It contains the infohash, the event the client provided with the announce and a timestamp.

func (StatDelta) LogFields

func (s StatDelta) LogFields() log.Fields

LogFields implements log.Fielder for StatDeltas.

type UserIdentifier

type UserIdentifier interface {
	DeriveID(*bittorrent.AnnounceRequest) (ID, error)
}

A UserIdentifier identifies an ID from a request.