torrent

package module
v0.0.0-...-eed0743 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 10, 2017 License: MPL-2.0 Imports: 49 Imported by: 0

README

torrent

CircleCI GoDoc

This repository implements BitTorrent-related packages and command-line utilities in Go. The emphasis is on use as a library from other projects. It's been used 24/7 in production by a downstream, private service since late 2014.

There is support for protocol encryption, DHT, PEX, uTP, and various extensions. See the package documentation for a more complete list. There are several data storage backends provided: blob, file, and mmap, and you can write your own, such as to store data on S3, or in a database. You can use the provided binaries in ./cmd, or use package torrent as a library for your own applications.

Many of the sub-packages can be used for other purposes: bencode, and tracker, in particular.

Installation

Install the library package with go get github.com/anacrolix/torrent, or the provided cmds with go get github.com/anacrolix/torrent/cmd/....

Library example

There is a small example in the package documentation.

Other public projects using torrent

Commands

Here I'll describe what some of the provided commands in ./cmd do.

Note that the godo command which is invoked in the following examples builds and executes a Go import path, like go run. It's easier to use this convention than to spell out the install/invoke cycle for every single example.

torrent

Downloads torrents from the command-line. This first example does not use godo.

$ go get github.com/anacrolix/torrent/cmd/torrent
# Now 'torrent' should be in $GOPATH/bin, which should be in $PATH.
$ torrent 'magnet:?xt=urn:btih:KRWPCX3SJUM4IMM4YF5RPHL6ANPYTQPU'
ubuntu-14.04.2-desktop-amd64.iso [===================================================================>]  99% downloading (1.0 GB/1.0 GB)
2015/04/01 02:08:20 main.go:137: downloaded ALL the torrents
$ md5sum ubuntu-14.04.2-desktop-amd64.iso
1b305d585b1918f297164add46784116  ubuntu-14.04.2-desktop-amd64.iso
$ echo such amaze
wow
torrentfs

torrentfs mounts a FUSE filesystem at -mountDir. The contents are the torrents described by the torrent files and magnet links at -metainfoDir. Data for read requests is fetched only as required from the torrent network, and stored at -downloadDir.

$ mkdir mnt torrents
$ godo github.com/anacrolix/torrent/cmd/torrentfs -mountDir=mnt -metainfoDir=torrents &
$ cd torrents
$ wget http://releases.ubuntu.com/14.04.2/ubuntu-14.04.2-desktop-amd64.iso.torrent
$ cd ..
$ ls mnt
ubuntu-14.04.2-desktop-amd64.iso
$ pv mnt/ubuntu-14.04.2-desktop-amd64.iso | md5sum
996MB 0:04:40 [3.55MB/s] [========================================>] 100%
1b305d585b1918f297164add46784116  -
torrent-magnet

Creates a magnet link from a torrent file. Note the extracted trackers, display name, and info hash.

$ godo github.com/anacrolix/torrent/cmd/torrent-magnet < ubuntu-14.04.2-desktop-amd64.iso.torrent
magnet:?xt=urn:btih:546cf15f724d19c4319cc17b179d7e035f89c1f4&dn=ubuntu-14.04.2-desktop-amd64.iso&tr=http%3A%2F%2Ftorrent.ubuntu.com%3A6969%2Fannounce&tr=http%3A%2F%2Fipv6.torrent.ubuntu.com%3A6969%2Fannounce

Documentation

Overview

Package torrent implements a torrent client. Goals include:

  • Configurable data storage, such as file, mmap, and piece-based.
  • Downloading on demand: torrent.Reader will request only the data required to satisfy Reads, which is ideal for streaming and torrentfs.

BitTorrent features implemented include:

  • Protocol obfuscation
  • DHT
  • uTP
  • PEX
  • Magnet links
  • IP Blocklists
  • Some IPv6
  • HTTP and UDP tracker clients
  • BEPs:
  • 3: Basic BitTorrent protocol
  • 5: DHT
  • 6: Fast Extension (have all/none only)
  • 7: IPv6 Tracker Extension
  • 9: ut_metadata
  • 10: Extension protocol
  • 11: PEX
  • 12: Multitracker metadata extension
  • 15: UDP Tracker Protocol
  • 20: Peer ID convention ("-GTnnnn-")
  • 23: Tracker Returns Compact Peer Lists
  • 27: Private torrents
  • 29: uTorrent transport protocol
  • 41: UDP Tracker Protocol Extensions
  • 42: DHT Security extension
  • 43: Read-only DHT Nodes
Example 
package main

import (
	"log"

	"github.com/anacrolix/torrent"
)

func main() {
	c, _ := torrent.NewClient(nil)
	defer c.Close()
	t, _ := c.AddMagnet("magnet:?xt=urn:btih:ZOCMZQIPFFW7OLLMIC5HUB6BPCSDEOQU")
	<-t.GotInfo()
	t.DownloadAll()
	c.WaitAll()
	log.Print("ermahgerd, torrent downloaded")
}

Output:
Example (FileReader) 
package main

import (
	"github.com/anacrolix/missinggo"

	"github.com/anacrolix/torrent"
)

func main() {
	var (
		t *torrent.Torrent
		f torrent.File
	)
	r := t.NewReader()
	defer r.Close()
	// Access the parts of the torrent pertaining to f. Data will be
	// downloaded as required, per the configuration of the torrent.Reader.
	_ = missinggo.NewSectionReadSeeker(r, f.Offset(), f.Length())
}

Output:

Index

Examples

Constants

View Source 
const (
	ExtensionBitDHT      = 0  // http://www.bittorrent.org/beps/bep_0005.html
	ExtensionBitExtended = 20 // http://www.bittorrent.org/beps/bep_0010.html
	ExtensionBitFast     = 2  // http://www.bittorrent.org/beps/bep_0006.html
)
View Source 
const (
	PiecePriorityNone      piecePriority = iota // Not wanted.
	PiecePriorityNormal                         // Wanted.
	PiecePriorityReadahead                      // May be required soon.
	// Succeeds a piece where a read occurred. Currently the same as Now, apparently due to issues with caching.
	PiecePriorityNext
	PiecePriorityNow // A Reader is reading in this piece.
)

Variables

View Source 
var DefaultHTTPClient = &http.Client{
	Timeout: time.Second * 15,
	Transport: &http.Transport{
		Dial: (&net.Dialer{
			Timeout: 15 * time.Second,
		}).Dial,
		TLSHandshakeTimeout: 15 * time.Second,
		TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
	},
}

Functions

func NewUtpSocket 

func NewUtpSocket(network, addr string) (utpSocket, error)

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Clients contain zero or more Torrents. A Client manages a blocklist, the TCP/UDP protocol ports, and DHT as desired.

func NewClient 

func NewClient(cfg *Config) (cl *Client, err error)

Creates a new client.

func (*Client) AddDHTNodes 

func (cl *Client) AddDHTNodes(nodes []string)

func (*Client) AddMagnet 

func (cl *Client) AddMagnet(uri string) (T *Torrent, err error)

func (*Client) AddTorrent 

func (cl *Client) AddTorrent(mi *metainfo.MetaInfo) (T *Torrent, err error)

func (*Client) AddTorrentFromFile 

func (cl *Client) AddTorrentFromFile(filename string) (T *Torrent, err error)

func (*Client) AddTorrentInfoHash 

func (cl *Client) AddTorrentInfoHash(infoHash metainfo.Hash) (t *Torrent, new bool)

func (*Client) AddTorrentInfoHashWithStorage 

func (cl *Client) AddTorrentInfoHashWithStorage(infoHash metainfo.Hash, specStorage storage.ClientImpl) (t *Torrent, new bool)

Adds a torrent by InfoHash with a custom Storage implementation. If the torrent already exists then this Storage is ignored and the existing torrent returned with `new` set to `false`

func (*Client) AddTorrentSpec 

func (cl *Client) AddTorrentSpec(spec *TorrentSpec) (t *Torrent, new bool, err error)

Add or merge a torrent spec. If the torrent is already present, the trackers will be merged with the existing ones. If the Info isn't yet known, it will be set. The display name is replaced if the new spec provides one. Returns new if the torrent wasn't already in the client. Note that any `Storage` defined on the spec will be ignored if the torrent is already present (i.e. `new` return value is `true`)

func (*Client) BadPeerIPs 

func (cl *Client) BadPeerIPs() []string

func (*Client) Close 

func (cl *Client) Close()

Stops the client. All connections to peers are closed and all activity will come to a halt.

func (*Client) DHT 

func (cl *Client) DHT() *dht.Server

func (*Client) IPBlockList 

func (cl *Client) IPBlockList() iplist.Ranger

func (*Client) ListenAddr 

func (cl *Client) ListenAddr() net.Addr

func (*Client) PeerID 

func (cl *Client) PeerID() [20]byte

func (*Client) SetIPBlockList 

func (cl *Client) SetIPBlockList(list iplist.Ranger)

func (*Client) Torrent 

func (cl *Client) Torrent(ih metainfo.Hash) (t *Torrent, ok bool)

Returns a handle to the given torrent, if it's present in the client.

func (*Client) Torrents 

func (cl *Client) Torrents() []*Torrent

Returns handles to all the torrents loaded in the Client.

func (*Client) WaitAll 

func (cl *Client) WaitAll() bool

Returns true when all torrents are completely downloaded and false if the client is stopped before that.

func (*Client) WriteStatus 

func (cl *Client) WriteStatus(_w io.Writer)

Writes out a human readable status of the client, such as for writing to a HTTP status page.

type Config 

type Config struct {
	// Store torrent file data in this directory unless TorrentDataOpener is
	// specified.
	DataDir string `long:"data-dir" description:"directory to store downloaded torrent data"`
	// The address to listen for new uTP and TCP bittorrent protocol
	// connections. DHT shares a UDP socket with uTP unless configured
	// otherwise.
	ListenAddr string `long:"listen-addr" value-name:"HOST:PORT"`
	// Don't announce to trackers. This only leaves DHT to discover peers.
	DisableTrackers bool `long:"disable-trackers"`
	DisablePEX      bool `long:"disable-pex"`
	// Don't create a DHT.
	NoDHT bool `long:"disable-dht"`
	// Overrides the default DHT configuration.
	DHTConfig dht.ServerConfig

	// Never send chunks to peers.
	NoUpload bool `long:"no-upload"`
	// Disable uploading even when it isn't fair.
	DisableAggressiveUpload bool `long:"disable-aggressive-upload"`
	// Upload even after there's nothing in it for us. By default uploading is
	// not altruistic, we'll upload slightly more than we download from each
	// peer.
	Seed bool `long:"seed"`
	// Events are data bytes sent in pieces. The burst must be large enough to
	// fit a whole chunk, which is usually 16 KiB.
	UploadRateLimiter *rate.Limiter
	// The events are bytes read from connections. The burst must be bigger
	// than the largest Read performed on a Conn minus one. This is likely to
	// be the larger of the main read loop buffer (~4096), and the requested
	// chunk size (~16KiB).
	DownloadRateLimiter *rate.Limiter

	// User-provided Client peer ID. If not present, one is generated automatically.
	PeerID string
	// For the bittorrent protocol.
	DisableUTP bool
	// For the bittorrent protocol.
	DisableTCP bool `long:"disable-tcp"`
	// Called to instantiate storage for each added torrent. Builtin backends
	// are in the storage package. If not set, the "file" implementation is
	// used.
	DefaultStorage storage.ClientImpl

	EncryptionPolicy

	IPBlocklist iplist.Ranger
	DisableIPv6 bool `long:"disable-ipv6"`
	// Perform logging and any other behaviour that will help debug.
	Debug bool `help:"enable debugging"`

	// HTTP client used to query the tracker endpoint. Default is DefaultHTTPClient
	HTTP *http.Client
	// Updated occasionally to when there's been some changes to client
	// behaviour in case other clients are assuming anything of us. See also
	// `bep20`.
	ExtendedHandshakeClientVersion string // default  "go.torrent dev 20150624"
	// Peer ID client identifier prefix. We'll update this occasionally to
	// reflect changes to client behaviour that other clients may depend on.
	// Also see `extendedHandshakeClientVersion`.
	Bep20 string // default "-GT0001-"

	NominalDialTimeout         time.Duration // default  time.Second * 30
	MinDialTimeout             time.Duration // default  5 * time.Second
	EstablishedConnsPerTorrent int           // default 80
	HalfOpenConnsPerTorrent    int           // default  80
	TorrentPeersHighWater      int           // default 200
	TorrentPeersLowWater       int           // default 50

	// Limit how long handshake can take. This is to reduce the lingering
	// impact of a few bad apples. 4s loses 1% of successful handshakes that
	// are obtained with 60s timeout, and 5% of unsuccessful handshakes.
	HandshakesTimeout time.Duration // default  20 * time.Second
}

Override Client defaults.

type ConnStats 

type ConnStats struct {
	// Torrent "piece" messages, or data chunks.
	ChunksWritten int64 // Num piece messages sent.
	ChunksRead    int64
	// Total bytes on the wire. Includes handshakes and encryption.
	BytesWritten int64
	BytesRead    int64
	// Data bytes, actual torrent data.
	DataBytesWritten int64
	DataBytesRead    int64
}

type EncryptionPolicy 

type EncryptionPolicy struct {
	DisableEncryption  bool
	ForceEncryption    bool // Don't allow unobfuscated connections.
	PreferNoEncryption bool
}

type ExtensionBit 

type ExtensionBit uint

type File 

type File struct {
	// contains filtered or unexported fields
}

Provides access to regions of torrent data that correspond to its files.

func (*File) Cancel 

func (f *File) Cancel()

func (*File) DisplayPath 

func (f *File) DisplayPath() string

The relative file path for a multi-file torrent, and the torrent name for a single-file torrent.

func (*File) Download 

func (f *File) Download()

Requests that all pieces containing data in the file be downloaded.

func (File) FileInfo 

func (f File) FileInfo() metainfo.FileInfo

func (*File) Length 

func (f *File) Length() int64

func (*File) Offset 

func (f *File) Offset() int64

Data for this file begins this far into the torrent.

func (File) Path 

func (f File) Path() string

func (*File) PrioritizeRegion 

func (f *File) PrioritizeRegion(off, len int64)

Requests that torrent pieces containing bytes in the given region of the file be downloaded.

func (*File) State 

func (f *File) State() (ret []FilePieceState)

Returns the state of pieces in this file.

func (*File) Torrent 

func (f *File) Torrent() *Torrent

type FilePieceState 

type FilePieceState struct {
	Bytes int64 // Bytes within the piece that are part of this File.
	PieceState
}

type Handle 

type Handle interface {
	io.Reader
	io.Seeker
	io.Closer
	io.ReaderAt
}

A file-like handle to some torrent data resource.

type Peer 

type Peer struct {
	Id     [20]byte
	IP     net.IP
	Port   int
	Source peerSource
	// Peer is known to support encryption.
	SupportsEncryption bool
}

type Piece 

type Piece struct {
	// contains filtered or unexported fields
}

func (*Piece) Info 

func (p *Piece) Info() metainfo.Piece

func (*Piece) Storage 

func (p *Piece) Storage() storage.Piece

func (*Piece) String 

func (p *Piece) String() string

func (*Piece) VerifyData 

func (p *Piece) VerifyData()

type PieceState 

type PieceState struct {
	Priority piecePriority
	// The piece is available in its entirety.
	Complete bool
	// The piece is being hashed, or is queued for hash.
	Checking bool
	// Some of the piece has been obtained.
	Partial bool
}

The current state of a piece.

type PieceStateChange 

type PieceStateChange struct {
	Index int
	PieceState
}

type PieceStateRun 

type PieceStateRun struct {
	PieceState
	Length int // How many consecutive pieces have this state.
}

Represents a series of consecutive pieces with the same state.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Accesses Torrent data via a Client. Reads block until the data is available. Seeks and readahead also drive Client behaviour.

func (*Reader) Close 

func (r *Reader) Close() error

func (*Reader) CurrentPos 

func (r *Reader) CurrentPos() int64

Return reader's current position.

func (*Reader) Read 

func (r *Reader) Read(b []byte) (n int, err error)

func (*Reader) ReadContext 

func (r *Reader) ReadContext(ctx context.Context, b []byte) (n int, err error)

func (*Reader) Seek 

func (r *Reader) Seek(off int64, whence int) (ret int64, err error)

func (*Reader) SetNonResponsive 

func (r *Reader) SetNonResponsive()

Disable responsive mode.

func (*Reader) SetReadahead 

func (r *Reader) SetReadahead(readahead int64)

Configure the number of bytes ahead of a read that should also be prioritized in preparation for further reads.

func (*Reader) SetResponsive 

func (r *Reader) SetResponsive()

Don't wait for pieces to complete and be verified. Read calls return as soon as they can when the underlying chunks become available.

func (*Reader) Torrent 

func (r *Reader) Torrent() *Torrent

type Torrent 

type Torrent struct {
	// contains filtered or unexported fields
}

Maintains state of torrent within a Client.

func (*Torrent) AddPeers 

func (t *Torrent) AddPeers(pp []Peer)

func (*Torrent) AddTrackers 

func (t *Torrent) AddTrackers(announceList [][]string)

func (*Torrent) BytesCompleted 

func (t *Torrent) BytesCompleted() int64

Number of bytes of the entire torrent we have completed.

func (*Torrent) BytesMissing 

func (t *Torrent) BytesMissing() int64

func (*Torrent) CancelPieces 

func (t *Torrent) CancelPieces(begin, end int)

func (*Torrent) Closed 

func (t *Torrent) Closed() <-chan struct{}

Returns a channel that is closed when the Torrent is closed.

func (*Torrent) DownloadAll 

func (t *Torrent) DownloadAll()

Marks the entire torrent for download. Requires the info first, see GotInfo.

func (*Torrent) DownloadPieces 

func (t *Torrent) DownloadPieces(begin, end int)

func (*Torrent) Drop 

func (t *Torrent) Drop()

Drop the torrent from the client, and close it. It's always safe to do this. No data corruption can, or should occur to either the torrent's data, or connected peers.

func (*Torrent) Files 

func (t *Torrent) Files() (ret []File)

Returns handles to the files in the torrent. This requires the metainfo is available first.

func (*Torrent) GotInfo 

func (t *Torrent) GotInfo() <-chan struct{}

Returns a channel that is closed when the info (.Info()) for the torrent has become available.

func (*Torrent) Info 

func (t *Torrent) Info() *metainfo.Info

Returns the metainfo info dictionary, or nil if it's not yet available.

func (*Torrent) InfoHash 

func (t *Torrent) InfoHash() metainfo.Hash

The torrent's infohash. This is fixed and cannot change. It uniquely identifies a torrent.

func (*Torrent) KnownSwarm 

func (t *Torrent) KnownSwarm() (ks []Peer)

KnownSwarm returns the known subset of the peers in the Torrent's swarm, including active, pending, and half-open peers.

func (*Torrent) Length 

func (t *Torrent) Length() int64

The completed length of all the torrent data, in all its files. This is derived from the torrent info, when it is available.

func (*Torrent) Metainfo 

func (t *Torrent) Metainfo() metainfo.MetaInfo

Returns a run-time generated metainfo for the torrent that includes the info bytes and announce-list as currently known to the client.

func (*Torrent) Name 

func (t *Torrent) Name() string

The current working name for the torrent. Either the name in the info dict, or a display name given such as by the dn value in a magnet link, or "".

func (*Torrent) NewReader 

func (t *Torrent) NewReader() (ret *Reader)

Returns a Reader bound to the torrent's data. All read calls block until the data requested is actually available.

func (*Torrent) NumPieces 

func (t *Torrent) NumPieces() int

The number of pieces in the torrent. This requires that the info has been obtained first.

func (*Torrent) Piece 

func (t *Torrent) Piece(i int) *Piece

func (*Torrent) PieceBytesMissing 

func (t *Torrent) PieceBytesMissing(piece int) int64

Get missing bytes count for specific piece.

func (*Torrent) PieceState 

func (t *Torrent) PieceState(piece int) PieceState

func (*Torrent) PieceStateRuns 

func (t *Torrent) PieceStateRuns() []PieceStateRun

Returns the state of pieces of the torrent. They are grouped into runs of same state. The sum of the state run lengths is the number of pieces in the torrent.

func (*Torrent) Seeding 

func (t *Torrent) Seeding() bool

Returns true if the torrent is currently being seeded. This occurs when the client is willing to upload without wanting anything in return.

func (*Torrent) SetDisplayName 

func (t *Torrent) SetDisplayName(dn string)

Clobbers the torrent display name. The display name is used as the torrent name if the metainfo is not available.

func (*Torrent) SetInfoBytes 

func (t *Torrent) SetInfoBytes(b []byte) (err error)

func (*Torrent) SetMaxEstablishedConns 

func (t *Torrent) SetMaxEstablishedConns(max int) (oldMax int)

func (*Torrent) Stats 

func (t *Torrent) Stats() TorrentStats

func (*Torrent) String 

func (t *Torrent) String() string

func (*Torrent) SubscribePieceStateChanges 

func (t *Torrent) SubscribePieceStateChanges() *pubsub.Subscription

The subscription emits as (int) the index of pieces as their state changes. A state change is when the PieceState for a piece alters in value.

func (*Torrent) VerifyData 

func (t *Torrent) VerifyData()

type TorrentSpec 

type TorrentSpec struct {
	// The tiered tracker URIs.
	Trackers  [][]string
	InfoHash  metainfo.Hash
	InfoBytes []byte
	// The name to use if the Name field from the Info isn't available.
	DisplayName string
	// The chunk size to use for outbound requests. Defaults to 16KiB if not
	// set.
	ChunkSize int
	Storage   storage.ClientImpl
}

Specifies a new torrent for adding to a client. There are helpers for magnet URIs and torrent metainfo files.

func TorrentSpecFromMagnetURI 

func TorrentSpecFromMagnetURI(uri string) (spec *TorrentSpec, err error)

func TorrentSpecFromMetaInfo 

func TorrentSpecFromMetaInfo(mi *metainfo.MetaInfo) (spec *TorrentSpec)

type TorrentStats 

type TorrentStats struct {
	ConnStats // Aggregates stats over all connections past and present.

	ActivePeers   int
	HalfOpenPeers int
	PendingPeers  int
	TotalPeers    int
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
magnet-metainfo
Converts magnet URIs and info hashes into torrent metainfo files.
 Converts magnet URIs and info hashes into torrent metainfo files.
torrent
Downloads torrents from the command-line.
 Downloads torrents from the command-line.
torrent-create
torrent-infohash
torrent-magnet
torrent-metainfo-pprint
torrent-pick
Downloads torrents from the command-line.
 Downloads torrents from the command-line.
torrent-verify
torrentfs
Mounts a FUSE filesystem backed by torrents and magnet links.
 Mounts a FUSE filesystem backed by torrents and magnet links.
tracker-announce
internal
cmd/bloom-estimate
cmd/bloom-false-positives
testutil
Package iplist handles the P2P Plaintext Format described by https://en.wikipedia.org/wiki/PeerGuardian#P2P_plaintext_format.
 Package iplist handles the P2P Plaintext Format described by https://en.wikipedia.org/wiki/PeerGuardian#P2P_plaintext_format.
cmd/pack-blocklist
Takes P2P blocklist text format in stdin, and outputs the packed format from the iplist package.
 Takes P2P blocklist text format in stdin, and outputs the packed format from the iplist package.
Package logonce implements an io.Writer facade that only performs distinct writes.
 Package logonce implements an io.Writer facade that only performs distinct writes.
Package storage implements storage backends for package torrent.
 Package storage implements storage backends for package torrent.
dirwatch
Package dirwatch provides filesystem-notification based tracking of torrent info files and magnet URIs in a directory.
 Package dirwatch provides filesystem-notification based tracking of torrent info files and magnet URIs in a directory.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.