downloader

package
v0.0.0-...-1f8a15b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 12, 2023 License: Apache-2.0 Imports: 47 Imported by: 0

Documentation

Overview

Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.

The filepath package uses either forward slashes or backslashes, depending on the operating system. To process paths such as URLs that always use forward slashes regardless of the operating system, see the path package.

Index

Constants

View Source 
const (
	Separator     = os.PathSeparator
	ListSeparator = os.PathListSeparator
)

Variables

View Source 
var SkipDir error = fs.SkipDir

SkipDir is used as a return value from WalkFuncs to indicate that the directory named in the call is to be skipped. It is not returned as an error by any function.

View Source 
var Trackers = [][]string{
	udpOrHttpTrackers,
}

Trackers - break down by priority tier

Functions

func AllTorrentPaths 

func AllTorrentPaths(dirs datadir.Dirs) ([]string, error)

func AllTorrentSpecs 

func AllTorrentSpecs(dirs datadir.Dirs) (res []*torrent.TorrentSpec, err error)

func BuildTorrentFilesIfNeed 

func BuildTorrentFilesIfNeed(ctx context.Context, dirs datadir.Dirs) error

BuildTorrentFilesIfNeed - create .torrent files from .seg files (big IO) - if .seg files were added manually

func BuildTorrentIfNeed 

func BuildTorrentIfNeed(ctx context.Context, fName, root string) (torrentFilePath string, err error)

func Clean 

func Clean(path string) string

Clean returns the shortest path name equivalent to path by purely lexical processing. It applies the following rules iteratively until no further processing can be done:

  1. Replace multiple Separator elements with a single one.
  2. Eliminate each . path name element (the current directory).
  3. Eliminate each inner .. path name element (the parent directory) along with the non-.. element that precedes it.
  4. Eliminate .. elements that begin a rooted path: that is, replace "/.." by "/" at the beginning of a path, assuming Separator is '/'.

The returned path ends in a slash only if it represents a root directory, such as "/" on Unix or `C:\` on Windows.

Finally, any occurrences of slash are replaced by Separator.

If the result of this process is an empty string, Clean returns the string ".".

See also Rob Pike, “Lexical File Names in Plan 9 or Getting Dot-Dot Right,” https://9p.io/sys/doc/lexnames.html

func CreateMetaInfo 

func CreateMetaInfo(info *metainfo.Info, mi *metainfo.MetaInfo) (*metainfo.MetaInfo, error)

func CreateTorrentFileFromInfo 

func CreateTorrentFileFromInfo(root string, info *metainfo.Info, mi *metainfo.MetaInfo) (err error)

func CreateTorrentFileIfNotExists 

func CreateTorrentFileIfNotExists(root string, info *metainfo.Info, mi *metainfo.MetaInfo) error

func CreateTorrentFromMetaInfo 

func CreateTorrentFromMetaInfo(root string, info *metainfo.Info, mi *metainfo.MetaInfo) error

func FromSlash 

func FromSlash(path string) string

FromSlash returns the result of replacing each slash ('/') character in path with a separator character. Multiple slashes are replaced by multiple separators.

func IsAbs 

func IsAbs(path string) bool

IsAbs reports whether the path is absolute.

func IsLocal deprecated 

func IsLocal(path string) bool

Deprecated: use `filepath.IsLocal` after drop go1.19 support

func Join 

func Join(elem ...string) string

Join joins any number of path elements into a single path, separating them with an OS specific Separator. Empty elements are ignored. The result is Cleaned. However, if the argument list is empty or all its elements are empty, Join returns an empty string. On Windows, the result will only be a UNC path if the first non-empty element is a UNC path.

func NewMdbxPieceCompletion 

func NewMdbxPieceCompletion(db kv.RwDB) (ret storage.PieceCompletion, err error)

func Proto2InfoHash 

func Proto2InfoHash(in *prototypes.H160) metainfo.Hash

func VerifyFile 

func VerifyFile(ctx context.Context, t *torrent.Torrent, completePieces *atomic.Uint64) error

Types

type AggStats 

type AggStats struct {
	MetadataReady, FilesTotal int32
	PeersUnique               int32
	ConnectionsTotal          uint64

	Completed bool
	Progress  float32

	BytesCompleted, BytesTotal     uint64
	DroppedCompleted, DroppedTotal uint64

	BytesDownload, BytesUpload uint64
	UploadRate, DownloadRate   uint64
}

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader - component which downloading historical files. Can use BitTorrent, or other protocols

func New 

func New(ctx context.Context, cfg *downloadercfg.Cfg, dirs datadir.Dirs, logger log.Logger, verbosity log.Lvl, discover bool) (*Downloader, error)
func (d *Downloader) AddInfoHashAsMagnetLink(ctx context.Context, infoHash metainfo.Hash, name string) error

func (*Downloader) AddNewSeedableFile 

func (d *Downloader) AddNewSeedableFile(ctx context.Context, name string) error

AddNewSeedableFile decides what we do depending on wether we have the .seg file or the .torrent file have .torrent no .seg => get .seg file from .torrent have .seg no .torrent => get .torrent from .seg

func (*Downloader) BuildTorrentFilesIfNeed 

func (d *Downloader) BuildTorrentFilesIfNeed(ctx context.Context) error

func (*Downloader) Close 

func (d *Downloader) Close()

func (*Downloader) MainLoopInBackground 

func (d *Downloader) MainLoopInBackground(silent bool)

func (*Downloader) PeerID 

func (d *Downloader) PeerID() []byte

func (*Downloader) ReCalcStats 

func (d *Downloader) ReCalcStats(interval time.Duration)

func (*Downloader) SnapDir 

func (d *Downloader) SnapDir() string

func (*Downloader) Stats 

func (d *Downloader) Stats() AggStats

func (*Downloader) StopSeeding 

func (d *Downloader) StopSeeding(hash metainfo.Hash) error

func (*Downloader) TorrentClient 

func (d *Downloader) TorrentClient() *torrent.Client

func (*Downloader) VerifyData 

func (d *Downloader) VerifyData(ctx context.Context, onlyFiles []string) error

type GrpcServer 

type GrpcServer struct {
	proto_downloader.UnimplementedDownloaderServer
	// contains filtered or unexported fields
}

func NewGrpcServer 

func NewGrpcServer(d *Downloader) (*GrpcServer, error)

func (*GrpcServer) Delete 

func (s *GrpcServer) Delete(ctx context.Context, request *proto_downloader.DeleteRequest) (*emptypb.Empty, error)

Delete - stop seeding, remove file, remove .torrent

func (*GrpcServer) Download 

func (s *GrpcServer) Download(ctx context.Context, request *proto_downloader.DownloadRequest) (*emptypb.Empty, error)

Download - create new .torrent ONLY if initialSync, everything else Erigon can generate by itself

func (*GrpcServer) Stats 

func (s *GrpcServer) Stats(ctx context.Context, request *proto_downloader.StatsRequest) (*proto_downloader.StatsReply, error)

func (*GrpcServer) Verify 

func (s *GrpcServer) Verify(ctx context.Context, request *proto_downloader.VerifyRequest) (*emptypb.Empty, error)

type WalkFunc 

type WalkFunc func(path string, info fs.FileInfo, err error) error

WalkFunc is the type of the function called by Walk to visit each file or directory.

The path argument contains the argument to Walk as a prefix. That is, if Walk is called with root argument "dir" and finds a file named "a" in that directory, the walk function will be called with argument "dir/a".

The directory and file are joined with Join, which may clean the directory name: if Walk is called with the root argument "x/../dir" and finds a file named "a" in that directory, the walk function will be called with argument "dir/a", not "x/../dir/a".

The info argument is the fs.FileInfo for the named path.

The error result returned by the function controls how Walk continues. If the function returns the special value SkipDir, Walk skips the current directory (path if info.IsDir() is true, otherwise path's parent directory). If the function returns the special value SkipAll, Walk skips all remaining files and directories. Otherwise, if the function returns a non-nil error, Walk stops entirely and returns that error.

The err argument reports an error related to path, signaling that Walk will not walk into that directory. The function can decide how to handle that error; as described earlier, returning the error will cause Walk to stop walking the entire tree.

Walk calls the function with a non-nil err argument in two cases.

First, if an os.Lstat on the root directory or any directory or file in the tree fails, Walk calls the function with path set to that directory or file's path, info set to nil, and err set to the error from os.Lstat.

Second, if a directory's Readdirnames method fails, Walk calls the function with path set to the directory's path, info, set to an fs.FileInfo describing the directory, and err set to the error from Readdirnames.

type WebSeeds 

type WebSeeds struct {
	// contains filtered or unexported fields
}

WebSeeds - allow use HTTP-based infrastrucutre to support Bittorrent network it allows download .torrent files and data files from trusted url's (for example: S3 signed url)

func (*WebSeeds) ByFileName 

func (d *WebSeeds) ByFileName(name string) (metainfo.UrlList, bool)

func (*WebSeeds) Discover 

func (d *WebSeeds) Discover(ctx context.Context, s3tokens []string, urls []*url.URL, files []string, rootDir string)

func (*WebSeeds) Len 

func (d *WebSeeds) Len() int

func (*WebSeeds) TorrentUrls 

func (d *WebSeeds) TorrentUrls() snaptype.TorrentUrls

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.