got

package module
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2020 License: MIT Imports: 25 Imported by: 0

README

Got.

Simple and fast concurrent downloader.

Installation ❘ CLI Usage ❘ Module Usage ❘ License

Tests

Comparison

Comparison in cloud server:


[root@centos-nyc-12 ~]# time got -o /tmp/test -c 20 http://www.ovh.net/files/1Gio.dat
URL: http://www.ovh.net/files/1Gio.dat done!

real    0m8.832s
user    0m0.203s
sys 0m3.176s


[root@centos-nyc-12 ~]# time curl http://www.ovh.net/files/1Gio.dat --output /tmp/test1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
								 Dload  Upload   Total   Spent    Left  Speed
100 1024M  100 1024M    0     0  35.6M      0  0:00:28  0:00:28 --:--:-- 34.4M

real    0m28.781s
user    0m0.379s
sys 0m1.970s

Installation

Download and install the latest release:
# go to tmp dir.
cd /tmp

# Download latest version.
curl -sfL https://git.io/getgot | sh

# Make the binary executable.
chmod +x /tmp/bin/got

# Move the binary to your PATH
sudo mv /tmp/bin/got /usr/bin/got
Or Go ahead compile it yourself:
go get github.com/melbahja/got/cmd/got
Or from the AUR

Install got for the latest release version or got-git for the latest development version.

Note: these packages are not maintained by melbahja

Command Line Tool Usage

Simple usage:
got https://example.com/file.mp4
You can specify destination path:
got -o /path/to/save https://example.com/file.mp4
You can download multiple URLs and save them to directory:
got --dir /path/to/dir https://example.com/file.mp4 https://example.com/file2.mp4
You can download multiple URLs from a file:
got --dir /path/to/dir -f urls.txt

You can pipe multiple URLs:

cat urls.txt | got --dir /path/to/dir
Docs for available flags:
got help

Module Usage

You can use Got to download large files in your go code, the usage is simple as the CLI tool:

package main

import "github.com/melbahja/got"

func main() {

	g := got.New()

	err := g.Download("http://localhost/file.ext", "/path/to/save")

	if err != nil {
		// ..
	}
}

For more see PkgDocs.

How It Works?

Got takes advantage of the HTTP range requests support in servers RFC 7233, if the server supports partial content Got split the file into chunks, then starts downloading and merging the chunks into the destinaton file concurrently.

License

Got is provided under the MIT License © Mohammed El Bahja.

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source 
var ChunkPool = &sync.Pool{
	New: func() interface{} {
		return new(Chunk)
	},
}

ChunkPool helps in multi *Download files.

View Source 
var DefaultClient = &http.Client{
	Transport: &http.Transport{
		MaxIdleConns:        10,
		IdleConnTimeout:     30 * time.Second,
		TLSHandshakeTimeout: 5 * time.Second,
		Proxy:               http.ProxyFromEnvironment,
	},
}

DefaultClient is the default http client for got requests.

View Source 
var DefaultFileName = "got.output"

DefaultFileName is the fallback name for GetFilename.

View Source 
var ErrDownloadAborted = errors.New("Operation aborted")

ErrDownloadAborted - When download is aborted by the OS before it is completed, ErrDownloadAborted will be triggered

View Source 
var HeaderSlice []GotHeader
View Source 
var UserAgent = "Got/1.0"

UserAgent is the default Got user agent to send http requests.

Functions

func CliDownload 

func CliDownload(args []string)

func GetFilename 

func GetFilename(URL string) string

GetFilename it returns default file name from a URL.

func NewRequest 

func NewRequest(ctx context.Context, method, URL string, header []GotHeader) (req *http.Request, err error)

NewRequest returns a new http.Request and error if any.

Types

type Chunk 

type Chunk struct {

	// Chunk start pos.
	Start uint64

	// Chunk end pos.
	End uint64

	// Path name where this chunk downloaded.
	Path string

	// Done to check is this chunk downloaded.
	Done chan struct{}
}

Chunk is a partial content range.

type Download 

type Download struct {
	Client *http.Client

	Concurrency uint

	URL, Dir, Dest string

	Interval, ChunkSize, MinChunkSize, MaxChunkSize uint64

	Header []GotHeader

	StopProgress bool
	// contains filtered or unexported fields
}

Download holds downloadable file config and infos.

Example 
// Just for testing
destPath := createTemp()
defer clean(destPath)

ctx := context.Background()

dl := got.NewDownload(ctx, testUrl, destPath)

// Init
if err := dl.Init(); err != nil {
	fmt.Println(err)
}

// Start download
if err := dl.Start(); err != nil {
	fmt.Println(err)
}

fmt.Println("Done")

Output:

Done

func NewDownload 

func NewDownload(ctx context.Context, URL, dest string) *Download

NewDownload returns new *Download with context.

func (Download) AvgSpeed 

func (d Download) AvgSpeed() uint64

AvgSpeed returns average download speed.

func (Download) Context 

func (d Download) Context() context.Context

Context returns download context.

func (*Download) DownloadChunk 

func (d *Download) DownloadChunk(c Chunk, dest *os.File) error

DownloadChunk downloads a file chunk.

func (Download) GetInfo 

func (d Download) GetInfo() (*Info, error)

GetInfo returns URL info, and error if any.

func (*Download) Init 

func (d *Download) Init() (err error)

Init set defaults and split file into chunks and gets Info, you should call Init before Start

func (Download) IsRangeable 

func (d Download) IsRangeable() bool

IsRangeable returns file server partial content support state.

func (*Download) Name 

func (d *Download) Name() string

Name returns the downloaded file path.

func (*Download) RunProgress 

func (d *Download) RunProgress(fn ProgressFunc)

RunProgress runs ProgressFunc based on Interval and updates lastSize.

func (Download) Size 

func (d Download) Size() uint64

Size returns downloaded size.

func (Download) Speed 

func (d Download) Speed() uint64

Speed returns download speed.

func (*Download) Start 

func (d *Download) Start() (err error)

Start downloads the file chunks, and merges them.

func (Download) TotalCost 

func (d Download) TotalCost() time.Duration

TotalCost returns download duration.

func (Download) TotalSize 

func (d Download) TotalSize() uint64

TotalSize returns file total size (0 if unknown).

func (*Download) Write 

func (d *Download) Write(b []byte) (int, error)

Write updates progress size.

type Got 

type Got struct {
	ProgressFunc

	Client *http.Client
	// contains filtered or unexported fields
}

Got holds got download config.

Example 
// Just for testing
destPath := createTemp()
defer clean(destPath)

g := got.New()

err := g.Download(testUrl, destPath)

if err != nil {
	log.Fatal(err)
	return
}

fmt.Println("done")

Output:

done
Example (WithContext) 
// Just for testing
destPath := createTemp()
defer clean(destPath)

ctx := context.Background()

g := got.NewWithContext(ctx)

err := g.Download(testUrl, destPath)

if err != nil {
	log.Fatal(err)
	return
}

fmt.Println("done")

Output:

done

func New 

func New() *Got

New returns new *Got with default context and client.

func NewWithContext 

func NewWithContext(ctx context.Context) *Got

NewWithContext wants Context and returns *Got with default http client.

func (Got) Do 

func (g Got) Do(dl *Download) error

Do inits and runs ProgressFunc if set and starts the Download.

func (Got) Download 

func (g Got) Download(URL, dest string) error

Download creates *Download item and runs it.

type GotHeader 

type GotHeader struct {
	Key   string
	Value string
}

type Info 

type Info struct {
	Size      uint64
	Name      string
	Rangeable bool
}

Info holds downloadable file info.

type ProgressFunc 

type ProgressFunc func(d *Download)

ProgressFunc to show progress state, called by RunProgress based on interval.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.