goup

package module
v0.0.0-...-8dfad59 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 20, 2024 License: MIT Imports: 40 Imported by: 1

README

goup

Utility to upload or download large files through HTTP in broken-point continuously.

This is done by splitting large file into small chunks; whenever the upload of a chunk fails, uploading is retried until the procedure completes. This allows uploads to automatically resume uploading after a network connection is lost, or program restarting either locally or to the server.

Features

  1. splitting large file into smaller chunks.
  2. uploading/downloading by HTTP.
  3. resume-able. chunk's hash will be checked before transfer.
  4. security data by AES-GCM based on PAKE.
  5. support download short path like goup -path /xx=/xx.zip, then the client can use http://127.0.0.1:2001/xx to download the xx.zip file.
API Method Req Content-Gulp Rsp Content-Gulp Other Headers Function
1. POST Filename 明文上传（文件作为 Body)
2. POST Session, Curve Curve, PAKE 生成会话秘钥
3. GET / Session, Range, Checksum Req: Content-Disposition 校验分块 checksum，返回 304 或 其它
4. POST / Session, Range, Salt Req: Content-Disposition 分块加密上传（加密分块作为 Body)
5. GET / HTML JS 上传页面 / 服务端文件列表（Accept: application/json 时）
6. GET / Session, Range, Checksum Range, Salt Rsp: Content-Type , Content-Disposition 分块加密下载
7. GET / Salt Rsp: Content-Type, Content-Length, Content-Disposition 明文下载
8. POST 明文上传（multipart-form)

Usage

$ goup -h
Usage of goup:
  -b    string Bearer token for client or server, auto for server to generate a random one
  -c    string Chunk size for client, unit MB (default 10)
  -t    int    Threads (go-routines) for client
  -f    string Upload file path for client
  -p    int    Listening port for server
  -r    string Rename to another filename
  -u    string Server upload url for client to connect to
  -P    string Password for PAKE
  -C    string Cipher AES256: AES-256 GCM, C20P1305: ChaCha20 Poly1305
  -v    bool   Show version
  1. Installation go install https://github.com/bingoohuang/ngg/goup
  2. At the server, goup
  3. At the client, goup -u http://a.b.c:2110/ -f 246.png
$ goup
2022/01/20 17:40:20 Listening on 2110
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 0-10485760/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 10485760-20971520/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 20971520-31457280/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 31457280-41943040/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 41943040-52428800/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 52428800-62914560/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 62914560-73400320/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 73400320-83886080/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 83886080-94371840/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 94371840-96894303/96894303

$ md5sum .goup-files/246.png
5849c383ab841e539673d50a49c3f39a  .goup-files/aDrive.dmg
$ sha256sum .goup-files/246.png
32677ca3a5b19ee5590a75d423b3fc2d754bd76900d50c26e0525681985189f8  .goup-files/aDrive.dmg

upload example:

$ goup -u http://127.0.0.1:2110/ -f 246.png
2022/01/20 17:44:08 Upload B821C49E4B1CBDBD started
40.00 MiB / 92.41 MiB [------------------>____________________] 43.29% ? p/s^C
$ diff -s /Users/bingoobjca/Downloads/goup/.goup-files/246.png 246.png
Binary files /Users/bingoobjca/Downloads/goup/.goup-files/246.png and 246.png differ
$ goup -u http://127.0.0.1:2110/ -f 246.png
2022/01/20 17:44:21 Upload 4739FE47D18ADF7F started
92.41 MiB / 92.41 MiB [---------------------------------------] 100.00% 121.54 MiB p/s
2022/01/20 17:44:21 Upload 4739FE47D18ADF7F completed
$ diff -s /Users/bingoobjca/Downloads/goup/.goup-files/246.png 246.png
Files /Users/bingoobjca/Downloads/goup/.goup-files/246.png and 246.png are identical
$ md5 246.png
MD5 (aDrive.dmg) = 5849c383ab841e539673d50a49c3f39a
$ sha256sum 246.png
32677ca3a5b19ee5590a75d423b3fc2d754bd76900d50c26e0525681985189f8  246.png

As you can see, even if you terminate the client uploading, you can resume uploading from the last breakpoint.

$ goup -u http://127.0.0.1:2110/246.png
2022/01/21 13:29:43 download started: .goup-files/246.png
92.41 MiB / 92.41 MiB [---------------------------------------] 100.00% 97.56 MiB p/s
2022/01/21 13:29:44 download complete: .goup-files/246.png

normal upload/download support

upload:

  1. Start the server without a chunk: goup -c0
  2. Test the upload: curl -F "file:=@a.mp4" localhost:2110

download:

  1. Start the server: goup
  2. Download
    1. by gurl: gurl :2110/246.png
    2. by curl: curl -LO 127.0.0.1:2110/246.png
    3. by goup disable chunk: goup -u :2110/246.png -c0

Resources

  1. SRP Go implementation of the Secure Remote Password (SRP) protocol.

    SRP is an authentication method that allows the use of user names and passwords over unencrypted channels without revealing the password to an eavesdropper. SRP also supplies a shared secret at the end of the authentication sequence that can be used to generate encryption keys.

  2. Encrypted-Content-Encoding HTTP Go implementation of Encrypted-Content-Encoding for HTTP (RFC 8188).

Documentation

Overview

Package goup contains golang upload/download server and client.

Index

Constants

View Source 
const (
	// Authorization is the header name for Authorization
	Authorization = "Authorization"
	// ContentDisposition is header name for Content-Disposition
	ContentDisposition = "Content-Disposition"
	// ContentType is the header name for Content-Type
	ContentType = "Content-Type"
	// ContentLength is the header name for Content-Length
	ContentLength = "Content-Length"
)

Variables

View Source 
var ErrMissingFile = errors.New("there is no uploaded file")

ErrMissingFile may be returned from FormFile when the is no uploaded file.

View Source 
var RootDir = "./.goup"

RootDir settings. When finished uploading with success files are stored inside it.

Functions

func Bearer 

func Bearer(token string, handle http.HandlerFunc) http.HandlerFunc

Bearer returns a Handler that authenticates via Bearer Auth. Writes a http.StatusUnauthorized if authentication fails.

func BearerTokenGenerate 

func BearerTokenGenerate() string

BearerTokenGenerate generates a random bearer token.

func CreateChunkReader 

func CreateChunkReader(fullPath string, partFrom, partTo uint64, limitRate uint64) (r io.ReadCloser, err error)

CreateChunkReader creates a chunk reader for the file.

func GetPartSize 

func GetPartSize(totalSize, chunkSize, idx uint64) uint64

GetPartSize get the part size of idx-th chunk.

func InitServer 

func InitServer() error

InitServer initializes the server.

func NewRewindableMultiReader 

func NewRewindableMultiReader(readers ...io.Reader) io.Reader

NewRewindableMultiReader returns a Reader that's the logical concatenation of the provided input readers. They're read sequentially. Once all inputs have returned EOF, Read will return EOF. If any of the readers return a non-nil, non-EOF error, Read will return that error.

func ParseFormFile 

func ParseFormFile(m *multipart.Form) (*multipart.FileHeader, error)

ParseFormFile returns the first file for the provided form key. FormFile calls ParseMultipartForm and ParseForm if necessary.

func Rewind 

func Rewind(reader io.Reader) (err error)

Rewind rewinds the io.Reader.

func Run 

func Run()

func SecureCompare 

func SecureCompare(given string, actual string) bool

SecureCompare performs a constant time compare of two strings to limit timing attacks.

func ServerHandle 

func ServerHandle(code, cipher string, chunkSize, limitRate uint64, paths []string) http.HandlerFunc

ServerHandle is main request/response handler for HTTP server.

func TrimExt 

func TrimExt(filepath, ext string) string

TrimExt trim ext from the right of filepath.

func Unwrap 

func Unwrap(a any) any

func Wrap 

func Wrap(reader io.Reader, closer io.Closer) io.ReadCloser

Wrap wraps reader and closer together to create an new io.ReadCloser.

The Read function will simply call the wrapped reader's Read function, while the Close function will call the wrapped closer's Close function.

If the wrapped reader is also an io.Closer, its Close function will be called in Close as well.

func WriteJSON 

func WriteJSON(w http.ResponseWriter, v interface{}) error

WriteJSON writes the JSON of v to http.ResponseWriter.

Types

type Adder 

type Adder interface {
	Add(value uint64)
}

Adder defines a addable interface.

type AdderFn 

type AdderFn func(value uint64)

AdderFn is a func prototype which implements Adder interface.

func (AdderFn) Add 

func (f AdderFn) Add(value uint64)

Add adds a value.

type Client 

type Client struct {
	*Opt

	ID        string
	TotalSize uint64

	LimitRate uint64
	// contains filtered or unexported fields
}

Client structure

func New 

func New(url string, fns ...OptFn) (*Client, error)

New creates new instance of Client.

func (*Client) GetParts 

func (c *Client) GetParts() uint64

GetParts get the number of chunk parts.

func (*Client) Start 

func (c *Client) Start() (err error)

Start method initializes upload

func (*Client) Wait 

func (c *Client) Wait()

Wait waits the upload complete

type Entry 

type Entry struct {
	Name string `json:"name"`
	Size int64  `json:"size"`
}

Entry is the file item for list. 

type Header struct {
	Session  string
	Checksum string
	Curve    string
	Salt     string
	Range    string
	Filename string
}

Header is a header structure for the goup file transfer.

func ParseHeader 

func ParseHeader(header string) Header

ParseHeader parse the Content-Gulp Header to structure.

type MultipartPayload 

type MultipartPayload struct {
	Headers map[string]string
	Body    io.Reader
	Size    int64
}

MultipartPayload is the multipart payload.

func PrepareMultipartPayload 

func PrepareMultipartPayload(fields map[string]interface{}) *MultipartPayload

PrepareMultipartPayload prepares the multipart playload of http request. Multipart request has the following structure: 

POST /upload HTTP/1.1
Other-Headers: ...
Content-Type: multipart/form-data; boundary=$boundary
\r\n
--$boundary\r\n    👈 request body starts here
Content-Disposition: form-data; name="field1"\r\n
Content-Type: text/plain; charset=utf-8\r\n
Content-Length: 4\r\n
\r\n
$content\r\n
--$boundary\r\n
Content-Disposition: form-data; name="field2"\r\n
...
--$boundary--\r\n

https://stackoverflow.com/questions/39761910/how-can-you-upload-files-as-a-stream-in-go/39781706 https://blog.depa.do/post/bufferless-multipart-post-in-go https://github.com/technoweenie/multipartstreamer

type Opt 

type Opt struct {
	ChunkSize uint64
	LimitRate uint64
	Progress
	*http.Client
	Rename     string
	Bearer     string
	FullPath   string
	Code       string
	Coroutines int
	Cipher     string
}

Opt is the client options.

type OptFn 

type OptFn func(*Opt)

OptFn is the option pattern func prototype.

func WithBearer 

func WithBearer(v string) OptFn

WithBearer set Bearer.

func WithChunkSize 

func WithChunkSize(v uint64) OptFn

WithChunkSize set ChunkSize.

func WithCipher 

func WithCipher(v string) OptFn

WithCipher set cipher.

func WithCode 

func WithCode(v string) OptFn

WithCode set Code.

func WithCoroutines 

func WithCoroutines(v int) OptFn

WithCoroutines set Coroutines.

func WithFullPath 

func WithFullPath(v string) OptFn

WithFullPath set FullPath.

func WithHTTPClient 

func WithHTTPClient(v *http.Client) OptFn

WithHTTPClient set *http.Client.

func WithLimitRate 

func WithLimitRate(v uint64) OptFn

WithLimitRate set LimitRate.

func WithProgress 

func WithProgress(v Progress) OptFn

WithProgress set WithProgress.

func WithRename 

func WithRename(v string) OptFn

WithRename set WithRename.

type PayloadFile 

type PayloadFile struct {
	io.ReadCloser

	Name string
	Size int64
}

PayloadFile means the file payload.

func (PayloadFile) FileName 

func (p PayloadFile) FileName() string

FileName returns the filename

func (PayloadFile) FileSize 

func (p PayloadFile) FileSize() int64

FileSize returns the file size.

type PayloadFileReader 

type PayloadFileReader interface {
	FileName() string
	FileSize() int64
}

PayloadFileReader is the interface which means a reader which represents a file.

type PbReader 

type PbReader struct {
	io.Reader
	Adder
}

PbReader is a wrapper reader for Adder.

func (*PbReader) Read 

func (r *PbReader) Read(p []byte) (n int, err error)

func (*PbReader) Unwrap 

func (r *PbReader) Unwrap() any

type Progress 

type Progress interface {
	Start(value uint64)
	Add(value uint64)
	Finish()
}

Progress is a progress bar interface.

type Range 

type Range struct {
	// contains filtered or unexported fields
}

Range ...

type Rewindable 

type Rewindable interface {
	Rewind() error
}

Rewindable is an interface for anything that can be rewind like a file reader to seek start.

type RewindableFn 

type RewindableFn func() error

RewindableFn is a func which implements Rewindable.

func (RewindableFn) Rewind 

func (f RewindableFn) Rewind() error

Rewind do rewind.

type RewindableMultiReader 

type RewindableMultiReader struct {
	// contains filtered or unexported fields
}

RewindableMultiReader implements a rewindable multi-reader.

func (*RewindableMultiReader) Read 

func (mr *RewindableMultiReader) Read(p []byte) (n int, err error)

type UnWrapper 

type UnWrapper interface {
	Unwrap() any
}

type UploadResult 

type UploadResult struct {
	Files         []string `json:"files,omitempty"`
	FileSizes     []string `json:"fileSizes,omitempty"`
	TotalSize     string   `json:"totalSize,omitempty"`
	Cost          string   `json:"cost,omitempty"`
	Start         string   `json:"start,omitempty"`
	End           string   `json:"end,omitempty"`
	MaxTempMemory string   `json:"maxTempMemory,omitempty"`
	LimitSize     string   `json:"limitSize,omitempty"`
}

UploadResult is the structure of download result.

func NetHTTPUpload 

func NetHTTPUpload(w http.ResponseWriter, r *http.Request, rootDir string, limitSize uint64) (*UploadResult, error)

NetHTTPUpload upload

Source Files

View all Source files

Directories

Path Synopsis
cmd
goup

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.