goup

package module
v0.0.0-...-b1debd4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 19, 2024 License: MIT Imports: 36 Imported by: 1

README

goup

Utility to upload or download large files through HTTP in broken-point continuously.

This is done by splitting large file into small chunks; whenever the upload of a chunk fails, uploading is retried until the procedure completes. This allows uploads to automatically resume uploading after a network connection is lost, or program restarting either locally or to the server.

Features

  1. splitting large file into smaller chunks.
  2. uploading/downloading by HTTP.
  3. resume-able. chunk's hash will be checked before transfer.
  4. security data by AES-GCM based on PAKE.
  5. support download short path like goup -path /xx=/xx.zip, then the client can use http://127.0.0.1:2001/xx to download the xx.zip file.
API Method Req Content-Gulp Rsp Content-Gulp Other Headers Function
1. POST Filename 明文上传(文件作为 Body)
2. POST Session, Curve Curve, PAKE 生成会话秘钥
3. GET / Session, Range, Checksum Req: Content-Disposition 校验分块 checksum,返回 304 或 其它
4. POST / Session, Range, Salt Req: Content-Disposition 分块加密上传(加密分块作为 Body)
5. GET / HTML JS 上传页面 / 服务端文件列表(Accept: application/json 时)
6. GET / Session, Range, Checksum Range, Salt Rsp: Content-Type , Content-Disposition 分块加密下载
7. GET / Salt Rsp: Content-Type, Content-Length, Content-Disposition 明文下载
8. POST 明文上传(multipart-form)

Usage

$ goup -h
Usage of goup:
  -b    string Bearer token for client or server, auto for server to generate a random one
  -c    string Chunk size for client, unit MB (default 10)
  -t    int    Threads (go-routines) for client
  -f    string Upload file path for client
  -p    int    Listening port for server
  -r    string Rename to another filename
  -u    string Server upload url for client to connect to
  -P    string Password for PAKE
  -C    string Cipher AES256: AES-256 GCM, C20P1305: ChaCha20 Poly1305
  -v    bool   Show version
  1. Installation go install https://github.com/bingoohuang/ngg/goup
  2. At the server, goup
  3. At the client, goup -u http://a.b.c:2110/ -f 246.png
$ goup
2022/01/20 17:40:20 Listening on 2110
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 0-10485760/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 10485760-20971520/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 20971520-31457280/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 31457280-41943040/96894303
2022/01/20 17:40:24 recieved file 246.png with sessionID CF79FC6EA88010E1, range bytes 41943040-52428800/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 52428800-62914560/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 62914560-73400320/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 73400320-83886080/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 83886080-94371840/96894303
2022/01/20 17:40:47 recieved file 246.png with sessionID 46CB3E789B10DDB5, range bytes 94371840-96894303/96894303

$ md5sum .goup-files/246.png
5849c383ab841e539673d50a49c3f39a  .goup-files/aDrive.dmg
$ sha256sum .goup-files/246.png
32677ca3a5b19ee5590a75d423b3fc2d754bd76900d50c26e0525681985189f8  .goup-files/aDrive.dmg

upload example:

$ goup -u http://127.0.0.1:2110/ -f 246.png
2022/01/20 17:44:08 Upload B821C49E4B1CBDBD started
40.00 MiB / 92.41 MiB [------------------>____________________] 43.29% ? p/s^C
$ diff -s /Users/bingoobjca/Downloads/goup/.goup-files/246.png 246.png
Binary files /Users/bingoobjca/Downloads/goup/.goup-files/246.png and 246.png differ
$ goup -u http://127.0.0.1:2110/ -f 246.png
2022/01/20 17:44:21 Upload 4739FE47D18ADF7F started
92.41 MiB / 92.41 MiB [---------------------------------------] 100.00% 121.54 MiB p/s
2022/01/20 17:44:21 Upload 4739FE47D18ADF7F completed
$ diff -s /Users/bingoobjca/Downloads/goup/.goup-files/246.png 246.png
Files /Users/bingoobjca/Downloads/goup/.goup-files/246.png and 246.png are identical
$ md5 246.png
MD5 (aDrive.dmg) = 5849c383ab841e539673d50a49c3f39a
$ sha256sum 246.png
32677ca3a5b19ee5590a75d423b3fc2d754bd76900d50c26e0525681985189f8  246.png

As you can see, even if you terminate the client uploading, you can resume uploading from the last breakpoint.

$ goup -u http://127.0.0.1:2110/246.png
2022/01/21 13:29:43 download started: .goup-files/246.png
92.41 MiB / 92.41 MiB [---------------------------------------] 100.00% 97.56 MiB p/s
2022/01/21 13:29:44 download complete: .goup-files/246.png

normal upload/download support

upload:

  1. Start the server without a chunk: goup -c0
  2. Test the upload: curl -F "file:=@a.mp4" localhost:2110

download:

  1. Start the server: goup
  2. Download
    1. by gurl: gurl :2110/246.png
    2. by curl: curl -LO 127.0.0.1:2110/246.png
    3. by goup disable chunk: goup -u :2110/246.png -c0

Resources

  1. SRP Go implementation of the Secure Remote Password (SRP) protocol.

    SRP is an authentication method that allows the use of user names and passwords over unencrypted channels without revealing the password to an eavesdropper. SRP also supplies a shared secret at the end of the authentication sequence that can be used to generate encryption keys.

  2. Encrypted-Content-Encoding HTTP Go implementation of Encrypted-Content-Encoding for HTTP (RFC 8188).

Documentation

Overview

Package goup contains golang upload/download server and client.

Index

Constants

View Source
const (
	// Authorization is the header name for Authorization
	Authorization = "Authorization"
	// ContentDisposition is header name for Content-Disposition
	ContentDisposition = "Content-Disposition"
	// ContentType is the header name for Content-Type
	ContentType = "Content-Type"
	// ContentLength is the header name for Content-Length
	ContentLength = "Content-Length"
)

Variables

View Source
var ErrMissingFile = errors.New("there is no uploaded file")

ErrMissingFile may be returned from FormFile when the is no uploaded file.

View Source
var RootDir = "./.goup"

RootDir settings. When finished uploading with success files are stored inside it.

Functions

func Bearer

func Bearer(token string, handle http.HandlerFunc) http.HandlerFunc

Bearer returns a Handler that authenticates via Bearer Auth. Writes a http.StatusUnauthorized if authentication fails.

func BearerTokenGenerate

func BearerTokenGenerate() string

BearerTokenGenerate generates a random bearer token.

func CreateChunkReader

func CreateChunkReader(fullPath string, partFrom, partTo uint64, limitRate uint64) (r io.ReadCloser, err error)

CreateChunkReader creates a chunk reader for the file.

func GetPartSize

func GetPartSize(totalSize, chunkSize, idx uint64) uint64

GetPartSize get the part size of idx-th chunk.

func InitServer

func InitServer() error

InitServer initializes the server.

func NetHTTPUpload

func NetHTTPUpload(w http.ResponseWriter, r *http.Request, rootDir string, limitSize uint64) error

NetHTTPUpload upload

func NewRewindableMultiReader

func NewRewindableMultiReader(readers ...io.Reader) io.Reader

NewRewindableMultiReader returns a Reader that's the logical concatenation of the provided input readers. They're read sequentially. Once all inputs have returned EOF, Read will return EOF. If any of the readers return a non-nil, non-EOF error, Read will return that error.

func ParseFormFile

func ParseFormFile(m *multipart.Form) (*multipart.FileHeader, error)

ParseFormFile returns the first file for the provided form key. FormFile calls ParseMultipartForm and ParseForm if necessary.

func Rewind

func Rewind(reader io.Reader) (err error)

Rewind rewinds the io.Reader.

func SecureCompare

func SecureCompare(given string, actual string) bool

SecureCompare performs a constant time compare of two strings to limit timing attacks.

func ServerHandle

func ServerHandle(code, cipher string, chunkSize, limitRate uint64, paths []string) http.HandlerFunc

ServerHandle is main request/response handler for HTTP server.

func TrimExt

func TrimExt(filepath, ext string) string

TrimExt trim ext from the right of filepath.

func Unwrap

func Unwrap(a any) any

func Wrap

func Wrap(reader io.Reader, closer io.Closer) io.ReadCloser

Wrap wraps reader and closer together to create an new io.ReadCloser.

The Read function will simply call the wrapped reader's Read function, while the Close function will call the wrapped closer's Close function.

If the wrapped reader is also an io.Closer, its Close function will be called in Close as well.

func WriteJSON

func WriteJSON(w http.ResponseWriter, v interface{}) error

WriteJSON writes the JSON of v to http.ResponseWriter.

Types

type Adder

type Adder interface {
	Add(value uint64)
}

Adder defines a addable interface.

type AdderFn

type AdderFn func(value uint64)

AdderFn is a func prototype which implements Adder interface.

func (AdderFn) Add

func (f AdderFn) Add(value uint64)

Add adds a value.

type Client

type Client struct {
	*Opt

	ID        string
	TotalSize uint64

	LimitRate uint64
	// contains filtered or unexported fields
}

Client structure

func New

func New(url string, fns ...OptFn) (*Client, error)

New creates new instance of Client.

func (*Client) GetParts

func (c *Client) GetParts() uint64

GetParts get the number of chunk parts.

func (*Client) Start

func (c *Client) Start() (err error)

Start method initializes upload

func (*Client) Wait

func (c *Client) Wait()

Wait waits the upload complete

type Entry

type Entry struct {
	Name string `json:"name"`
	Size int64  `json:"size"`
}

Entry is the file item for list.

type Header struct {
	Session  string
	Checksum string
	Curve    string
	Salt     string
	Range    string
	Filename string
}

Header is a header structure for the goup file transfer.

func ParseHeader

func ParseHeader(header string) Header

ParseHeader parse the Content-Gulp Header to structure.

type MultipartPayload

type MultipartPayload struct {
	Headers map[string]string
	Body    io.Reader
	Size    int64
}

MultipartPayload is the multipart payload.

func PrepareMultipartPayload

func PrepareMultipartPayload(fields map[string]interface{}) *MultipartPayload

PrepareMultipartPayload prepares the multipart playload of http request. Multipart request has the following structure:

POST /upload HTTP/1.1
Other-Headers: ...
Content-Type: multipart/form-data; boundary=$boundary
\r\n
--$boundary\r\n    👈 request body starts here
Content-Disposition: form-data; name="field1"\r\n
Content-Type: text/plain; charset=utf-8\r\n
Content-Length: 4\r\n
\r\n
$content\r\n
--$boundary\r\n
Content-Disposition: form-data; name="field2"\r\n
...
--$boundary--\r\n

https://stackoverflow.com/questions/39761910/how-can-you-upload-files-as-a-stream-in-go/39781706 https://blog.depa.do/post/bufferless-multipart-post-in-go https://github.com/technoweenie/multipartstreamer

type Opt

type Opt struct {
	ChunkSize uint64
	LimitRate uint64
	Progress
	*http.Client
	Rename     string
	Bearer     string
	FullPath   string
	Code       string
	Coroutines int
	Cipher     string
}

Opt is the client options.

type OptFn

type OptFn func(*Opt)

OptFn is the option pattern func prototype.

func WithBearer

func WithBearer(v string) OptFn

WithBearer set Bearer.

func WithChunkSize

func WithChunkSize(v uint64) OptFn

WithChunkSize set ChunkSize.

func WithCipher

func WithCipher(v string) OptFn

WithCipher set cipher.

func WithCode

func WithCode(v string) OptFn

WithCode set Code.

func WithCoroutines

func WithCoroutines(v int) OptFn

WithCoroutines set Coroutines.

func WithFullPath

func WithFullPath(v string) OptFn

WithFullPath set FullPath.

func WithHTTPClient

func WithHTTPClient(v *http.Client) OptFn

WithHTTPClient set *http.Client.

func WithLimitRate

func WithLimitRate(v uint64) OptFn

WithLimitRate set LimitRate.

func WithProgress

func WithProgress(v Progress) OptFn

WithProgress set WithProgress.

func WithRename

func WithRename(v string) OptFn

WithRename set WithRename.

type PayloadFile

type PayloadFile struct {
	io.ReadCloser

	Name string
	Size int64
}

PayloadFile means the file payload.

func (PayloadFile) FileName

func (p PayloadFile) FileName() string

FileName returns the filename

func (PayloadFile) FileSize

func (p PayloadFile) FileSize() int64

FileSize returns the file size.

type PayloadFileReader

type PayloadFileReader interface {
	FileName() string
	FileSize() int64
}

PayloadFileReader is the interface which means a reader which represents a file.

type PbReader

type PbReader struct {
	io.Reader
	Adder
}

PbReader is a wrapper reader for Adder.

func (*PbReader) Read

func (r *PbReader) Read(p []byte) (n int, err error)

func (*PbReader) Unwrap

func (r *PbReader) Unwrap() any

type Progress

type Progress interface {
	Start(value uint64)
	Add(value uint64)
	Finish()
}

Progress is a progress bar interface.

type Range

type Range struct {
	// contains filtered or unexported fields
}

Range ...

type Rewindable

type Rewindable interface {
	Rewind() error
}

Rewindable is an interface for anything that can be rewind like a file reader to seek start.

type RewindableFn

type RewindableFn func() error

RewindableFn is a func which implements Rewindable.

func (RewindableFn) Rewind

func (f RewindableFn) Rewind() error

Rewind do rewind.

type RewindableMultiReader

type RewindableMultiReader struct {
	// contains filtered or unexported fields
}

RewindableMultiReader implements a rewindable multi-reader.

func (*RewindableMultiReader) Read

func (mr *RewindableMultiReader) Read(p []byte) (n int, err error)

type UnWrapper

type UnWrapper interface {
	Unwrap() any
}

type UploadResult

type UploadResult struct {
	Files         []string
	FileSizes     []string
	TotalSize     string
	Cost          string
	Start         string
	End           string
	MaxTempMemory string
	LimitSize     string
}

UploadResult is the structure of download result.

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL