fileupload

README

This is an upload intermediary service

The features to be done now are as follows

• The client uploads the file and receives it from the server
• The client bridges the http upload and the server receives it.
• Both the client and the server should support HTTP and grpc.
• Add gRPC upload function
• Unified HTTP and gRPC upload interfaces
• Support multi-file upload (by calling uploader/receiver multiple times)
• Provides better error handling and resource management
• Added folder upload support
• Implemented the function of resumable transmission
• Buffer pools are used to optimize memory usage
• Maintain interface consistency
• Better error handling is provided
• Provides HTTP to gRPC bridge upload
• The bridge from HTTP to gRPC supports resumable upload
• HTTP to gRPC bridging optimizes memory usage using buffer pools
• Contextual support is provided for the HTTP to gRPC bridge upload

Implementation of the underlying logic

• The user clicks the upload button to upload the file (HTTP).
• The server receives the uploader that bridges HTTP and gRPC) and forwards it to the grpc server.
• The grpc server receives the data and saves it to the file/OSS.

Examples:

func main() {

 // Create a gRPC Builder

  grpcBuilder := NewBuilder(
    WithServiceType(ServiceTypeGRPC),
    WithURI("grpc-server-address:port"),
  )

  // Create a bridge uploader
  bridgeUploader := NewBridgeUploader(grpcBuilder, &uploadBuilder{
    bufPool: &sync.Pool{
      New: func() interface{} {
        return make([]byte, 32*1024)
      },
    },
  })

  // Register the HTTP processor
  http.HandleFunc("/upload", bridgeUploader.ServeHTTP)
  http.HandleFunc("/upload/resume", bridgeUploader.ServeHTTPWithResume)
  // Start the HTTP server
  http.ListenAndServe(":8080", nil)

}

Workflow:

• The client uploads the file via HTTP
• BridgeUploader receives HTTP requests
• Use HTTPReceiver to read the contents of the file
• Create a gRPC uploader and forward the file
• Return the gRPC response to the HTTP client

Documentation

Overview

Package fileupload implements the functions, types, and interfaces for the module.

Package fileupload implements the functions, types, and interfaces for the module.

Index

• Constants
• type BridgeUploader
• type FileHeader
• type FileInfo
• type Receiver
• type ServiceType
  • func (i ServiceType) String() string
• type UploadResponse
• type Uploader

Constants

const (
	ModTimeFormat = "Mon, 02 Jan 2006 15:04:05 MST"
)

Types

type BridgeUploader

type BridgeUploader interface {
	ServeHTTP(w http.ResponseWriter, r *http.Request)
}

type FileHeader

type FileHeader interface {
	// GetFilename returns the name of the file.
	GetFilename() string

	// GetSize returns the size of the file in bytes.
	GetSize() uint32

	// GetModTime returns the last modified time of the file as a string.
	GetModTime() uint32

	// GetModTimeString returns the last modified time of the file as a Unix timestamp.
	GetModTimeString() string

	// GetContentType returns the MIME type of the file.
	GetContentType() string

	// GetHeader returns a map of additional file headers.
	GetHeader() map[string]string

	// GetIsDir returns true if the file is a directory.
	GetIsDir() bool
}

FileHeader represents a file header with metadata.

type FileInfo

type FileInfo interface {
	fs.FileInfo
	ContentType() string
}

type Receiver

type Receiver interface {
	// GetFileHeader retrieves the file header after the file has been received.
	GetFileHeader(ctx context.Context) (FileHeader, error)
	// ReceiveFile receives the file first and then retrieves the header.
	ReceiveFile(ctx context.Context) (io.ReadCloser, error)
	// GetOffset returns current offset for resume support
	GetOffset(ctx context.Context) (int64, error)
	// Finalize finalizes the receipt process.
	Finalize(ctx context.Context, resp UploadResponse) error
}

type ServiceType

type ServiceType int

ServiceType represents the type of the service.

const (
	ServiceTypeGRPC    ServiceType = iota
	ServiceTypeHTTP    ServiceType = iota
	ServiceTypeUnknown ServiceType = iota
)

func (ServiceType) String

func (i ServiceType) String() string

type UploadResponse

type UploadResponse interface {
	// GetSuccess indicates whether the file upload was successful.
	GetSuccess() bool

	// GetHash returns the hash of the uploaded file.
	GetHash() string

	// GetPath returns the path where the file was uploaded.
	GetPath() string

	// GetSize returns the size of the uploaded file in bytes.
	GetSize() uint32

	// GetFailReason returns the failure reason of the file upload.
	GetFailReason() string
}

UploadResponse represents a response to a file upload request.

type Uploader

type Uploader interface {
	// SetFileHeader sets the file header after the file has been uploaded.
	SetFileHeader(ctx context.Context, header FileHeader) error
	// UploadFile uploads the file first and then sets the header.
	UploadFile(ctx context.Context, rd io.Reader) error
	// Resume resumes upload from specified offset
	Resume(ctx context.Context, offset int64) error
	// Finalize finalizes the upload process.
	Finalize(ctx context.Context) (UploadResponse, error)
}