process

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 17, 2025 License: Apache-2.0 Imports: 24 Imported by: 2

Documentation

Index

Constants

View Source 
const (
	CANS_PROTO_FLAG           = "CANS_PROTO_"
	CANS_ARCHIVE_FORMAT_ZIP   = "zip"
	CANS_ARCHIVE_FORMAT_TAR   = "tar"
	CANS_ARCHIVE_FORMAT_TARGZ = "tar.gz"
)

Variables

This section is empty.

Functions

func AddCansProtoRequestHeader added in v0.6.0 

func AddCansProtoRequestHeader(fname, cipher string, chunksNum, chunksId int, totalSize int64, isSplit bool, archiveFormat string) func(req *http.Request)

AddCansProtoRequestHeader is used to set the request header of CANS PROTOCOL based on file block upload

Receive parameter:

  • fname: the name of the file.
  • cipher: symmetric encryption key, used to encrypt data using AES
  • chunksNum: total number of file chunks.
  • chunksId: index of the current chunk to be uploaded ([0,chunksNum)).
  • totalSize: chunks total size (byte), can be obtained from the first return value of SplitFile
  • isSplit: indicate whether the file can be split into different cans.
  • archiveFormat: Specifies the compression format of the file. If it is "", no compression is performed. Currently supported formats are: "zip", "tar", and "tar.gz"

Return parameter:

  • handleFunc: function to set request headers.

func AddFileRequestHeader added in v0.6.0 

func AddFileRequestHeader(territory, bucket, account, message, sig, contentType string) func(req *http.Request)

AddUploadFileRequestHeader is used to set the request header for file upload

Receive parameter:

  • bucket: the territory to which the file will be uploaded, formerly known as bucket
  • territory: the territory(a space block) in which you would like your data to be stored
  • account: CESS account to which the territory belongs
  • message: message to sign
  • sig: signature of the above message using the above CESS account
  • contentType: chunks total size (byte), can be obtained from the first return value of SplitFile

Return parameter:

  • handleFunc: function to set request headers.

func AddUploadChunkRequestHeader added in v0.6.0 

func AddUploadChunkRequestHeader(fname, cipher string, chunksNum, chunksId int, totalSize int64) func(req *http.Request)

AddUploadChunkRequestHeader is used to set the request header for file chunk upload request

Receive parameter:

  • fname: the name of the file.
  • cipher: symmetric encryption key, used to encrypt data using AES
  • chunksNum: total number of file chunks.
  • chunksId: index of the current chunk to be uploaded ([0,chunksNum)).
  • totalSize: chunks total size (byte), can be obtained from the first return value of SplitFile

Return parameter:

  • handleFunc: function to set request headers.

func BatchEncryptWithAES added in v0.6.0 

func BatchEncryptWithAES(files []string, cipher string, saveDir string) ([]string, error)

BatchEncryptWithAES encrypt a batch of files with AES

Receive parameter:

  • files: file list
  • cipher: encryption and decryption cipher
  • saveDir: encrypted files save directory

Return parameter:

  • string: encrypted files
  • error: error message

func CalcUsedSpace added in v0.7.0 

func CalcUsedSpace(size int64) uint64

func CheckAccount added in v0.7.0 

func CheckAccount(cli chain.Chainer, territory string, size int64) error

func CheckFile added in v0.7.0 

func CheckFile(file string) (int64, error)

func DownloadCanFile added in v0.6.0 

func DownloadCanFile(url, mnemonic, savepath, fid, filename, cipher string, sid int) error

DownloadCanFile download files in the directory from the gateway with CANS PROTOCOL,

Receive parameter:

  • url: the address of the gateway.
  • mnemonic: user's CESS account mnemonic.
  • savepath: file path to store downloaded file.
  • fid: file's FID on chain metadata.
  • filename: name of sub file in cans, if it is an empty string, the specified segment(can) is downloaded.
  • cipher: symmetric encryption key, used to encrypt data using AES.
  • sid: segment hash, if filename is an empty string, the specified segment(can) is downloaded.

Return parameter:

  • response: file(if successful).
  • error: error message.

func DownloadFragmentFromMiner added in v0.7.0 

func DownloadFragmentFromMiner(cli chain.Chainer, minerpuk []byte, fid, fragment string, start, end uint64) ([]byte, error)

func DownloadSegment added in v0.7.0 

func DownloadSegment(cli chain.Chainer, savedir string, fid, segmentHash string, fragments []chain.FragmentInfo, size uint64) (string, error)

func EncryptWithAES added in v0.6.0 

func EncryptWithAES(file string, cipher string, saveDir string) (string, error)

EncryptWithAES encrypt a file with AES

Receive parameter:

  • file: file
  • cipher: encryption and decryption cipher
  • saveDir: encrypted file save directory

Return parameter:

  • string: encrypted file
  • error: error message

func ExtractSegmenthash 

func ExtractSegmenthash(segment []string) []string

func FillAndCut added in v0.6.0 

func FillAndCut(file string, saveDir string) ([]string, error)

FillAndCut fill and cut files

Receive parameter:

  • file: the file to be processed
  • saveDir: segment save directory

Return parameter:

  • []string: segment list
  • error: error message

func FillAndCutWithAESEncryption added in v0.6.0 

func FillAndCutWithAESEncryption(file string, cipher string, saveDir string) ([]string, error)

FillAndCutWithAESEncryption fill and cut files, then encrypt using AES algorithm

Receive parameter:

  • file: the file to be processed
  • cipher: encryption and decryption cipher
  • saveDir: segment save directory

Return parameter:

  • []string: segment list
  • error: error message

func FullProcessing added in v0.6.0 

func FullProcessing(file string, cipher string, savedir string) ([]chain.SegmentDataInfo, string, error)

FullProcessing perform full process processing on the file

Receive parameter:

  • file: the file to be processed
  • cipher: encryption and decryption cipher
  • saveDir: saved directory after processing

Return parameter:

  • []segmentDataInfo: segment and fragment information of the file
  • string: [fid] unique identifier for the file
  • error: error message

func Redundancy added in v0.6.0 

func Redundancy(segment []string, saveDir string) ([]chain.SegmentDataInfo, string, error)

Redundancy calculate redundancy for files

Receive parameter:

  • segment: the file to be processed
  • saveDir: fragment save directory

Return parameter:

  • []chain.SegmentDataInfo: segment info
  • string: fid
  • error: error message

func RetrieveFile added in v0.6.0 

func RetrieveFile(url, fid, mnemonic, savepath string) error

RetrieveFile downloads files from the gateway

  • url: gateway url
  • fid: fid
  • mnemonic: polkadot account mnemonic
  • savepath: file save path

Return:

  • string: fid
  • error: error message

func RetrieveFileFromMiners added in v0.7.0 

func RetrieveFileFromMiners(rpcs []string, mnemonic, fid, cipher, savedir string) error

RetrieveFileFromMiners Retrieve a storaged file from storage miners

Receive parameter:

  • rpcs: rpc address list
  • fid: [fid] unique identifier for the file
  • cipher: decryption password, if any
  • savedir: file save directory, final save location: <savedir>/<fid>

Return parameter:

  • error: error message

Preconditions:

  1. the file to be downloaded needs to have been stored in the miner

func RetrieveObject added in v0.6.0 

func RetrieveObject(url, fid, mnemonic string) (io.ReadCloser, error)

RetrieveObject gets the object from the gateway

  • url: gateway url
  • fid: fid
  • mnemonic: polkadot account mnemonic

Return:

  • io.ReadCloser: object
  • error: error message

func Retrievefile added in v0.7.0 

func Retrievefile(cli chain.Chainer, fmeta chain.FileMetadata, fid, savedir, cipher string) (string, error)

func SplitFile added in v0.6.0 

func SplitFile(fpath, chunksDir string, chunkSize int64, filling bool) (int64, int, error)

Split File into Chunks.

Receive parameter:

  • fpath: the path of the file to be split.
  • chunksDir: directory path to store file chunks, please do not mix it elsewhere.
  • chunkSize: the size of each chunk, it does not exceed the file size

Return parameter:

  • int64: chunks total size (byte).
  • int: number of file chunks.
  • error: error message.

func SplitFileWithstandardSize added in v0.6.0 

func SplitFileWithstandardSize(fpath, chunksDir string) (int64, int, error)

Split File into Chunks with standard size. It split file into chunks of the default size and fills the last chunk that does not meet the size.

Receive parameter:

  • fpath: the path of the file to be split.
  • chunksDir: directory path to store file chunks, please do not mix it elsewhere.

Return parameter:

  • int64: chunks total size (byte).
  • int: number of file chunks.
  • error: error message.

func StorageToMiners added in v0.7.0 

func StorageToMiners(cli chain.Chainer, fragmentGroup [][]string, fid string, completedMiner, wantMiner []string) error

func StoreBatchFragmentsToMiner added in v0.7.0 

func StoreBatchFragmentsToMiner(cli chain.Chainer, fragments []string, fid, account string) error

func StoreFile added in v0.6.0 

func StoreFile(url, file, territory, mnemonic string) (string, error)

StoreFile stores files to the gateway

Receive parameter:

  • url: gateway url
  • file: stored file
  • territory: territory name
  • mnemonic: polkadot account mnemonic

Return parameter:

  • string: [fid] unique identifier for the file.
  • error: error message.

Preconditions:

  1. Account requires purchasing space, refer to [BuySpace] interface.
  2. Authorize the space usage rights of the account to the gateway account, refer to the [AuthorizeSpace] interface.

Explanation:

  • Account refers to the account where you configured mnemonic when creating an SDK.

func StoreFileToMiners added in v0.7.0 

func StoreFileToMiners(file string, mnemonic string, territory string, timeout time.Duration, rpcs []string, wantMiner []string) (string, error)

StoreFileToMiners store a file to some miners

Receive parameter:

  • file: stored file
  • mnemonic: account mnemonic
  • territory: territory name
  • timeout: timeout for waiting for block transaction to complete
  • rpcs: rpc address list
  • wantMiner: the wallet account of the miner you want to store. if it is empty, will be randomly selected.

Return parameter:

  • string: [fid] unique identifier for the file
  • error: error message

Preconditions:

  1. your account needs to have money, and will be automatically created if the territory you specify does not exist.
  2. if the number of miners you specify is less than 12, file storage will be exited if even one fails.
  3. if the number of miners you specify is greater than 11, no other miners will be found for storage.

func StoreObject added in v0.6.0 

func StoreObject(url string, territory, mnemonic string, reader io.Reader) (string, error)

StoreObject stores object to the gateway

Receive parameter:

  • url: gateway url
  • territory: territory name
  • mnemonic: polkadot account mnemonic
  • reader: strings, byte data, file streams, network streams, etc

Return parameter:

  • string: [fid] unique identifier for the file
  • error: error message

Preconditions:

  1. Account requires purchasing space, refer to [BuySpace] interface.
  2. Authorize the space usage rights of the account to the gateway account, refer to the [AuthorizeSpace] interface.

Explanation:

  • Account refers to the account where you configured mnemonic when creating an SDK.

func StoreToAllDesignatedMiners added in v0.7.0 

func StoreToAllDesignatedMiners(cli chain.Chainer, fragmentGroup [][]string, fid string, completedMiner, wantMiner []string) error

func UploadFileChunk added in v0.6.0 

func UploadFileChunk(url, mnemonic, file, territory, bucket string, addExtendHeader func(*http.Request)) (string, error)

UploadFileChunk upload chunk of file to the gateway

Receive parameter:

  • url: the address of the gateway.
  • mnemonic: the space owner's CESS account mnemonic
  • file: file path to store file chunks.
  • territory: the territory(a space block) in which you would like your data to be stored
  • bucket: the bucket name to store user data.
  • fname: the name of the file.
  • chunksNum: total number of file chunks.
  • chunksId: index of the current chunk to be uploaded ([0,chunksNum)).
  • totalSize: chunks total size (byte), can be obtained from the first return value of SplitFile

Return parameter:

  • response: chunk ID or file's FID(if all chunks are uploaded successfully).
  • error: error message.

func UploadFileChunks added in v0.6.0 

func UploadFileChunks(url, mnemonic, chunksDir, territory, bucket, fname, cipher string, chunksNum int, totalSize int64) (string, error)

UploadFileChunks upload file chunks in the directory to the gateway as much as possible, chunks will be removed after being uploaded, if the chunks are transferred successfuly.

Receive parameter:

  • url: the address of the gateway.
  • mnemonic: the space owner's CESS account mnemonic
  • chunksDir: directory path to store file chunks, please do not mix it elsewhere.
  • territory: the territory(a space block) in which you would like your data to be stored
  • bucket: the bucket name to store user data.
  • fname: the name of the file.
  • cipher: symmetric encryption key, used to encrypt data using AES
  • chunksNum: total number of file chunks.
  • totalSize: chunks total size (byte), can be obtained from the first return value of SplitFile

Return parameter:

  • response: file's FID(if all chunks are uploaded successfully).
  • error: error message.

func UploadFilesWithCansProto added in v0.6.0 

func UploadFilesWithCansProto(url, mnemonic, filesDir, territory, bucket, archiveFormat, cipher string, isSplit bool) (string, error)

UploadFilesWithCansProto upload files in the directory to the gateway with CANS PROTOCOL,

Receive parameter:

  • url: the address of the gateway.
  • mnemonic: the space owner's CESS account mnemonic
  • filesDir: directory path to store file chunks, please do not mix it elsewhere.
  • territory: the territory(a space block) in which you would like your data to be stored
  • bucket: the bucket name to store user data.
  • archiveFormat: Specifies the compression format of the file. If it is "", no compression is performed. Currently supported formats are: "zip", "tar", and "tar.gz"
  • cipher: symmetric encryption key, used to encrypt data using AES
  • isSplit: indicate whether the file can be split into different cans.

Return parameter:

  • response: file's FID(if all chunks are uploaded successfully).
  • error: error message.

func UploadFragmentToMiner added in v0.7.0 

func UploadFragmentToMiner(cli chain.Chainer, addr string, fid string, file string) error

Types

type CansRequestParams added in v0.6.0 

type CansRequestParams struct {
	SegmentIndex int    `json:"segment_index"`
	SubFile      string `json:"sub_file"`
	Cipher       string `json:"cipher"`
}

type RespType added in v0.7.0 

type RespType struct {
	Code int    `json:"code"`
	Msg  string `json:"msg"`
	Data any    `json:"data"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.