util

package
v1.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2024 License: MIT Imports: 25 Imported by: 4

Documentation

Index

Constants

View Source 
const (
	MerkleChunkSize     = 64
	MaxMerkleLeavesSize = 64 * 1024
	FixedMerkleLeaves   = 1024
	FixedMTDepth        = 11
)
View Source 
const (
	Left = iota
	Right
)
View Source 
const (
	START_LENGTH = 64
	ADD_LENGTH   = 320
)

Variables

View Source 
var ErrNilHttpConsensusMaps = errors.New("nil_httpconsensusmaps")
View Source 
var (

	// ErrNoItem there is no item anymore
	ErrNoItem = errors.New("rand: there is no item anymore")
)

Functions

func DecodeAndMHash added in v1.8.14 

func DecodeAndMHash(h1, h2 string) string

DecodeAndMHash will decode hex-encoded string to []byte format. This function should only be used with hex-encoded string otherwise the result will be obsolute.

func GetNewUUID added in v1.8.13 

func GetNewUUID() uuid.UUID

GetNewUUID will give new version1 uuid. It will panic if any error occurred

func GetRandom 

func GetRandom(in []string, n int) []string

GetRandom returns n random slice from in If n > len(in), then this will return a shuffled version of in

func GetSHA1Uuid added in v1.8.13 

func GetSHA1Uuid(u uuid.UUID, name string) uuid.UUID

GetSHA1Uuid will give version 5 uuid. It depends on already existing uuid. The main idea is to synchronize uuid among blobbers. So for example, if a file is being uploaded to 4 blobbers then we require that file in each blobber have same uuid. All the goroutine that assigns uuid, calculates hashes and commits to blobber will use initial version 1 uuid and updates the uuid with recently calculated uuid so that the file will get same uuid in all blobbers.

func Hash 

func Hash(text string) string

Hash - the hashing used for the merkle tree construction

func HashStringToBytes 

func HashStringToBytes(hash string) []byte

HashStringToBytes - convert a hex hash string to bytes

func MHash 

func MHash(h1 string, h2 string) string

MHash - merkle hashing of a pair of child hashes

func MHashBytes added in v1.8.14 

func MHashBytes(h1, h2 []byte) []byte

func MaxInt 

func MaxInt(x, y int) int

func MaxInt64 

func MaxInt64(x, y int64) int64

func MinInt 

func MinInt(x, y int) int

func MinInt64 

func MinInt64(x, y int64) int64

func ParseCoinStr added in v1.8.12 

func ParseCoinStr(vs string) (uint64, error)

func Shuffle 

func Shuffle(in []string) (shuffle []string)

func ToHex 

func ToHex(buf []byte) string

ToHex - converts a byte array to hex encoding with upper case

func VerifyMerklePath 

func VerifyMerklePath(hash string, path *MTPath, root string) bool

Types

type FixedMerklePath added in v1.8.14 

type FixedMerklePath struct {
	LeafHash []byte   `json:"leaf_hash"`
	RootHash []byte   `json:"root_hash"`
	Nodes    [][]byte `json:"nodes"`
	LeafInd  int
}

FixedMerklePath is used to verify existence of leaf hash for fixed merkle tree

func (FixedMerklePath) VerifyMerklePath added in v1.8.14 

func (fp FixedMerklePath) VerifyMerklePath() bool

type FixedMerkleTree added in v1.2.88 

type FixedMerkleTree struct {
	// Leaves will store hash digester that calculates sha256 hash of the leaf content
	Leaves []Hashable `json:"leaves,omitempty"`
	// contains filtered or unexported fields
}

FixedMerkleTree A trusted mekerle tree for outsourcing attack protection. see section 1.8 on whitepager see detail on https://github.com/0chain/blobber/wiki/Protocols#what-is-fixedmerkletree

func NewFixedMerkleTree added in v1.2.88 

func NewFixedMerkleTree() *FixedMerkleTree

NewFixedMerkleTree create a FixedMerkleTree with specify hash method

func (*FixedMerkleTree) CalculateMerkleRoot added in v1.8.14 

func (fmt *FixedMerkleTree) CalculateMerkleRoot()

func (*FixedMerkleTree) Finalize added in v1.8.14 

func (fmt *FixedMerkleTree) Finalize() error

Finalize will set isFinal to true and sends remaining bytes for leaf hash calculation

func (*FixedMerkleTree) GetMerkleRoot added in v1.2.88 

func (fmt *FixedMerkleTree) GetMerkleRoot() string

GetMerkleRoot get merkle root.

func (*FixedMerkleTree) GetMerkleTree added in v1.2.88 

func (fmt *FixedMerkleTree) GetMerkleTree() MerkleTreeI

GetMerkleRoot is only for interface compliance.

func (*FixedMerkleTree) Reload added in v1.2.88 

func (fmt *FixedMerkleTree) Reload(reader io.Reader) error

Reload reset and reload leaves from io.Reader

func (*FixedMerkleTree) Write added in v1.2.88 

func (fmt *FixedMerkleTree) Write(b []byte) (int, error)

Write will write data to the leaves once MaxMerkleLeavesSize(64 KB) is reached. Since each 64KB is divided into 1024 pieces with 64 bytes each, once data len reaches 64KB then it will be written to leaf hashes. The remaining data that is not multiple of 64KB will be written to leaf hashes by Finalize() function. This can be used to write stream of data as well. fmt.Finalize() is required after data write is complete.

type GetRequest 

type GetRequest struct {
	*PostRequest
}

func NewHTTPGetRequest 

func NewHTTPGetRequest(url string) (*GetRequest, error)

NewHTTPGetRequest create a GetRequest instance with 60s timeout

func NewHTTPGetRequestContext 

func NewHTTPGetRequestContext(ctx context.Context, url string) (*GetRequest, error)

NewHTTPGetRequestContext create a GetRequest with context and url

func (*GetRequest) Get 

func (r *GetRequest) Get() (*GetResponse, error)

type GetResponse 

type GetResponse struct {
	*PostResponse
}

type Hashable 

type Hashable interface {
	GetHash() string
	GetHashBytes() []byte
	Write(b []byte) (int, error)
}

Hashable - anything that can provide it's hash

type HttpClient 

type HttpClient interface {
	Do(req *http.Request) (*http.Response, error)
}
var Client HttpClient

type HttpConsensusMaps added in v1.11.0 

type HttpConsensusMaps struct {
	ConsensusThresh int
	MaxConsensus    int

	WinMap          map[string]json.RawMessage
	WinMapConsensus map[string]int

	WinError string
	WinInfo  string
}

func NewHttpConsensusMaps added in v1.11.0 

func NewHttpConsensusMaps(consensusThresh int) *HttpConsensusMaps

func (*HttpConsensusMaps) Add added in v1.11.0 

func (c *HttpConsensusMaps) Add(statusCode int, respBody string) error

func (*HttpConsensusMaps) GetValue added in v1.11.0 

func (c *HttpConsensusMaps) GetValue(name string) (json.RawMessage, bool)

type MTPath 

type MTPath struct {
	Nodes     []string `json:"nodes"`
	LeafIndex int      `json:"leaf_index"`
}

MTPath - The merkle tree path

type MerklePathForMultiLeafVerification added in v1.8.14 

type MerklePathForMultiLeafVerification struct {
	// RootHash that was signed by the client
	RootHash []byte
	// Nodes contains a slice for each merkle node level. Each slice contains hash that will
	// be concatenated with the calculated hash from the level below.
	// It is used together with field Index [][]int
	// Length of Nodes will be according to number of blocks requested. If whole data is requested then
	// blobber will send nil for Nodes i.e. length of Nodes will become zero.
	Nodes [][][]byte `json:"nodes"`
	// Index slice that determines whether to concatenate hash to left or right.
	// It should have maximum of length 2 and minimum of 0. It is used together with field Nodes [][][]byte
	Index [][]int `json:"index"`
	// DataSize is size of data received by the blobber for the respective file.
	// It is not same as actual file size
	DataSize int64
	// contains filtered or unexported fields
}

MerklePathForMultiLeafVerification is used to verify multiple blocks with single instance of merkle path. Usually client would request with counter incremented by 10. So if the block size is 64KB and counter is incremented by 10 then client is requesting 640 KB of data. Blobber can then provide sinlge merkle path instead of sending 10 merkle paths.

func (*MerklePathForMultiLeafVerification) VerifyMultipleBlocks added in v1.8.14 

func (m *MerklePathForMultiLeafVerification) VerifyMultipleBlocks(data []byte) error

VerifyMultipleBlocks will verify merkle path for continuous data which is multiple of 64KB blocks

There can be at most 2 hashes in the input for each depth i.e. of the format below: h1, data1, data2, data3, data4, h2 Note that data1, data2, data3,... should be continuous data

i#3 h14 i#2 h12 h13 i#1 h7 h8 h9 h10 i#0 h0, h1, h2, h3, h4, h5, h6

Consider there are 7 leaves(0...6) as shown above. Now if client wants data from 1-3 then blobber needs to provide:

1. One node from i#0, [h0]; data1 will generate h1,data2 will generate h2 and so on... 2. Zero node from i#1; h0 and h1 will generate h7 and h2 and h3 will generate h8 3. One node from i#2, h[13]; h7 and h8 will generate h12. Now to get h14, we need h13 which will be provided by blobber

i#5 h37 i#4 h35 h36 i#3 h32 h33 h34 i#2 h27 h28 h29 h30 h31 i#1 h18 h19 h20 h21 h22 h23 h24 h25, h26 i#0 h0, h1, h2, h3, h4, h5, h6, h7, h8, h9, h10, h11, h12, h13, h14, h15, h16, h17

Consider there are 16 leaves(0..15) with total data = 16*64KB as shown above. If client wants data from 3-10 then blobber needs to provide: 1. Two nodes from i#0, [h2, h11] 2. One node from i#1, [h16] 3. One node from i#2, [h27]

If client had required data from 2-9 then blobber would have to provide: 1. Zero nodes from i#0 2. Two nodes from i#1, [h16, h21] 3. One node from i#2, [h27]

type MerkleTree 

type MerkleTree struct {
	// contains filtered or unexported fields
}

MerkleTree - A data structure that implements MerkleTreeI interface

func (*MerkleTree) ComputeTree 

func (mt *MerkleTree) ComputeTree(hashes []Hashable)

ComputeTree - given the leaf nodes, compute the merkle tree

func (*MerkleTree) GetLeafIndex 

func (mt *MerkleTree) GetLeafIndex(hash Hashable) int

GetLeafIndex - Get the index of the leaf node in the tree

func (*MerkleTree) GetPath 

func (mt *MerkleTree) GetPath(hash Hashable) *MTPath

GetPath - get the path that can be used to verify the merkle tree

func (*MerkleTree) GetPathByIndex 

func (mt *MerkleTree) GetPathByIndex(idx int) *MTPath

GetPathByIndex - get the path of a leaf node at index i

func (*MerkleTree) GetRoot 

func (mt *MerkleTree) GetRoot() string

GetRoot - get the root of the merkle tree

func (*MerkleTree) GetTree 

func (mt *MerkleTree) GetTree() []string

GetTree - get the entire merkle tree

func (*MerkleTree) SetTree 

func (mt *MerkleTree) SetTree(leavesCount int, tree []string) error

SetTree - set the entire merkle tree

func (*MerkleTree) VerifyPath 

func (mt *MerkleTree) VerifyPath(hash Hashable, path *MTPath) bool

VerifyPath - given a leaf node and the path, verify that the node is part of the tree

type MerkleTreeI 

type MerkleTreeI interface {
	//API to create a tree from leaf nodes
	ComputeTree(hashes []Hashable)
	GetRoot() string
	GetTree() []string

	//API to load an existing tree
	SetTree(leavesCount int, tree []string) error

	// API for verification when the leaf node is known
	GetPath(hash Hashable) *MTPath               // Server needs to provide this
	VerifyPath(hash Hashable, path *MTPath) bool //This is only required by a client but useful for testing

	/* API for random verification when the leaf node is uknown
	(verification of the data to hash used as leaf node is outside this API) */
	GetPathByIndex(idx int) *MTPath
}

MerkleTreeI - a merkle tree interface required for constructing and providing verification

type PostRequest 

type PostRequest struct {
	// contains filtered or unexported fields
}

func NewHTTPPostRequest 

func NewHTTPPostRequest(url string, data interface{}) (*PostRequest, error)

func (*PostRequest) Post 

func (r *PostRequest) Post() (*PostResponse, error)

type PostResponse 

type PostResponse struct {
	Url        string
	StatusCode int
	Status     string
	Body       string
}

type Rand added in v1.2.82 

type Rand struct {
	// contains filtered or unexported fields
}

Rand a progressive rand

func NewRand added in v1.2.82 

func NewRand(max int) Rand

NewRand create a ProgressiveRand instance

func (*Rand) Next added in v1.2.82 

func (r *Rand) Next() (int, error)

Next get next random item

type SecureSerializableValue 

type SecureSerializableValue struct {
	Buffer []byte
}

SecureSerializableValue - a proxy persisted value that just tracks the encoded bytes of a persisted value

func (*SecureSerializableValue) Decode 

func (spv *SecureSerializableValue) Decode(buf []byte) error

Decode - implement interface

func (*SecureSerializableValue) Encode 

func (spv *SecureSerializableValue) Encode() []byte

Encode - implement interface

func (*SecureSerializableValue) GetHash 

func (spv *SecureSerializableValue) GetHash() string

GetHash - implement interface

func (*SecureSerializableValue) GetHashBytes 

func (spv *SecureSerializableValue) GetHashBytes() []byte

GetHashBytes - implement interface

type SecureSerializableValueI 

type SecureSerializableValueI interface {
	Serializable
	Hashable
}

SecureSerializableValueI an interface that makes a serializable value secure with hashing

type Serializable 

type Serializable interface {
	Encode() []byte
	Decode([]byte) error
}

Serializable interface

type StringHashable 

type StringHashable struct {
	Hash string
}

func NewStringHashable 

func NewStringHashable(hash string) *StringHashable

func (*StringHashable) GetHash 

func (sh *StringHashable) GetHash() string

func (*StringHashable) GetHashBytes 

func (sh *StringHashable) GetHashBytes() []byte

func (StringHashable) Write added in v1.8.14 

func (StringHashable) Write(_ []byte) (int, error)

type ValidationTree added in v1.8.14 

type ValidationTree struct {
	// contains filtered or unexported fields
}

func NewValidationTree added in v1.8.14 

func NewValidationTree(dataSize int64) *ValidationTree

func (*ValidationTree) CalculateDepth added in v1.8.14 

func (v *ValidationTree) CalculateDepth() int

func (*ValidationTree) Finalize added in v1.8.14 

func (v *ValidationTree) Finalize() error

func (*ValidationTree) GetDataSize added in v1.8.14 

func (v *ValidationTree) GetDataSize() int64

func (*ValidationTree) GetLeaves added in v1.8.14 

func (v *ValidationTree) GetLeaves() [][]byte

func (*ValidationTree) GetValidationRoot added in v1.8.14 

func (v *ValidationTree) GetValidationRoot() []byte

func (*ValidationTree) SetLeaves added in v1.8.14 

func (v *ValidationTree) SetLeaves(leaves [][]byte)

func (*ValidationTree) Write added in v1.8.14 

func (v *ValidationTree) Write(b []byte) (int, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.