chunksPkg

package
v0.0.0-...-2de43c5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 20, 2024 License: GPL-3.0 Imports: 42 Imported by: 0

README

chifra chunks

The chifra chunks routine provides tools for interacting with, checking the validity of, cleaning up, and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data on the index, Bloom filters, addresses, and appearances. While still in its early stages, this tool will eventually allow users to clean their local index, clean their remote index, study the indexes, etc. Stay tuned.

Purpose:
  Manage, investigate, and display the Unchained Index.

Usage:
  chifra chunks <mode> [flags] [blocks...] [address...]

Arguments:
  mode - the type of data to process (required)
    One of [ manifest | index | blooms | pins | addresses | appearances | stats ]
  blocks - an optional list of blocks to intersect with chunk ranges

Flags:
  -c, --check              check the manifest, index, or blooms for internal consistency
  -i, --pin                pin the manifest or each index chunk and bloom
  -p, --publish            publish the manifest to the Unchained Index smart contract
  -r, --remote             prior to processing, retrieve the manifest from the Unchained Index smart contract
  -b, --belongs strings    in index mode only, checks the address(es) for inclusion in the given index chunk
  -F, --first_block uint   first block to process (inclusive)
  -L, --last_block uint    last block to process (inclusive)
  -m, --max_addrs uint     the max number of addresses to process in a given chunk
  -d, --deep               if true, dig more deeply during checking (manifest only)
  -e, --rewrite            for the --pin --deep mode only, writes the manifest back to the index folder (see notes)
  -U, --count              for certain modes only, display the count of records
  -s, --sleep float        for --remote pinning only, seconds to sleep between API calls
  -x, --fmt string         export format, one of [none|json*|txt|csv]
  -v, --verbose            enable verbose output
  -h, --help               display this help screen

Notes:
  - Mode determines which type of data to display or process.
  - Certain options are only available in certain modes.
  - If blocks are provided, only chunks intersecting with those blocks are displayed.
  - The --truncate option updates the manifest and removes local data, but does not alter remote pins.
  - The --belongs option is only available in the index mode.
  - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode.
  - The --pin option requires a locally running IPFS node or a pinning service API key.
  - The --publish option requires a private key.
  - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher.
  - Without --rewrite, the manifest is written to the temporary cache. With it, the manifest is rewritten to the index folder.

Data models produced by this tool:

Other Options

All tools accept the following additional flags, although in some cases, they have no meaning.

  -v, --version         display the current version of the tool
      --output string   write the results to file 'fn' and return the filename
      --append          for --output command only append to instead of replace contents of file
      --file string     specify multiple sets of command line options in a file

Note: For the --file string option, you may place a series of valid command lines in a file using any valid flags. In some cases, this may significantly improve performance. A semi-colon at the start of any line makes it a comment.

Note: If you use --output --append option and at the same time the --file option, you may not switch export formats in the command file. For example, a command file with two different commands, one with --fmt csv and the other with --fmt json will produce both invalid CSV and invalid JSON.

Copyright (c) 2024, TrueBlocks, LLC. All rights reserved. Generated with goMaker.

Documentation

Overview

chunksPkg implements the chifra chunks command.

The chifra chunks routine provides tools for interacting with, checking the validity of, cleaning up, and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data on the index, Bloom filters, addresses, and appearances. While still in its early stages, this tool will eventually allow users to clean their local index, clean their remote index, study the indexes, etc. Stay tuned.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetChunkStats 

func GetChunkStats(chain, path string) (s types.ChunkStats, err error)

func ResetOptions 

func ResetOptions(testMode bool)

func RunChunks 

func RunChunks(cmd *cobra.Command, args []string) error

RunChunks handles the chunks command for the command line. Returns error only as per cobra.

func ServeChunks 

func ServeChunks(w http.ResponseWriter, r *http.Request) error

ServeChunks handles the chunks command for the API. Returns an error.

Types

type ChunksOptions 

type ChunksOptions struct {
	Mode       string                   `json:"mode,omitempty"`       // The type of data to process
	Blocks     []string                 `json:"blocks,omitempty"`     // An optional list of blocks to intersect with chunk ranges
	BlockIds   []identifiers.Identifier `json:"blockIds,omitempty"`   // Block identifiers
	Check      bool                     `json:"check,omitempty"`      // Check the manifest, index, or blooms for internal consistency
	Pin        bool                     `json:"pin,omitempty"`        // Pin the manifest or each index chunk and bloom
	Publish    bool                     `json:"publish,omitempty"`    // Publish the manifest to the Unchained Index smart contract
	Publisher  string                   `json:"publisher,omitempty"`  // For some query options, the publisher of the index
	Truncate   base.Blknum              `json:"truncate,omitempty"`   // Truncate the entire index at this block (requires a block identifier)
	Remote     bool                     `json:"remote,omitempty"`     // Prior to processing, retrieve the manifest from the Unchained Index smart contract
	Belongs    []string                 `json:"belongs,omitempty"`    // In index mode only, checks the address(es) for inclusion in the given index chunk
	Diff       bool                     `json:"diff,omitempty"`       // Compare two index portions (see notes)
	FirstBlock base.Blknum              `json:"firstBlock,omitempty"` // First block to process (inclusive)
	LastBlock  base.Blknum              `json:"lastBlock,omitempty"`  // Last block to process (inclusive)
	MaxAddrs   uint64                   `json:"maxAddrs,omitempty"`   // The max number of addresses to process in a given chunk
	Deep       bool                     `json:"deep,omitempty"`       // If true, dig more deeply during checking (manifest only)
	Rewrite    bool                     `json:"rewrite,omitempty"`    // For the --pin --deep mode only, writes the manifest back to the index folder (see notes)
	List       bool                     `json:"list,omitempty"`       // For the pins mode only, list the remote pins
	Unpin      bool                     `json:"unpin,omitempty"`      // For the pins mode only, if true reads local ./unpins file for valid CIDs and remotely unpins each (skips non-CIDs)
	Count      bool                     `json:"count,omitempty"`      // For certain modes only, display the count of records
	Tag        string                   `json:"tag,omitempty"`        // Visits each chunk and updates the headers with the supplied version string (vX.Y.Z-str)
	Sleep      float64                  `json:"sleep,omitempty"`      // For --remote pinning only, seconds to sleep between API calls
	Globals    globals.GlobalOptions    `json:"globals,omitempty"`    // The global options
	Conn       *rpc.Connection          `json:"conn,omitempty"`       // The connection to the RPC server
	BadFlag    error                    `json:"badFlag,omitempty"`    // An error flag if needed
	// EXISTING_CODE
	PublisherAddr base.Address `json:"-"`
}

ChunksOptions provides all command options for the chifra chunks command.

func ChunksFinishParseInternal 

func ChunksFinishParseInternal(w io.Writer, values url.Values) *ChunksOptions

func GetChunksOptions 

func GetChunksOptions(args []string, g *globals.GlobalOptions) *ChunksOptions

GetChunksOptions returns the options for this tool so other tools may use it.

func GetOptions 

func GetOptions() *ChunksOptions

func (*ChunksOptions) CheckDeep 

func (opts *ChunksOptions) CheckDeep(cacheMan *manifest.Manifest, report *types.ReportCheck) error

CheckDeep digs deep into the data. In `index` mode, it opens each index and checks that all addresses in the index return true when checked against its corresponding Bloom filter. In `manifest` mode, it checks that each IPFS hash in the manifest is actually pinned. The later requires a locally running IPFS node.

func (*ChunksOptions) CheckHashes 

func (opts *ChunksOptions) CheckHashes(cacheMan *manifest.Manifest, contractMan *manifest.Manifest, report *types.ReportCheck) error

CheckHashes looks at all the hashes in both the locally cached manifest and the manifest retrieved from the smart contract and simply checks the lengths of the bloom and index IPFS hashes.

func (*ChunksOptions) CheckInternal 

func (opts *ChunksOptions) CheckInternal(fileNames []string, blockNums []base.Blknum, report *types.ReportCheck) error

CheckInternal reads the header of each chunk on disc looking for the Magic number and the hash of the spec version for expected values.

func (*ChunksOptions) CheckManContents 

func (opts *ChunksOptions) CheckManContents(man *manifest.Manifest, report *types.ReportCheck) error

CheckManContents spins through the manifest and makes sure all the bloom and index CIDs are present. It does not check that the CIDs are available.

func (*ChunksOptions) CheckManifest 

func (opts *ChunksOptions) CheckManifest(arrayA, arrayB []string, report *types.ReportCheck) error

CheckManifest takes two arrays (either onDisc vs. LocalManifest, onDisc vs. RemoteManifest, or LocalManifest vs. RemoteManifest) and compares them for equality. If everything is up to date, all three arrays should be identical. Only the block ranges are in the arrays.

func (*ChunksOptions) CheckSequential 

func (opts *ChunksOptions) CheckSequential(fnArray, cacheArray, remoteArray []string, allowMissing bool, report *types.ReportCheck) error

CheckSequential checks that the range of blocks in each of three arrays (onDisc, LocalManifest, RemoteManifest) "cover" the range without gaps. (There may be a bug for allow_missing chains where gaps are allowed.) It also makes sure than snap-to-grids happen where they should and that non-snaps have at least appsPerChunks records and snaps have exactly appsPerChunks records or less.

func (*ChunksOptions) CheckSizes 

func (opts *ChunksOptions) CheckSizes(fileNames []string, blockNums []base.Blknum, cacheManifest *manifest.Manifest, remoteManifest *manifest.Manifest, report *types.ReportCheck) error

CheckSizes compares the file on disc to the file size suggested in the manifest

func (*ChunksOptions) CheckStaging 

func (opts *ChunksOptions) CheckStaging(lastBlock uint64, allow_missing bool, report *types.ReportCheck) error

CheckStaging checks the staging file which should be names first-second.txt

  1. Makes sure that finalized + 1 == first
  2. Makes sure the first < last
  3. Makes sure that the first block inside is == first if allow_missing == false, > otherwise
  4. Makes sure that the last block inside is == last if allow_missing == false, < otherwise

func (*ChunksOptions) CheckVersion 

func (opts *ChunksOptions) CheckVersion(fileNames []string, blockNums []base.Blknum, report *types.ReportCheck) error

CheckVersion reads the header of each chunk on disc looking for the Magic number and the hash of the spec version for expected values.

func (*ChunksOptions) ChunksInternal 

func (opts *ChunksOptions) ChunksInternal(rCtx *output.RenderCtx) error

ChunksInternal handles the internal workings of the chunks command. Returns an error.

func (*ChunksOptions) HandleAddresses 

func (opts *ChunksOptions) HandleAddresses(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleAppearances 

func (opts *ChunksOptions) HandleAppearances(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleBlooms 

func (opts *ChunksOptions) HandleBlooms(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleCheck 

func (opts *ChunksOptions) HandleCheck(rCtx *output.RenderCtx, blockNums []base.Blknum) error

HandleCheck looks at three different arrays: index files on disc, manifest on disc, and manifest in the smart contract. It tries to check these three sources for cosnsistency. Smart contract rules, so it is checked more thoroughly.

func (*ChunksOptions) HandleCounts 

func (opts *ChunksOptions) HandleCounts(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleDiff 

func (opts *ChunksOptions) HandleDiff(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleDump 

func (opts *ChunksOptions) HandleDump(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleIndex 

func (opts *ChunksOptions) HandleIndex(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleIndexBelongs 

func (opts *ChunksOptions) HandleIndexBelongs(rCtx *output.RenderCtx, blockNums []base.Blknum) error

HandleIndexBelongs displays the resolved records in a chunk given a single address

func (*ChunksOptions) HandleList 

func (opts *ChunksOptions) HandleList(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleManifest 

func (opts *ChunksOptions) HandleManifest(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandlePin 

func (opts *ChunksOptions) HandlePin(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandlePublish 

func (opts *ChunksOptions) HandlePublish(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleShow 

func (opts *ChunksOptions) HandleShow(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleStats 

func (opts *ChunksOptions) HandleStats(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleTag 

func (opts *ChunksOptions) HandleTag(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleTruncate 

func (opts *ChunksOptions) HandleTruncate(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) HandleUnpin 

func (opts *ChunksOptions) HandleUnpin(rCtx *output.RenderCtx, blockNums []base.Blknum) error

func (*ChunksOptions) String 

func (opts *ChunksOptions) String() string

String implements the Stringer interface

type CompareState 

type CompareState struct {
	// contains filtered or unexported fields
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.