dotc1z

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 24, 2023 License: Apache-2.0 Imports: 27 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidFile        = fmt.Errorf("c1z: invalid file")
	ErrMaxSizeExceeded    = errors.New("c1z: max decoded size exceeded, increase DecoderMaxDecodedSize")
	ErrWindowSizeExceeded = errors.New("c1z: window size exceeded, increase DecoderMaxMemory")
)
View Source 
var C1ZFileHeader = []byte("C1ZF\x00")

Functions

func C1ZFileCheckHeader 

func C1ZFileCheckHeader(f io.ReadSeeker) (bool, error)

C1ZFileCheckHeader reads len(C1ZFileHeader) bytes from the given io.ReadSeeker and compares them to C1ZFileHeader. Returns true if the header is valid. Returns any errors from Read() or Seek(). If a nil error is returned, the given io.ReadSeeker will be pointing to the first byte of the stream, and is suitable to be passed to NewC1ZFileDecoder.

func NewC1FileReader 

func NewC1FileReader(ctx context.Context, dbFilePath string) (connectorstore.Reader, error)

NewC1FileReader returns a connectorstore.Reader implementation for the given sqlite db file path.

func NewC1ZFileDecoder 

func NewC1ZFileDecoder(f io.Reader, opts ...DecoderOption) (io.ReadCloser, error)

NewC1ZFileDecoder wraps a given .c1z io.Reader that validates the .c1z and decompresses/decodes the underlying file. Defaults: 32MiB max memory and 2GiB max decoded size You must close the resulting io.ReadCloser when you are done, do not forget to close the given io.Reader if necessary.

func NewDecoder added in v0.0.24 

func NewDecoder(f io.Reader, opts ...DecoderOption) (*decoder, error)

NewDecoder wraps a given .c1z file io.Reader and returns an io.Reader for the underlying decoded/uncompressed file.

func ReadHeader added in v0.0.24 

func ReadHeader(reader io.Reader) error

ReadHeader reads len(C1ZFileHeader) bytes from the given io.Reader and compares them to C1ZFileHeader, returning an error if they don't match. If possible, ReadHeader will Seek() to the start of the stream before checking the header bytes. On return, the reader will be pointing to the first byte after the header.

Types

type C1File added in v0.0.24 

type C1File struct {
	// contains filtered or unexported fields
}

func NewC1File added in v0.0.24 

func NewC1File(ctx context.Context, dbFilePath string) (*C1File, error)

Returns a C1File instance for the given db filepath.

func NewC1ZFile added in v0.0.24 

func NewC1ZFile(ctx context.Context, outputFilePath string) (*C1File, error)

Returns a new C1File instance with its state stored at the provided filename.

func (*C1File) CheckpointSync added in v0.0.24 

func (c *C1File) CheckpointSync(ctx context.Context, syncToken string) error

func (*C1File) Cleanup added in v0.0.24 

func (c *C1File) Cleanup(ctx context.Context) error

func (*C1File) CloneSync added in v0.0.24 

func (c *C1File) CloneSync(ctx context.Context, outPath string, syncID string) error

CloneSync uses sqlite hackery to directly copy the pertinent rows into a new database. 1. Create a new empty sqlite database in a temp file 2. Open the c1z that we are cloning to get a db handle 3. Execute an ATTACH query to bring our empty sqlite db into the context of our db connection 4. Select directly from the cloned db and insert directly into the new database. 5. Close and save the new database as a c1z at the configured path.

func (*C1File) Close added in v0.0.24 

func (c *C1File) Close() error

Close ensures that the sqlite database is flushed to disk, and if any changes were made we update the original database with our changes.

func (*C1File) CurrentSyncStep added in v0.0.24 

func (c *C1File) CurrentSyncStep(ctx context.Context) (string, error)

func (*C1File) DeleteSyncRun added in v0.0.24 

func (c *C1File) DeleteSyncRun(ctx context.Context, syncID string) error

DeleteSyncRun removes all the objects with a given syncID from the database.

func (*C1File) EndSync added in v0.0.24 

func (c *C1File) EndSync(ctx context.Context) error

EndSync updates the current sync_run row with the end time, and removes any other objects that don't have the current sync ID.

func (*C1File) GetAsset added in v0.0.24 

func (c *C1File) GetAsset(ctx context.Context, request *v2.AssetServiceGetAssetRequest) (string, io.Reader, error)

GetAsset fetches the specified asset from the database, and returns the content type and an io.Reader for the caller to read the asset from.

func (*C1File) GetEntitlement added in v0.0.24 

func (c *C1File) GetEntitlement(ctx context.Context, request *reader_v2.EntitlementsReaderServiceGetEntitlementRequest) (*reader_v2.EntitlementsReaderServiceGetEntitlementResponse, error)

func (*C1File) GetGrant added in v0.0.24 

func (c *C1File) GetGrant(ctx context.Context, request *reader_v2.GrantsReaderServiceGetGrantRequest) (*reader_v2.GrantsReaderServiceGetGrantResponse, error)

func (*C1File) GetResource added in v0.0.24 

func (c *C1File) GetResource(ctx context.Context, request *reader_v2.ResourcesReaderServiceGetResourceRequest) (*reader_v2.ResourcesReaderServiceGetResourceResponse, error)

func (*C1File) GetResourceType added in v0.0.24 

func (c *C1File) GetResourceType(ctx context.Context, request *reader_v2.ResourceTypesReaderServiceGetResourceTypeRequest) (*reader_v2.ResourceTypesReaderServiceGetResourceTypeResponse, error)

func (*C1File) LatestFinishedSync added in v0.0.31 

func (c *C1File) LatestFinishedSync(ctx context.Context) (string, error)

func (*C1File) LatestSyncID added in v0.0.24 

func (c *C1File) LatestSyncID(ctx context.Context) (string, error)

func (*C1File) ListEntitlements added in v0.0.24 

func (c *C1File) ListEntitlements(ctx context.Context, request *v2.EntitlementsServiceListEntitlementsRequest) (*v2.EntitlementsServiceListEntitlementsResponse, error)

func (*C1File) ListGrants added in v0.0.24 

func (c *C1File) ListGrants(ctx context.Context, request *v2.GrantsServiceListGrantsRequest) (*v2.GrantsServiceListGrantsResponse, error)

func (*C1File) ListGrantsForEntitlement added in v0.0.24 

func (c *C1File) ListGrantsForEntitlement(
	ctx context.Context,
	request *reader_v2.GrantsReaderServiceListGrantsForEntitlementRequest,
) (*reader_v2.GrantsReaderServiceListGrantsForEntitlementResponse, error)

func (*C1File) ListGrantsForResourceType added in v0.0.24 

func (c *C1File) ListGrantsForResourceType(
	ctx context.Context,
	request *reader_v2.GrantsReaderServiceListGrantsForResourceTypeRequest,
) (*reader_v2.GrantsReaderServiceListGrantsForResourceTypeResponse, error)

func (*C1File) ListResourceTypes added in v0.0.24 

func (c *C1File) ListResourceTypes(ctx context.Context, request *v2.ResourceTypesServiceListResourceTypesRequest) (*v2.ResourceTypesServiceListResourceTypesResponse, error)

func (*C1File) ListResources added in v0.0.24 

func (c *C1File) ListResources(ctx context.Context, request *v2.ResourcesServiceListResourcesRequest) (*v2.ResourcesServiceListResourcesResponse, error)

func (*C1File) ListSyncRuns added in v0.0.24 

func (c *C1File) ListSyncRuns(ctx context.Context, pageToken string, pageSize int) ([]*syncRun, string, error)

func (*C1File) PreviousSyncID added in v0.0.24 

func (c *C1File) PreviousSyncID(ctx context.Context) (string, error)

func (*C1File) PutAsset added in v0.0.24 

func (c *C1File) PutAsset(ctx context.Context, assetRef *v2.AssetRef, contentType string, data []byte) error

PutAsset stores the given asset in the database.

func (*C1File) PutEntitlement added in v0.0.24 

func (c *C1File) PutEntitlement(ctx context.Context, entitlement *v2.Entitlement) error

func (*C1File) PutGrant added in v0.0.24 

func (c *C1File) PutGrant(ctx context.Context, grant *v2.Grant) error

func (*C1File) PutResource added in v0.0.24 

func (c *C1File) PutResource(ctx context.Context, resource *v2.Resource) error

func (*C1File) PutResourceType added in v0.0.24 

func (c *C1File) PutResourceType(ctx context.Context, resourceType *v2.ResourceType) error

func (*C1File) StartSync added in v0.0.24 

func (c *C1File) StartSync(ctx context.Context) (string, bool, error)

StartSync generates a sync ID to be associated with all objects discovered during this run.

func (*C1File) Stats added in v0.0.24 

func (c *C1File) Stats(ctx context.Context) (map[string]int64, error)

Stats introspects the database and returns the count of objects for the given sync run.

func (*C1File) ViewSync added in v0.0.24 

func (c *C1File) ViewSync(ctx context.Context, syncID string) error

type DecoderOption added in v0.0.24 

type DecoderOption func(*decoderOptions) error

DecoderOption is an option for creating a decoder.

func WithContext 

func WithContext(ctx context.Context) DecoderOption

WithContext sets a context, when cancelled, will cause subequent calls to Read() to return ctx.Error().

func WithDecoderMaxDecodedSize 

func WithDecoderMaxDecodedSize(n uint64) DecoderOption

WithDecoderMaxDecodedSize sets the maximum size of the decoded stream. This can be used to cap the resulting decoded stream size. Maximum is 1 << 63 bytes. Default is 1GiB.

func WithDecoderMaxMemory 

func WithDecoderMaxMemory(n uint64) DecoderOption

WithDecoderMaxMemory sets the maximum window size for streaming operations. This can be used to control memory usage of potentially hostile content. Maximum is 1 << 63 bytes. Default is 32MiB.

Source Files

View all Source files

Directories

Path Synopsis
local
s3

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.