README
¶
The grailbio/base project includes all the packages used by many other grailbio Go packages:
-
API documentation: godoc.org/github.com/grailbio/base
-
Issue tracker: github.com/grailbio/base/issues
-
-
recordio: encrypted and compressed record oriented files with indexing support
-
file: unified file API for the local file system and S3
-
digest: common in-memory and serialized representation of digests
-
data: support for measuring and displaying quantities of data
-
intervalmap: fast interval tree
-
limiter: concurrency limiter with context support
-
traverse: concurrent and parallel slice traversal
-
state: file-based state management
-
syncqueue: various flavors of producer-consumer queues
-
unsafe: conversion from []byte to string, etc.
-
compress/libdeflate: efficient block compression/decompression
-
bitset: []uintptr bitset support
-
simd: fast operations on []byte
-
tsv: simple and efficient TSV writer
-
cloud/spotadvisor: provides an interface for fetching and utilizing AWS Spot Advisor data
-
cloud/spotfeed: provides interfaces for interacting with the AWS spot data feed format for files hosted on S3
Directories
¶
| Path | Synopsis |
|---|---|
|
Package admit contains utilities for admission control.
|
Package admit contains utilities for admission control. |
|
Package backgroundcontext manages the singleton v23 context.
|
Package backgroundcontext manages the singleton v23 context. |
|
Package bitset provides support for treating a []uintptr as a bitset.
|
Package bitset provides support for treating a []uintptr as a bitset. |
|
cloud
|
|
|
awssession
Package awssession provides a simple way to obtain AWS session.Session using GRAIL tickets.
|
Package awssession provides a simple way to obtain AWS session.Session using GRAIL tickets. |
|
spotadvisor
Package spotadvisor provides an interface for utilizing spot instance interrupt rate data and savings data from AWS.
|
Package spotadvisor provides an interface for utilizing spot instance interrupt rate data and savings data from AWS. |
|
spotfeed
Package spotfeed is used for querying spot-data-feeds provided by AWS.
|
Package spotfeed is used for querying spot-data-feeds provided by AWS. |
|
cmd
|
|
|
gofat
Command gofat is a simple utility to make fat binaries in the fatbin format (see github.com/grailbio/base/fatbin).
|
Command gofat is a simple utility to make fat binaries in the fatbin format (see github.com/grailbio/base/fatbin). |
|
grail-access
Command grail-access creates Vanadium credentials (also called principals) using either Google ID tokens (the default) or the AWS IAM role attached to an EC2 instance (requested using the '-ec2' flag).
|
Command grail-access creates Vanadium credentials (also called principals) using either Google ID tokens (the default) or the AWS IAM role attached to an EC2 instance (requested using the '-ec2' flag). |
|
grail-access/remote
Package remote implements sending (and receiving) of blessings to remote machines over SSH.
|
Package remote implements sending (and receiving) of blessings to remote machines over SSH. |
|
grail-fuse/gfs
Package gfs implements FUSE on top oh grailfile.
|
Package gfs implements FUSE on top oh grailfile. |
|
grail-role
Command role creates Vanadium principals for a Vanadium role account.
|
Command role creates Vanadium principals for a Vanadium role account. |
|
grail-role-group
Usage:
|
Usage: |
|
grail-ssh
Command that simplifies connecting to GRAIL systems using ssh with ssh certificates for authentication.
|
Command that simplifies connecting to GRAIL systems using ssh with ssh certificates for authentication. |
|
grail-ticket
Command grail-ticket retrieves a ticket from a ticket-server.
|
Command grail-ticket retrieves a ticket from a ticket-server. |
|
ticket-server
Command ticket-server runs a Vanadium server that provides restricted access to tickets.
|
Command ticket-server runs a Vanadium server that provides restricted access to tickets. |
|
Package cmdutil provides utility routines for implementing command line tools.
|
Package cmdutil provides utility routines for implementing command line tools. |
|
interactive
Package interactive is switching defaults for logging to not output anything to stderr.
|
Package interactive is switching defaults for logging to not output anything to stderr. |
|
common
|
|
|
Package compress provides convenience functions for creating compressors and uncompressors based on filenames.
|
Package compress provides convenience functions for creating compressors and uncompressors based on filenames. |
|
Package config is used to configure software systems.
|
Package config is used to configure software systems. |
|
http
Package http defines profile providers for local HTTP servers.
|
Package http defines profile providers for local HTTP servers. |
|
crypto
|
|
|
encryption
Package encryption provides support for encrypting and decrypting data and files with granular key management.
|
Package encryption provides support for encrypting and decrypting data and files with granular key management. |
|
Package data provides functionality for measuring and displaying data quantities.
|
Package data provides functionality for measuring and displaying data quantities. |
|
diagnostic
|
|
|
dump
Package dump provides the endpoint "debug/dump", registered with http.DefaultServeMux, which returns a dump of useful diagnostic information as a tarball.
|
Package dump provides the endpoint "debug/dump", registered with http.DefaultServeMux, which returns a dump of useful diagnostic information as a tarball. |
|
Package digest provides a generalized representation for digests computed with cryptographic hash functions.
|
Package digest provides a generalized representation for digests computed with cryptographic hash functions. |
|
Package errors implements an error type that defines standard interpretable error codes for common error conditions.
|
Package errors implements an error type that defines standard interpretable error codes for common error conditions. |
|
Package eventlog provides logging of semi-structured events, particularly in service of downstream analysis, e.g.
|
Package eventlog provides logging of semi-structured events, particularly in service of downstream analysis, e.g. |
|
Package fatbin implements a simple fat binary format, and provides facilities for creating fat binaries and accessing its variants.
|
Package fatbin implements a simple fat binary format, and provides facilities for creating fat binaries and accessing its variants. |
|
Package file provides basic file operations across multiple file-system types.
|
Package file provides basic file operations across multiple file-system types. |
|
fsnode
fsnode represents a filesystem as a directed graph (probably a tree for many implementations).
|
fsnode represents a filesystem as a directed graph (probably a tree for many implementations). |
|
fsnodefuse
fsnodefuse implements github.com/hanwen/go-fuse/v2/fs for fsnode.T. It's a work-in-progress.
|
fsnodefuse implements github.com/hanwen/go-fuse/v2/fs for fsnode.T. It's a work-in-progress. |
|
s3file
Package s3file implements grail file interface for S3.
|
Package s3file implements grail file interface for S3. |
|
s3file/internal/cmd/resolvetest
resolvetest simply resolves a hostname at an increasing time interval to observe the diversity in DNS lookup addresses for the host.
|
resolvetest simply resolves a hostname at an increasing time interval to observe the diversity in DNS lookup addresses for the host. |
|
Package flock implements a simple POSIX file-based advisory lock.
|
Package flock implements a simple POSIX file-based advisory lock. |
|
Package grail contains the Init function that all programs are expected to call.
|
Package grail contains the Init function that all programs are expected to call. |
|
biofs/biofseventlog
biofseventlog creates usage events for biofs, a GRAIL-internal program.
|
biofseventlog creates usage events for biofs, a GRAIL-internal program. |
|
gtl
|
|
|
tests
Code generated by generate_randomized_freepool.py.
|
Code generated by generate_randomized_freepool.py. |
|
Package intervalmap stores a set of (potentially overlapping) intervals.
|
Package intervalmap stores a set of (potentially overlapping) intervals. |
|
ioctx adds context.Context to io APIs.
|
ioctx adds context.Context to io APIs. |
|
fsctx
fsctx adds context.Context to io/fs APIs.
|
fsctx adds context.Context to io/fs APIs. |
|
Package iofmt contains various IO utilities used for formatting output.
|
Package iofmt contains various IO utilities used for formatting output. |
|
Package limiter implements a concurrency limiter with support for contexts.
|
Package limiter implements a concurrency limiter with support for contexts. |
|
Package log provides simple level logging.
|
Package log provides simple level logging. |
|
Package logio implements a failure-tolerant log, typically used as a write-ahead log.
|
Package logio implements a failure-tolerant log, typically used as a write-ahead log. |
|
Package mapio implements a sorted, on-disk map, similar to the SSTable data structure used in Bigtable [1], Cassandra [2], and others.
|
Package mapio implements a sorted, on-disk map, similar to the SSTable data structure used in Bigtable [1], Cassandra [2], and others. |
|
Package must provides a handful of functions to express fatal assertions in Go programs.
|
Package must provides a handful of functions to express fatal assertions in Go programs. |
|
Package psort includes functions for parallel sorting.
|
Package psort includes functions for parallel sorting. |
|
Package recordio implements the recordio file format.
|
Package recordio implements the recordio file format. |
|
recordioflate
Package recordioflate provides the "flate" transformer.
|
Package recordioflate provides the "flate" transformer. |
|
recordioiov
Package recordioiov provides utility functions for dealing with [][]bytes, used by recordio transformers.
|
Package recordioiov provides utility functions for dealing with [][]bytes, used by recordio transformers. |
|
Package retry contains utilities for implementing retry logic.
|
Package retry contains utilities for implementing retry logic. |
|
security
|
|
|
identity
Package identity defines interfaces for Vanadium identity providers.
|
Package identity defines interfaces for Vanadium identity providers. |
|
keycrypt
Package keycrypt implements an API for storing and retrieving opaque blobs of data stored in a secure fashion.
|
Package keycrypt implements an API for storing and retrieving opaque blobs of data stored in a secure fashion. |
|
keycrypt/file
Package file implements a file-based keycrypt.
|
Package file implements a file-based keycrypt. |
|
keycrypt/kms
Package kms implements a Keycrypt using AWS's KMS service and S3.
|
Package kms implements a Keycrypt using AWS's KMS service and S3. |
|
ssh/certificateauthority
Package certificateauthority implements an x509 certificate authority.
|
Package certificateauthority implements an x509 certificate authority. |
|
tls/certificateauthority
Package certificateauthority implements an x509 certificate authority.
|
Package certificateauthority implements an x509 certificate authority. |
|
Package shutdown implements a global process shutdown mechanism.
|
Package shutdown implements a global process shutdown mechanism. |
|
Package simd provides access to SIMD-based implementations of several common operations on byte arrays which the compiler cannot be trusted to autovectorize within the next several years.
|
Package simd provides access to SIMD-based implementations of several common operations on byte arrays which the compiler cannot be trusted to autovectorize within the next several years. |
|
Package state implements atomic file-based state management with support for advisory locking.
|
Package state implements atomic file-based state management with support for advisory locking. |
|
Package stateio implements persistent state mechanism based on log files that interleave indexed state snapshots with state updates.
|
Package stateio implements persistent state mechanism based on log files that interleave indexed state snapshots with state updates. |
|
Package status provides facilities for reporting statuses from a number of tasks working towards a common goal.
|
Package status provides facilities for reporting statuses from a number of tasks working towards a common goal. |
|
stress
|
|
|
oom
Package oom contains a single function to trigger Linux kernel OOMs.
|
Package oom contains a single function to trigger Linux kernel OOMs. |
|
sync
|
|
|
once
Package once contains utilities for managing actions that must be performed exactly once.
|
Package once contains utilities for managing actions that must be performed exactly once. |
|
Package traverse provides primitives for concurrent and parallel traversal of slices or user-defined collections.
|
Package traverse provides primitives for concurrent and parallel traversal of slices or user-defined collections. |
|
Package tsv provides a simple TSV writer which takes care of number->string conversions and tabs, and is far more performant than fmt.Fprintf (thanks to use of strconv.Append{Uint,Float}).
|
Package tsv provides a simple TSV writer which takes care of number->string conversions and tabs, and is far more performant than fmt.Fprintf (thanks to use of strconv.Append{Uint,Float}). |
|
Package vcontext creates a singleton vanadium Context object.
|
Package vcontext creates a singleton vanadium Context object. |
|
web
|
|
|
webutil
Package webutil implements utility routines for web-related operations and applications.
|
Package webutil implements utility routines for web-related operations and applications. |
|
Package writehash provides a set of utility functions to hash common types into hashes.
|
Package writehash provides a set of utility functions to hash common types into hashes. |
Click to show internal directories.
Click to hide internal directories.