Documentation
¶
Overview ¶
Package disk provides disk-based implementation of the storage.Store interface.
The disk.Store implementation uses an embedded key-value store to persist policies and data. Policy modules are stored as raw byte strings with one module per key. Data is mapped to the underlying key-value store with the assistance of caller-supplied "partitions". Partitions allow the caller to control the portions of the /data namespace that are mapped to individual keys. Operations that span multiple keys (e.g., a read against the entirety of /data) are more expensive than reads that target a specific key because the storage layer has to reconstruct the object from individual key-value pairs and page all of the data into memory. By supplying partitions that align with lookups in the policies, callers can optimize policy evaluation.
Partitions are specified as a set of storage paths (e.g., {/foo/bar} declares a single partition at /foo/bar). Each partition tells the store that values under the partition path should be mapped to individual keys. Values that fall outside of the partitions are stored at adjacent keys without further splitting. For example, given the partition set {/foo/bar}, /foo/bar/abcd and /foo/bar/efgh are be written to separate keys. All other values under /foo are not split any further (e.g., all values under /foo/baz would be written to a single key). Similarly, values that fall outside of partitions are stored under individual keys at the root (e.g., the full extent of the value at /qux would be stored under one key.) There is support for wildcards in partitions: {/foo/*} will cause /foo/bar/abc and /foo/buz/def to be written to separate keys. Multiple wildcards are supported (/tenants/*/users/*/bindings), and they can also appear at the end of a partition (/users/*).
All keys written by the disk.Store implementation are prefixed as follows:
/<schema_version>/<partition_version>/<type>
The <schema_version> value represents the version of the schema understood by this version of OPA. Currently this is always set to 1. The <partition_version> value represents the version of the partition layout supplied by the caller. Currently this is always set to 1. Currently, the disk.Store implementation only supports _additive_ changes to the partitioning layout, i.e., new partitions can be added as long as they do not overlap with existing unpartitioned data. The <type> value is either "data" or "policies" depending on the value being stored.
The disk.Store implementation attempts to be compatible with the inmem.store implementation however there are some minor differences:
* Writes that add partitioned values implicitly create an object hierarchy containing the value (e.g., `add /foo/bar/abcd` implicitly creates the structure `{"foo": {"bar": {"abcd": ...}}}`). This is unavoidable because of how nested /data values are mapped to key-value pairs.
* Trigger events do not include a set of changed paths because the underlying key-value store does not make them available.
Example (Store) ¶
package main
import (
"context"
"fmt"
"io/ioutil"
"os"
"github.com/open-policy-agent/opa/logging"
"github.com/open-policy-agent/opa/storage"
"github.com/open-policy-agent/opa/storage/disk"
"github.com/open-policy-agent/opa/util"
)
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {
ctx := context.Background()
// Create a temporary directory for store.
dir, err := ioutil.TempDir("", "opa_disk_example")
check(err)
// Cleanup temporary directory after finishing.
defer os.RemoveAll(dir)
// Create a new disk-based store.
store, err := disk.New(ctx, logging.NewNoOpLogger(), nil, disk.Options{
Dir: dir,
Partitions: []storage.Path{
storage.MustParsePath("/authz/tenants"),
},
})
check(err)
// Insert data into the store. The `storage.WriteOne` function automatically
// opens a write transaction, applies the operation, and commits the
// transaction in one-shot.
err = storage.WriteOne(ctx, store, storage.AddOp, storage.MustParsePath("/"), util.MustUnmarshalJSON([]byte(`{
"authz": {
"tenants": {
"acmecorp.openpolicyagent.org": {
"tier": "gold"
},
"globex.openpolicyagent.org" :{
"tier": "silver"
}
}
}
}`)))
check(err)
// Close the store so that it can be reopened.
err = store.Close(ctx)
check(err)
// Re-create the disk-based store using the same options.
store2, err := disk.New(ctx, logging.NewNoOpLogger(), nil, disk.Options{
Dir: dir,
Partitions: []storage.Path{
storage.MustParsePath("/authz/tenants"),
},
})
check(err)
// Read value persisted above and inspect the result.
value, err := storage.ReadOne(ctx, store2, storage.MustParsePath("/authz/tenants/acmecorp.openpolicyagent.org"))
check(err)
err = store2.Close(ctx)
check(err)
fmt.Println(value)
}
Output: map[tier:gold]
Index ¶
- Variables
- type Options
- type Store
- func (db *Store) Abort(ctx context.Context, txn storage.Transaction)
- func (db *Store) Close(context.Context) error
- func (db *Store) Commit(ctx context.Context, txn storage.Transaction) error
- func (db *Store) DeletePolicy(ctx context.Context, txn storage.Transaction, id string) error
- func (db *Store) GC(logger logging.Logger)
- func (db *Store) GetPolicy(ctx context.Context, txn storage.Transaction, id string) ([]byte, error)
- func (db *Store) ListPolicies(ctx context.Context, txn storage.Transaction) ([]string, error)
- func (db *Store) MakeDir(_ context.Context, txn storage.Transaction, path storage.Path) error
- func (db *Store) NewTransaction(ctx context.Context, params ...storage.TransactionParams) (storage.Transaction, error)
- func (db *Store) Read(ctx context.Context, txn storage.Transaction, path storage.Path) (interface{}, error)
- func (db *Store) Register(_ context.Context, txn storage.Transaction, config storage.TriggerConfig) (storage.TriggerHandle, error)
- func (db *Store) UpsertPolicy(ctx context.Context, txn storage.Transaction, id string, bs []byte) error
- func (db *Store) Write(ctx context.Context, txn storage.Transaction, op storage.PatchOp, ...) error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidPartitionPath = errors.New("invalid storage path")
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
Dir string // specifies directory to store data inside of
Partitions []storage.Path // data prefixes that enable efficient layout
Badger string // badger-internal configurables
}
Options contains parameters that configure the disk-based store.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store provides a disk-based implementation of the storage.Store interface.
func New ¶
func New(ctx context.Context, logger logging.Logger, prom prometheus.Registerer, opts Options) (*Store, error)
New returns a new disk-based store based on the provided options.
func (*Store) Abort ¶
func (db *Store) Abort(ctx context.Context, txn storage.Transaction)
Abort implements the storage.Store interface.
func (*Store) DeletePolicy ¶
DeletePolicy implements the storage.Policy interface.
func (*Store) ListPolicies ¶
ListPolicies implements the storage.Policy interface.
func (*Store) MakeDir ¶
MakeDir makes Store a storage.MakeDirer, to avoid the superfluous MakeDir steps -- MakeDir is implicit in the disk storage's data layout, since
{"foo": {"bar": {"baz": 10}}}
writes value `10` to key `/foo/bar/baz`.
Here, we only check if it's a write transaction, for consistency with other implementations, and do nothing.
func (*Store) NewTransaction ¶
func (db *Store) NewTransaction(ctx context.Context, params ...storage.TransactionParams) (storage.Transaction, error)
NewTransaction implements the storage.Store interface.
func (*Store) Read ¶
func (db *Store) Read(ctx context.Context, txn storage.Transaction, path storage.Path) (interface{}, error)
Read implements the storage.Store interface.
func (*Store) Register ¶
func (db *Store) Register(_ context.Context, txn storage.Transaction, config storage.TriggerConfig) (storage.TriggerHandle, error)
Register implements the storage.Trigger interface.
func (*Store) UpsertPolicy ¶
func (db *Store) UpsertPolicy(ctx context.Context, txn storage.Transaction, id string, bs []byte) error
UpsertPolicy implements the storage.Policy interface.
Source Files