Documentation
¶
Overview ¶
Example (E2e) ¶
ExampleE2e showcases a more complex integration.
package main
import (
"context"
"fmt"
"github.com/alcionai/clues"
"github.com/alcionai/corso/src/pkg/fault"
)
var ctx = context.Background()
func connectClient() error { return nil }
func dependencyCall() error { return nil }
func getData() ([]string, error) { return nil, nil }
func storeData([]string, *fault.Bus) {}
type mockOper struct {
Errors *fault.Bus
}
func newOperation() mockOper { return mockOper{fault.New(true)} }
func (m mockOper) Run() *fault.Bus { return m.Errors }
func main() {
oper := newOperation()
// imagine that we're a user, calling into corso SDK.
// (fake funcs used here to minimize example bloat)
//
// The operation is our controller, we expect it to
// generate a new fault.Bus when constructed, and
// to return that struct when we call Run()
errs := oper.Run()
// Let's investigate what went on inside. Since we're at
// the top of our controller, and returning a fault.Bus,
// all the error handlers set the Fail() case.
/* Run() */
func() *fault.Bus {
if err := connectClient(); err != nil {
// Fail() here; we're top level in the controller
// and this is a non-recoverable issue
return oper.Errors.Fail(err)
}
data, err := getData()
if err != nil {
return oper.Errors.Fail(err)
}
// storeData will aggregate iterated errors into
// oper.Errors.
storeData(data, oper.Errors)
// return oper.Errors here, in part to ensure it's
// non-nil, and because we don't know if we've
// aggregated any iterated errors yet.
return oper.Errors
}()
// What about the lower level handling? storeData didn't
// return an error, so what's happening there?
/* storeData */
err := func(data []any, errs *fault.Bus) error {
// this is downstream in our code somewhere
storer := func(a any) error {
if err := dependencyCall(); err != nil {
// we're not passing in or calling fault.Bus here,
// because this isn't the iteration handler, it's just
// a regular error.
return clues.Wrap(err, "dependency")
}
return nil
}
el := errs.Local()
for _, d := range data {
if el.Failure() != nil {
break
}
if err := storer(d); err != nil {
// Since we're at the top of the iteration, we need
// to add each error to the fault.localBus struct.
el.AddRecoverable(ctx, err)
}
}
// at the end of the func, we need to return local.Failure()
// just in case the local bus promoted an error to the failure
// position. If we don't return it like normal error handling,
// then we'll lose scope of that error.
return el.Failure()
}(nil, nil)
if err != nil {
fmt.Println("errored", err)
}
// At the end of the oper.Run, when returning to the interface
// layer, we investigate the results.
if errs.Failure() != nil {
// handle the primary error
fmt.Println("err occurred", errs.Failure())
}
for _, err := range errs.Recovered() {
// handle each recoverable error
fmt.Println("recoverable err occurred", err)
}
}
Output:
Index ¶
- Constants
- func UnmarshalErrorsTo(e *Errors) func(io.ReadCloser) error
- type AddSkipper
- type Alert
- type Bus
- func (e *Bus) AddAlert(ctx context.Context, a *Alert)
- func (e *Bus) AddRecoverable(ctx context.Context, err error)
- func (e *Bus) AddSkip(ctx context.Context, s *Skipped)
- func (e *Bus) Alerts() []Alert
- func (e *Bus) Errors() *Errors
- func (e *Bus) Fail(err error) *Bus
- func (e *Bus) FailFast() bool
- func (e *Bus) Failure() error
- func (e *Bus) ItemsAndRecovered() ([]Item, []error)
- func (e *Bus) Local() *Bus
- func (e *Bus) Recovered() []error
- func (e *Bus) Skipped() []Skipped
- type Errors
- type Item
- type ItemType
- type SkipCause
- type Skipped
- func ContainerSkip(cause SkipCause, namespace, id, name string, addtl map[string]any) *Skipped
- func EmailSkip(cause SkipCause, user, id string, addtl map[string]any) *Skipped
- func FileSkip(cause SkipCause, namespace, id, name string, addtl map[string]any) *Skipped
- func OwnerSkip(cause SkipCause, namespace, id, name string, addtl map[string]any) *Skipped
Examples ¶
Constants ¶
const ( AddtlCreatedBy = "created_by" AddtlLastModBy = "last_modified_by" AddtlContainerID = "container_id" AddtlContainerName = "container_name" AddtlContainerPath = "container_path" AddtlMalwareDesc = "malware_description" )
const (
AlertPreviousPathCollision = "previous_path_collision"
)
const LabelForceNoBackupCreation = "label_forces_no_backup_creations"
temporary hack identifier see: https://github.com/alcionai/corso/pull/2510#discussion_r1113532530 TODO: https://github.com/alcionai/corso/issues/4003
Variables ¶
This section is empty.
Functions ¶
func UnmarshalErrorsTo ¶
func UnmarshalErrorsTo(e *Errors) func(io.ReadCloser) error
UnmarshalErrorsTo produces a func that complies with the unmarshaller type in streamStore.
Types ¶
type AddSkipper ¶
AddSkipper presents an interface that allows callers to write additional skipped items to the complying struct.
type Alert ¶
Alerts are informational-only notifications. The purpose of alerts is to provide a means of end-user communication about important events without needing to generate runtime failures or recoverable errors. When generating an alert, no other fault feature (failure, recoverable, skip, etc) should be in use. IE: Errors do not also get alerts, since the error itself is a form of end-user communication already.
func (Alert) Headers ¶
Headers returns the human-readable names of properties of a skipped Item for printing out to a terminal.
func (Alert) MinimumPrintable ¶
type Bus ¶
type Bus struct {
// contains filtered or unexported fields
}
func New ¶
New constructs a new error with default values in place.
Example ¶
ExampleNew highlights assumptions and best practices for generating fault.Bus structs.
package main
import (
"context"
"github.com/alcionai/corso/src/pkg/fault"
)
var ctrl any
type mockController struct {
errors any
}
func main() {
// New fault.Bus instances should only get generated during initialization.
// Such as when starting up a new Backup or Restore Operation.
// Configuration (eg: failFast) is set during construction and cannot
// be updated.
ctrl = mockController{
errors: fault.New(false),
}
}
Output:
func (*Bus) AddAlert ¶
AddAlert appends a record of an Alert message to the fault bus. Importantly, alerts are not errors, exceptions, or skipped items. An alert should only be generated if no other fault functionality is in use, but that we still want the end user to clearly and plainly receive a notification about a runtime event.
Example ¶
ExampleBus_AddAlert showcases when to use AddAlert.
package main
import (
"context"
"fmt"
"github.com/alcionai/corso/src/pkg/fault"
)
var ctx = context.Background()
func main() {
errs := fault.New(false)
// Some events should be communicated to the end user without recording an
// error to the operation. Logs aren't sufficient because we don't promote
// log messages to the terminal. But errors and skips are too heavy and hacky
// to use. In these cases, we can create informational Alerts.
//
// Only the message gets shown to the user. But since we're persisting this
// data along with the backup details and other fault info, we have the option
// of packing any other contextual data that we want.
errs.AddAlert(ctx, fault.NewAlert(
"something important happened!",
"deduplication-namespace",
"file-id",
"file-name",
map[string]any{"foo": "bar"}))
// later on, after processing, end users can scrutinize the alerts.
fmt.Println(errs.Alerts()[0].String())
// Alert: something important happened!
}
Output:
func (*Bus) AddRecoverable ¶
AddRecoverable appends the error to the slice of recoverable errors (ie: bus.recoverable). If failFast is true, the first added error will get copied to bus.failure, causing the bus to identify as non-recoverably failed.
Example ¶
ExampleBus_AddRecoverable describes the assumptions and best practices for aggregating iterable or recoverable errors.
package main
import (
"context"
"github.com/alcionai/clues"
"github.com/alcionai/corso/src/pkg/fault"
)
var (
items = []string{}
ctx = context.Background()
)
func getIthItem(i int) error { return nil }
func main() {
errs := fault.New(false)
// AddRecoverable() is used to record any recoverable error.
//
// What counts as a recoverable error? That's up to the given
// implementation. Normally, it's an inability to process one
// of many items within an iteration (ex: couldn't download 1 of
// 1000 emails). But just because an error occurred during a loop
// doesn't mean it's recoverable, ex: a failure to retrieve the next
// page when accumulating a batch of resources isn't usually
// recoverable. The choice is always up to the function at hand.
//
// AddRecoverable() should only get called as the top-most location
// of error handling within the recoverable process. Child functions
// should stick to normal golang error handling and expect the upstream
// controller to call AddRecoverable() for you.
for i := range items {
clientBasedGetter := func(i int) error {
if err := getIthItem(i); err != nil {
// lower level calls don't AddRecoverable to the fault.Bus.
// they stick to normal golang error handling.
return clues.Wrap(err, "dependency")
}
return nil
}
if err := clientBasedGetter(i); err != nil {
// Here at the top of the loop is the correct place
// to aggregate the error using fault.
// Side note: technically, you should use a local bus
// here (see below) instead of errs.
errs.AddRecoverable(ctx, err)
}
}
// Iteration should exit anytime the fault failure is non-nil.
// fault.Bus does not expose the failFast flag directly. Instead,
// when failFast is true, errors from AddRecoverable() automatically
// promote to the Failure() spot. Recoverable handling only needs to
// check the errs.Failure(). If it is non-nil, then the loop should break.
for i := range items {
if errs.Failure() != nil {
// if failFast == true errs.AddRecoverable() was called,
// we'll catch the error here.
break
}
if err := getIthItem(i); err != nil {
errs.AddRecoverable(ctx, err)
}
}
}
Output:
func (*Bus) AddSkip ¶
AddSkip appends a record of a Skipped item to the fault bus. Importantly, skipped items are not the same as recoverable errors. An item should only be skipped under the following conditions. All other cases should be handled as errors. 1. The conditions for skipping the item are well-known and well-documented. End users need to be able to understand both the conditions and identifications of skips. 2. Skipping avoids a permanent and consistent failure. If the underlying reason is transient or otherwise recoverable, the item should not be skipped.
Example ¶
ExampleBus_AddSkip showcases when to use AddSkip instead of an error.
package main
import (
"context"
"fmt"
"github.com/alcionai/corso/src/pkg/fault"
)
var ctx = context.Background()
func main() {
errs := fault.New(false)
// Some conditions cause well-known problems that we want Corso to skip
// over, instead of error out. An initial case is when Graph API identifies
// a file as containing malware. We can't download the file: it'll always
// error. Our only option is to skip it.
errs.AddSkip(ctx, fault.FileSkip(
fault.SkipMalware,
"deduplication-namespace",
"file-id",
"file-name",
map[string]any{"foo": "bar"}))
// later on, after processing, end users can scrutinize the skipped items.
fmt.Println(errs.Skipped()[0].String())
}
Output: skipped processing file: malware_detected
func (*Bus) Alerts ¶
Alerts returns the slice of alerts generated during runtime. If the bus is a local alerts, this only returns the local failure, and will not return parent data.
func (*Bus) Errors ¶
Errors returns the plain record of errors that were aggregated within a fult Bus.
func (*Bus) Fail ¶
Fail sets the non-recoverable error (ie: bus.failure) in the bus. If a failure error is already present, the error gets added to the recoverable slice for purposes of tracking.
Example ¶
ExampleBus_Fail describes the assumptions and best practices for setting the Failure error.
package main
import (
"fmt"
"github.com/alcionai/clues"
"github.com/alcionai/corso/src/pkg/fault"
)
func connectClient() error { return nil }
func dependencyCall() error { return nil }
func main() {
errs := fault.New(false)
// Fail() is used to record non-recoverable errors.
//
// Fail() should only get called in the last step before returning
// a fault.Bus from a controller. In all other cases, you
// can stick to standard golang error handling and expect some upstream
// controller to call Fail() for you (if necessary).
topLevelHandler := func(errs *fault.Bus) *fault.Bus {
if err := connectClient(); err != nil {
return errs.Fail(err)
}
return errs
}
if errs := topLevelHandler(errs); errs.Failure() != nil {
fmt.Println(errs.Failure())
}
// Only the top-most func in the stack should set the failure.
// IE: Fail() is not Wrap(). In lower levels, errors should get
// wrapped and returned like normal, and only handled by fault
// at the end.
lowLevelCall := func() error {
if err := dependencyCall(); err != nil {
// wrap here, deeper into the stack
return clues.Wrap(err, "dependency")
}
return nil
}
if err := lowLevelCall(); err != nil {
// fail here, at the top of the stack
errs.Fail(err)
}
}
Output:
func (*Bus) Failure ¶
Failure returns the primary error. If not nil, this indicates the operation exited prior to completion. If the bus is a local instance, this only returns the local failure, and will not return parent data.
Example ¶
ExampleBus_Failure describes retrieving the non-recoverable error.
package main
import (
"context"
"fmt"
"github.com/alcionai/clues"
"github.com/alcionai/corso/src/pkg/fault"
)
var ctx = context.Background()
func main() {
errs := fault.New(false)
errs.Fail(clues.New("catastrophe"))
// Failure() returns the primary failure.
err := errs.Failure()
fmt.Println(err)
// if multiple Failures occur, each one after the first gets
// added to the Recoverable slice as an overflow measure.
errs.Fail(clues.New("another catastrophe"))
errSl := errs.Recovered()
for _, e := range errSl {
fmt.Println(e)
}
// If Failure() is nil, then you can assume the operation completed.
// A complete operation is not necessarily an error-free operation.
// Recoverable errors may still have been added using AddRecoverable(ctx, err).
// Make sure you check both.
// If failFast is set to true, then the first recoerable error Added gets
// promoted to the Err() position.
errs = fault.New(true)
errs.AddRecoverable(ctx, clues.New("not catastrophic, but still becomes the Failure()"))
err = errs.Failure()
fmt.Println(err)
}
Output: catastrophe another catastrophe not catastrophic, but still becomes the Failure()
func (*Bus) ItemsAndRecovered ¶
ItemsAndRecovered returns the items that failed along with other recoverable errors
func (*Bus) Local ¶
Local constructs a new bus with a local reference to handle error aggregation in a constrained scope. This allows the caller to review recoverable errors and failures within only the current codespace, as opposed to the global set of errors. The function that spawned the local bus should always return `bus.Failure()` to ensure that hard failures are propagated back upstream.
Example ¶
package main
import (
"context"
"fmt"
"github.com/alcionai/corso/src/pkg/fault"
)
var (
items = []string{}
ctx = context.Background()
)
func getIthItem(i int) error { return nil }
func main() {
// It is common for Corso to run operations in parallel,
// and for iterations to be nested within iterations. To
// avoid mistakenly returning an error that was sourced
// from some other async iteration, recoverable instances
// are aggrgated into a Local.
errs := fault.New(false)
el := errs.Local()
err := func() error {
for i := range items {
if el.Failure() != nil {
break
}
if err := getIthItem(i); err != nil {
// instead of calling errs.AddRecoverable(ctx, err), we call the
// local bus's Add method. The error will still get
// added to the errs.Recovered() set. But if this err
// causes the run to fail, only this local bus treats
// it as the causal failure.
el.AddRecoverable(ctx, err)
}
}
return el.Failure()
}()
if err != nil {
// handle the Failure() that appeared in the local bus.
fmt.Println("failure occurred", errs.Failure())
}
}
Output:
func (*Bus) Recovered ¶
Recovered returns the slice of errors that occurred in recoverable points of processing. This is often during iteration where a single failure (ex: retrieving an item), doesn't require the entire process to end. If the bus is a local instance, this only returns the local recovered errors, and will not return parent data.
type Errors ¶
type Errors struct {
// Failure identifies a non-recoverable error. This includes
// non-start cases (ex: cannot connect to client), hard-
// stop issues (ex: credentials expired) or conscious exit
// cases (ex: iteration error + failFast config).
Failure *clues.ErrCore `json:"failure"`
// Recovered is the set of NON-Item errors that accumulated
// through a runtime under best-effort processing conditions.
// They imply that an error occurred, but the process was able
// to move on and complete afterwards. Any error that can be
// serialized to a fault.Item is found in the Items set instead.
Recovered []*clues.ErrCore `json:"recovered"`
// Items are the reduction of all errors (both the failure and the
// recovered values) in the Errors struct into a slice of items,
// deduplicated by their Namespace + ID.
Items []Item `json:"items"`
// Skipped is the accumulation of skipped items. Skipped items
// are not errors themselves, but instead represent some permanent
// inability to process an item, due to a well-known cause.
Skipped []Skipped `json:"skipped"`
// Alerts contain purely informational messages and data. They
// represent situations where the end user should be aware of some
// occurrence that is not an error, exception, skipped data, or
// other runtime/persistence impacting issue.
Alerts []Alert
// If FailFast is true, then the first Recoverable error will
// promote to the Failure spot, causing processing to exit.
FailFast bool `json:"failFast"`
}
Errors provides the errors data alone, without sync controls or adders/setters. Expected to get called at the end of processing, as a way to aggregate results.
type Item ¶
type Item struct {
// deduplication namespace; the maximally-unique boundary of the
// item ID. The scope of this boundary depends on the service.
// ex: exchange items are unique within their category, drive items
// are only unique within a given drive.
Namespace string `json:"namespace"`
// deduplication identifier; the ID of the observed item.
ID string `json:"id"`
// a human-readable reference: file/container name, email, etc
Name string `json:"name"`
// tracks the type of item represented by this entry.
Type ItemType `json:"type"`
// Error() of the causal error, or a sentinel if this is the
// source of the error. In case of ID collisions, the first
// item takes priority.
Cause string `json:"cause"`
// Additional is a catch-all map for storing data that might
// be relevant to particular types or contexts of items without
// being globally relevant. Ex: parent container references,
// created-by ids, last modified, etc. Should be used sparingly,
// only for information that might be immediately relevant to the
// end user.
Additional map[string]any `json:"additional"`
}
Item contains a concrete reference to a thing that failed during processing. The categorization of the item is determined by its Type: file, container, or reourceOwner.
Item is compliant with the error interface so that it can be aggregated with the fault bus, and deserialized using the errors.As() func. The idea is that fault,Items, during processing, will get packed into bus.AddRecoverable (or failure) as part of standard error handling, and later deserialized by the end user (cli or sdk) for surfacing human-readable and identifiable points of failure.
func ContainerErr ¶
ContainerErr produces a Container-type Item for tracking erroneous items
func (Item) Headers ¶
Headers returns the human-readable names of properties of an Item for printing out to a terminal.
func (Item) MinimumPrintable ¶
type SkipCause ¶
type SkipCause string
SkipCause identifies the well-known conditions to Skip an item. It is important that skip cause enumerations do not overlap with general error handling. Skips must be well known, well documented, and consistent. Transient failures, undocumented or unknown conditions, and arbitrary handling should never produce a skipped item. Those cases should get handled as normal errors.
const ( // SkipMalware identifies a malware detection case. Files that graph // api identifies as malware cannot be downloaded or uploaded, and will // permanently fail any attempts to backup or restore. SkipMalware SkipCause = "malware_detected" // SkipOneNote identifies that a file was skipped because it // was a OneNote file that remains inaccessible (503 server response) // regardless of the number of retries. //nolint:lll // https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa#onenotenotebooks SkipOneNote SkipCause = "inaccessible_one_note_file" // SkipInvalidRecipients identifies that an email was skipped because Exchange // believes it is not valid and fails any attempt to read it. SkipInvalidRecipients SkipCause = "invalid_recipients_email" // SkipCorruptData identifies that an email was skipped because graph reported // that the email data was corrupt and failed all attempts to read it. SkipCorruptData SkipCause = "corrupt_data" // SkipKnownEventInstance503s identifies cases where we have a pre-configured list // of event IDs where the events are known to fail with a 503 due to there being // too many instances to retrieve from graph api. SkipKnownEventInstance503s SkipCause = "known_event_instance_503" )
type Skipped ¶
type Skipped struct {
Item Item `json:"item"`
}
Skipped items are permanently unprocessable due to well-known conditions. In order to skip an item, the following conditions should be met: 1. The conditions for skipping the item are well-known and well-documented. End users need to be able to understand both the conditions and identifications of skips. 2. Skipping avoids a permanent and consistent failure. If the underlying reason is transient or otherwise recoverable, the item should not be skipped.
Skipped wraps Item primarily to minimize confusion when sharing the fault interface. Skipped items are not errors, and Item{} errors are not the basis for a Skip.
func ContainerSkip ¶
ContainerSkip produces a Container-kind Item for tracking skipped items.
func (Skipped) Headers ¶
Headers returns the human-readable names of properties of a skipped Item for printing out to a terminal.
Source Files