enthistory

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2024 License: Apache-2.0 Imports: 24 Imported by: 1

README

Build status Go Report Card Build status Quality Gate Status Go Reference

enthistory

enthistory is a powerful extension for generating history tables using ent - the plugin will add-on to your existing entc usage and enumerate over your current schemas to create new "history" schemas containing an inventory of the changes related to the existing tables.

Credit to flume/enthistory for the inspiration - we chose to create our own for a number of reasons, some being:

  • We have existing patterns within the datumforge/datum repo which would today require an import of the entx package and likely be a non-starter for the original authors
  • We have more complex schemas, mixins, code gen usage; when attempting to use the originally developed plugin we ran into numerous problems based on the types / methods we had already chosen and was easier to short-term directly update with the changes we needed
  • integration with and/or mutual code updates for our "soft delete" constructs to continue to function
  • Specific desires / levels of control regarding data retention and tracking
  • Authorization policies specific to using openFGA may be harder for others to adopt

Installation

You can install enthistory by running the following command:

go get github.com/datumforge/enthistory@latest

In addition to installing enthistory, you need to already have, or create two files (entc.go and generate.go) - this can be within your ent directory, but full instructions can be found in the upstream godoc documentation. The entc.go file should reference the ent history plugin via enthistory.New, and the options you include for the plugin depend on your desired implementation (see the Configuration section below) but you can use the following example for reference:

//go:build ignore

package main

import (
	"log"
	"github.com/datumforge/enthistory"
	"entgo.io/ent/entc"
)

func main() {
	// create new extension with options
	historyExt := enthistory.New(
		enthistory.WithAuditing(),
	)

	// generate the history schemas
	if err := historyExt.GenerateSchemas(); err != nil {
		log.Fatalf("generating history schema: %v", err)
	}

	// run ent generate with extension for other templates
	if err := entc.Generate("./schema",
		&gen.Config{},
		entc.Extensions(
			historyExt,
		),
	); err != nil {
		log.Fatal("running ent codegen:", err)
	}
}

Be sure to read the upstream ent documentation describing the differences between entc and ent, but assuming you're using entc as a package you would want the minimum reference to the run the code generate processes with entc command like below:

package ent

//go:generate go run -mod=mod entc.go

You can additionally call other packages such as mockery within your generate.go - the datum repo could be a good reference point for this.

Usage

Querying History

After generating your history tables from your schema, you can use the ent client to query the history tables. The generated code automatically creates history tables for every table in your schema and hooks them up to the ent client.

You can query the history tables directly, just like any other ent table. You can also retrieve the history of a specific row using the History() method.

enthistory tracks the user who updates a row if you provide a key during initialization. You can store a user's ID, email, IP address, etc., in the context with the key you provide to track it in the history.

Here's an example that demonstrates these features:

// Create
client := enttest.Open(t, "sqlite3", "file:ent?mode=memory&_fk=1")
// Activate the history hooks on the client
client.WithHistory()
character, _ := client.Character.Create().SetName("Marceline").Save(ctx)
characterHistory, _ := character.History().All(ctx)
fmt.Println(len(characterHistory)) // 1

// Update
character, _ = character.Update().SetName("Marshall Lee").Save(ctx)
characterHistory, _ = character.History().All(ctx)
fmt.Println(len(characterHistory)) // 2

// Delete
client.Character.DeleteOne(character)
characterHistory, _ = character.History().All(ctx)
fmt.Println(len(characterHistory)) // 3

In addition to regular queries, you can perform common history queries such as retrieving the earliest history, the latest history, and the history of a row at a specific point in time. enthistory provides functions for these queries:

character, _ := client.Character.Query().First(ctx)

// Get the earliest history for this character (i.e., when the character was created)
earliest, _ := character.History().Earliest(ctx)

// Get the latest history for this character (i.e., the current state of the actual character)
latest, _ := character.History().Latest(ctx)

// Get the history for this character as it was at a given point in time
// (i.e., the state of the actual character at the given point in time)
historyNow, _ := character.History().AsOf(ctx, time.Now())

You can also use the .Next() and .Prev() methods to navigate to the next or previous history entries in time:

character, _ := client.Character.Query().First(ctx)

// Get the earliest history for this character (i.e., when the character was created)
earliest, _ := character.History().Earliest(ctx)

// Get the next history after the earliest history
next, _ := earliest.Next(ctx)

// Get the previous history before the next history
prev, _ := next.Prev(ctx)

// prev would now be the earliest history once again
fmt.Println(prev.ID == earliest.ID) // true
Restoring History

If you need to rollback a row in the database to a specific history entry, you can use the .Restore() function to accomplish that. NOTE: do not attempt to do this in your production environment or otherwise without testing in advance and creating your own SOP's around these types of procedures. By rolling back you are effectively overwriting your primary data source with a new entry, so use with caution!

Here's an example:

// Let's say we create this character
simon, _ := client.Character.Create().SetName("Simon Petrikov").Save(ctx)
// And we update the character's name
iceking, _ := simon.Update().SetName("Ice King").Save(ctx)
// We can find the exact point in history we want to restore, in this case, the oldest history entry
icekingHistory, _ := iceking.History().Order(ent.Asc(characterhistory.FieldHistoryTime)).First(ctx)
// And we can restore the value back to the original table
restored, _ = icekingHistory.Restore(ctx)

fmt.Println(simon.ID == restored.ID) // true
fmt.Println(simon.Name == restored.Name) // true
// The restoration is also tracked in history
simonHistory, _ := restored.History().All(ctx)
fmt.Println(len(simonHistory)) // 3
Auditing

enthistory includes tools for "auditing" history tables by providing a means of exporting the data inside of them. You can enable auditing by using the enthistory.WithAuditing() option when initializing the extension. The main tool for auditing is the Audit() method, which builds an audit log of the history tables that you can export as a file, upload to S3, or inspect.

Here's an example of how to use the Audit() method to export an audit log as a CSV file:

auditTable, _ := client.Audit(ctx)

The audit log contains six columns when user tracking is enabled. Here's an example of how the audit log might look:

Table Ref Id History Time Operation Changes Updated By
CharacterHistory 1 Sat Mar 18 16:31:31 2023 INSERT age: 47 name: "Simon Petrikov" 75
CharacterHistory 1 Sat Mar 18 16:31:31 2023 UPDATE name: "Simon Petrikov" -> "Ice King" 75
CharacterHistory 1 Sat Mar 18 16:31:31 2023 DELETE age: 47 name: "Ice King" 75

You can also build your own custom audit log using the .Diff() method on history models. The Diff() method returns the older history, the newer history, and the changes to fields when comparing the newer history to the older history.

Configuration Options

enthistory provides several configuration options to customize its behavior.

Setting All Tracked Fields as Nillable and/or Immutable

By default, enthistory does not modify the columns in the history tables that are being tracked from your original tables; it simply copies their state from ent when loading them.

However, you may want to set all tracked fields in the history tables as either Nillable or Immutable for various reasons. You can use the enthistory.WithNillableFields() option to set them all as Nillable, or enthistory.WithImmutableFields() to set them all as Immutable.

Note: Setting enthistory.WithNillableFields() will remove the ability to call the Restore() function on a history object. Setting all fields to Nillable causes the history tables to diverge from the original tables, and the unpredictability of that means the Restore() function cannot be generated.

History Time Indexing

By default, an index is not placed on the history_time field. If you want to enable indexing on the history_time field, you can use the enthistory.WithHistoryTimeIndex() configuration option. This option gives you more control over indexing based on your specific needs.

Updated By

To track which users are making changes to your tables, you can use the enthistory.WithUpdatedBy() option when initializing the extension. You need to provide a key name (string) and specify the type of value (enthistory.ValueTypeInt for integers or enthistory.ValueTypeString for strings). The value corresponding to the key should be stored in the context using context.WithValue(). If you don't plan to use this feature, you can omit - you may also already have an existing audit mixin or similar which tracks the user performing the action, in which case, these fields would already be contained within the created history tables.

// Example for tracking user ID
enthistory.WithUpdatedBy("userId", enthistory.ValueTypeInt)

// Example for tracking user email
enthistory.WithUpdatedBy("userEmail", enthistory.ValueTypeString)
Deleted By

To track which users are making changes to your tables, you can use the enthistory.WithDeletedBy() option when initializing the extension. You need to provide a key name (string) and specify the type of value (enthistory.ValueTypeInt for integers or enthistory.ValueTypeString for strings). The value corresponding to the key should be stored in the context using context.WithValue(). If you don't plan to use this feature, you can omit it.

// Example for tracking user ID
enthistory.WithDeletedBy("userId", enthistory.ValueTypeInt)

// Example for tracking user email
enthistory.WithDeletedBy("userEmail", enthistory.ValueTypeString)
Auditing

As mentioned earlier, you can enable auditing by using the enthistory.WithAuditing() configuration option when initializing the extension.

Excluding History on a Schema

enthistory is designed to always track history, but in cases where you don't want to generate history tables for a particular schema, you can apply annotations to the schema to exclude it. Here's an example:

func (Character) Annotations() []schema.Annotation {
    return []schema.Annotation{
        enthistory.Annotations{
            // Exclude history tables for this schema
            Exclude: true,
        },
    }
}
Setting a Schema Path

If you want to set an alternative schema location other than ent/schema, you can use the enthistory.WithSchemaPath() configuration option. The schema path should be the same as the one set in the entc.Generate function. If you don't plan to set an alternative schema location, you can omit this option.

func main() {
    entc.Generate("./schema2",
        &gen.Config{},
        entc.Extensions(
            enthistory.NewHistoryExtension(
                enthistory.WithSchemaPath("./schema2")
            ),
        ),
    )
}

For a complete example of using a custom schema path, refer to the custompaths example.

Setting a Schema Name

If you want to set the schema name for entsql, you can use the enthistory.WithSchemaName() configuration option. This can be used in conjunction with ent Multiple Schema Migrations and the Schema Config features.

Adding GQL Query

If you are using gqlgen and want to generate the query resolvers for the history schemas, you can use the enthistory.WithGQLQuery() configuration option. With this enabled, ent.resolvers with be created, such as:

// TodoHistories is the resolver for the todoHistories field.
func (r *queryResolver) TodoHistories(ctx context.Context, after *entgql.Cursor[string], first *int, before *entgql.Cursor[string], last *int, orderBy *generated.TodoHistoryOrder, where *generated.OrganizationHistoryWhereInput) (*generated.TodoHistoryConnection, error) {
	panic(fmt.Errorf("not implemented: TodoHistories - todoHistories"))
}

Adding a Skipper Function

If you want to conditionally skip saving history data, you can use the enthistory.WithSkipper() configuration option. This should be the string representation that returns true or false. The function has access to the mutation object and the context. For example:

    skipper := `
        hasFeature := m.CheckFeature(ctx)

        return !hasFeature
    `

	historyExt := enthistory.NewHistoryExtension(
        enthistory.WithSkipper(skipper),
    )

Caveats

Here are a few caveats to keep in mind when using enthistory:

Edges

To track edges with history, you need to manage your own through tables. enthistory does not hook into the ent-generated through tables automatically, but managing through tables manually is straightforward. Note that if you use the setters for edges on the main schema tables, the history on the through tables won't be tracked. To track history on through tables, you must update the through tables directly with the required information.

Instead of using .AddFriends() like this:

finn, _ := client.Character.Create().SetName("Finn the Human").Save(ctx)
jake, _ := client.Character.Create().SetName("Jake the Dog").Save(ctx)
finn, _ = finn.Update().AddFriends(jake).Save(ctx)

You should use the Friendship through table:

finn, _ := client.Character.Create().SetName("Finn the Human").Save(ctx)
jake, _ := client.Character.Create().SetName("Jake the Dog").Save(ctx)
friendship, _ := client.Friendship.Create().SetCharacterID(finn.ID).SetFriendID(jake.ID).Save(ctx)

For more information on through tables and edges, refer to the ent documentation.

Enums

If your ent schemas contain enum fields, it is recommended to create Go enums and set the GoType on the enum field. This is because ent generates a unique enum type for both your schema and the history table schema, which may not work well together.

Instead of using .Values() like this:

field.Enum("action").
    Values("PUSH", "PULL")

Use .GoType() like this:

field.Enum("action").
    GoType(types.Action(""))

For more information on enums, refer to the ent documentation.

Contributing

Please read the contributing guide as well as the Developer Certificate of Origin. You will be required to sign all commits to the Datum project, so if you're unfamiliar with how to set that up, see github's documentation.

Security

We take the security of our software products and services seriously, including all of the open source code repositories managed through our Github Organizations, such as datumforge. If you believe you have found a security vulnerability in any of our repositories, please report it to us through coordinated disclosure.

Please do NOT report security vulnerabilities through public github issues, discussions, or pull requests!

Instead, please send an email to security@datum.net with as much information as possible to best help us understand and resolve the issues. See the security policy attached to this repository for more details.

Questions?

You can open a github issue on this repository, or email us at info@datum.net

Documentation

Overview

Package enthistory provides code generation to add history tables for an ent schema

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrUnsupportedIDType is returned when id type other than string or int is used
	ErrUnsupportedIDType = errors.New("unsupported id type, only int and strings are allowed")

	// ErrUnsupportedType is returned when the object type is not supported
	ErrUnsupportedType = errors.New("unsupported type")

	// ErrNoIDType is returned when the id type cannot be determined from the schema
	ErrNoIDType = errors.New("could not get id type for schema")

	// ErrInvalidSchemaPath is returned when the schema path cannot be determined
	ErrInvalidSchemaPath = errors.New("invalid schema path, unable to find package name in path")

	// ErrFailedToGenerateTemplate is returned when the template cannot be generated
	ErrFailedToGenerateTemplate = errors.New("failed to generate template")

	// ErrFailedToWriteTemplate is returned when the template cannot be written
	ErrFailedToWriteTemplate = errors.New("failed to write template")
)

Functions

func HistoryHooks 

func HistoryHooks[T Mutation]() []ent.Hook

HistoryHooks returns a list of hooks that can be used to create history entries

func On 

func On(hk ent.Hook, op ent.Op) ent.Hook

On is a helper function that allows you to create a hook that only runs on a specific operation

Types

type Annotations 

type Annotations struct {
	Exclude   bool `json:"exclude,omitempty"`   // Will exclude history tracking for this schema
	IsHistory bool `json:"isHistory,omitempty"` // DO NOT APPLY TO ANYTHING EXCEPT HISTORY SCHEMAS
}

Annotations of the history extension

func (Annotations) Name 

func (Annotations) Name() string

Name of the annotation

type AuthzSettings added in v0.1.0 

type AuthzSettings struct {
	// Enabled is a boolean that tells the extension to generate the authz policy
	Enabled bool
	// FirstRun is a boolean that tells the extension to only generate the policies after the first run
	FirstRun bool
	// AllowedRelation is the name of the relation that should be used to restrict
	// all audit log queries to users with that role, if not set the interceptor will not be added
	AllowedRelation string
}

type Config 

type Config struct {
	IncludeUpdatedBy bool
	UpdatedBy        *UpdatedBy
	Auditing         bool
	SchemaPath       string
	SchemaName       string
	Query            bool
	Skipper          string
	FieldProperties  *FieldProperties
	HistoryTimeIndex bool
	Auth             AuthzSettings
}

Config is the configuration for the history extension

func (Config) Name 

func (c Config) Name() string

Name of the Config

type ExtensionOption 

type ExtensionOption = func(*HistoryExtension)

func WithAllowedRelation added in v0.1.0 

func WithAllowedRelation(relation string) ExtensionOption

WithAllowedRelation sets the relation that should be used to restrict all audit log queries to users with that role

func WithAuditing 

func WithAuditing() ExtensionOption

WithAuditing allows you to turn on the code generation for the `.Audit()` method

func WithAuthzPolicy added in v0.1.0 

func WithAuthzPolicy() ExtensionOption

func WithFirstRun added in v0.1.0 

func WithFirstRun(firstRun bool) ExtensionOption

WithFirstRun tells the extension to generate the history schema on the first run which leaves out the entfga policy

func WithGQLQuery added in v0.0.7 

func WithGQLQuery() ExtensionOption

WithGQLQuery adds the entgql Query annotation to the history schema in order to allow for querying

func WithHistoryTimeIndex 

func WithHistoryTimeIndex() ExtensionOption

WithHistoryTimeIndex allows you to add an index to the "history_time" fields

func WithImmutableFields 

func WithImmutableFields() ExtensionOption

WithImmutableFields allows you to set all tracked fields in history to Immutable

func WithNillableFields 

func WithNillableFields() ExtensionOption

WithNillableFields allows you to set all tracked fields in history to Nillable except enthistory managed fields (history_time, ref, operation, updated_by, & deleted_by)

func WithSchemaName added in v0.0.6 

func WithSchemaName(schemaName string) ExtensionOption

WithSchemaName allows you to set an alternative schema name This can be used to set a schema name for multi-schema migrations and SchemaConfig feature https://entgo.io/docs/multischema-migrations/

func WithSchemaPath 

func WithSchemaPath(schemaPath string) ExtensionOption

WithSchemaPath allows you to set an alternative schemaPath Defaults to "./schema"

func WithSkipper added in v0.0.7 

func WithSkipper(skipper string) ExtensionOption

WithSkipper allows you to set a skipper function to skip history tracking

func WithUpdatedBy 

func WithUpdatedBy(key string, valueType ValueType) ExtensionOption

WithUpdatedBy sets the key and type for pulling updated_by from the context, usually done via a middleware to track which users are making which changes

func WithUpdatedByFromSchema added in v0.1.1 

func WithUpdatedByFromSchema(valueType ValueType, nillable bool) ExtensionOption

WithUpdatedByFromSchema uses the original update_by value in the schema and includes in the audit results

type FieldProperties 

type FieldProperties struct {
	Nillable  bool
	Immutable bool
}

FieldProperties is a struct that holds the properties for the fields in the history schema

type HistoryExtension 

type HistoryExtension struct {
	entc.DefaultExtension
	// contains filtered or unexported fields
}

HistoryExtension implements entc.Extension.

func New added in v0.1.0 

func New(opts ...ExtensionOption) *HistoryExtension

New creates a new history extension

func (*HistoryExtension) Annotations 

func (h *HistoryExtension) Annotations() []entc.Annotation

Annotations of the HistoryExtension

func (*HistoryExtension) GenerateSchemas added in v0.1.0 

func (h *HistoryExtension) GenerateSchemas() error

GenerateSchemas generates the history schema for all schemas in the schema path this should be called before the entc.Generate call so the schemas exist at the time of code generation

func (*HistoryExtension) SetFirstRun added in v0.1.0 

func (h *HistoryExtension) SetFirstRun(firstRun bool)

SetFirstRun sets the first run value for the history extension outside of the options

func (*HistoryExtension) Templates 

func (h *HistoryExtension) Templates() []*gen.Template

Templates returns the generated templates which include the client, history query, history from mutation and an optional auditing template

type Mutation 

type Mutation interface {
	Op() ent.Op
	CreateHistoryFromCreate(ctx context.Context) error
	CreateHistoryFromUpdate(ctx context.Context) error
	CreateHistoryFromDelete(ctx context.Context) error
}

Mutation is an interface that must be implemented by all mutations that are

type Mutator 

type Mutator interface {
	Mutate(context.Context, Mutation) (ent.Value, error)
}

Mutator is an interface that must be implemented by all mutators that are

type OpType 

type OpType string

OpType is the ent operation type in string form 

const (
	// OpTypeInsert is the insert (create) operation
	OpTypeInsert OpType = "INSERT"
	// OpTypeUpdate is the update operation
	OpTypeUpdate OpType = "UPDATE"
	// OpTypeDelete is the delete operation
	OpTypeDelete OpType = "DELETE"
)

func (OpType) MarshalGQL 

func (op OpType) MarshalGQL(w io.Writer)

MarshalGQL implement the Marshaler interface for gqlgen

func (*OpType) Scan 

func (op *OpType) Scan(v any) error

Scan implements the `database/sql.Scanner` interface for the `OpType` type and is used to convert a value from the database into an `OpType` value.

func (OpType) String 

func (op OpType) String() string

String value of the operation

func (*OpType) UnmarshalGQL 

func (op *OpType) UnmarshalGQL(v interface{}) error

UnmarshalGQL implement the Unmarshaler interface for gqlgen

func (OpType) Value 

func (op OpType) Value() (driver.Value, error)

Value of the operation type

func (OpType) Values 

func (OpType) Values() (kinds []string)

Values provides list valid values for Enum.

type UpdatedBy 

type UpdatedBy struct {
	Nillable bool
	// contains filtered or unexported fields
}

UpdatedBy is a struct that holds the key and type for the updated_by field

type ValueType 

type ValueType uint
const (
	ValueTypeInt ValueType = iota
	ValueTypeString
)

func (ValueType) ValueType 

func (ValueType) ValueType() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.