backend

Documentation

Overview

Package backend implements the configurable backend

A backend manages a Postgres-SQL database and provides an auto-generated RESTful-API for it.

Configuration

The configuration is done entirely via JSON. It consists of collections, singletons, blobs and relations

Example:

  {
	"collections": [
	  {
		"resource": "user",
		"external_index": "identity"
		"schema_id": "https://backend.com/schemas/user.json"
	  },
	  {
		"resource": "device",
		"external_index": "thing"
	  }
	],
	"singletons": [
	  {
		"resource": "user/profile",
	  }
	],
	"relations": [
	  {
		"left": "device",
		"right": "user"
	  }
	]
  }

The example creates one resource "user" with an external unique index "identity".

A user has a child resource "user/profile", which is declared as a singleton, i.e. every user can only have one single profile. Hence a profile does not have an id of its own, but uses the user_id as its primary identifier, and there is a convenient singular resource accessor for a user's profile.

Finally there is a relation from device to user which creates two more virtual child resources "user/device" and "device/user".

This configuration creates the following REST routes:

/users GET,POST,PUT,PATCH
/users/{user_id} GET,PUT,PATCH,DELETE
/devices GET,POST,PUT,PATCH
/devices/{device_id} GET,PUT,PATCH,DELETE
/users/{user_id}/devices GET
/users/{user_id}/devices/{device_id} GET,PUT,DELETE
/devices/{device_id}/users GET
/devices/{device_id}/users/{user_id} GET,PUT,DELETE
/users/{user_id}/profile GET,PUT,PATCH,DELETE
/users/{user_id}/profiles GET,POST,PUT,PATCH
/users/{user_id}/profiles/{user_id} GET,PUT,PATCH,DELETE

The models look like this:

User
{
	"user_id": UUID,
	"identity": STRING
	"timestamp": TIMESTAMP
	"revision": INTEGER
	...
}

Profile
{
	"profile_id": UUID
	"user_id": UUID,
	"timestamp": TIMESTAMP
	"revision": INTEGER
	...
}

Device
{
	"device_id": UUID,
	"thing": STRING
	"timestamp": TIMESTAMP
	"revision": INTEGER
	...
}

We can now create a user with a simple POST:

  curl http://localhost:3000/users -d'{"identity":"test@test.com", "name":"Jonathan Test"}'
  {
	"timestamp": "2020-03-23T16:01:08.138302Z",
 	"identity": "test@test.com",
	"name": "Jonathan Test",
 	"user_id": "f879572d-ac69-4020-b7f8-a9b3e628fd9d"
  }

We can create a device:

  curl http://localhost:3000/devices -d'{"thing":"12345"}'
  {
 	"timestamp": "2020-03-23T16:07:23.57638Z",
	"device_id": "783b3674-34d5-497d-892a-2b48cf99296d",
	"thing": "12345"
  }

And we can assign this device to the test user:

curl -X PUT http://localhost:3000/users/f879572d-ac69-4020-b7f8-a9b3e628fd9d/devices/783b3674-34d5-497d-892a-2b48cf99296d
204 No Content

Now we can query the devices of this specific user:

  curl http://localhost:3000/users/f879572d-ac69-4020-b7f8-a9b3e628fd9d/devices
  [
 	{
	  "timestamp": "2020-03-23T16:07:23.57638Z",
	  "device_id": "783b3674-34d5-497d-892a-2b48cf99296d",
	  "thing": "12345"
	 }
  ]

This adds a profile to the user, or updates the user's profile:

  curl-X PUT http://localhost:3000/users/f879572d-ac69-4020-b7f8-a9b3e628fd9d/profile -d'{"nickname":"jonathan"}'
  {
 	"timestamp": "2020-03-23T16:25:15.738091Z",
 	"profile_id": "9a09030c-516f-4dcd-a2fc-dedad219457d",
	"nickname": "jonathan",
	"user_id": "f879572d-ac69-4020-b7f8-a9b3e628fd9d"
  }

Shortcut Routes

The above example can be made even more user friendly, by adding shortcut routes for the authenticated user. Say we have a role "userrole" which contains a selector for a user resource. Then we can declare a shortcut with

  	...
	"shortcuts": [
	  {
		"shortcut": "user",
		"target" : "user",
		"roles": ["userrole"]
	  }
	...

This creates additional REST routes where every path segment /users/{user_id} is replaced with the shortcut /user for all generated routes. For example, instead of querying a user's devices with users/f879572d-ac69-4020-b7f8-a9b3e628fd9d/devices you would simply query /user/devices.

Revisions

Every item has an integer property "revision", which is incremented every time the item is updated. Revisions can be used to make updates safe in systems with multiple concurrent writers. If a PUT or PATCH request contains a non-zero revision number which does not match the item's current revision, then the request is discarded and the conflicting newer version of the object is returned with an error status (409 - Conflict). A PUT or PATCH request with a revision of zero, or no revision at all, will not be checked for possible conflicts.

Wildcard Queries

You can replace any id in a path segment with the keyword "all". For example, if some administrators wants to retrieve all profiles from all users, they would query

GET /users/all/profiles

Schema Validation

Every resource by default is essentially a free-form JSON object. This gives a high degree of flexibility, but is prone to errors. Therefore you can define a JSON schema ID for any Singleton or Collection resource. If the "schema_id" is defined, any attempt to PUT, POST or PATCH this resource will be validated against this schema. If validation fails, error 400 will be returned.

Default Properties

Any Singleton or Collection resource can have an additional property "default", which defines default properties for all instances. Default properties are automatically added whenever objects are created or updated in the database. In addition, they are also added when older versions of objects are read from the database. Default properties are especially useful in combination with schema validation, as they make it possible to add new required properties without having to migrate all existing objects in the database.

Static Properties

In the example above, we have extended the user and the device collections with an external index. Likewise it is possible to extend resource with list of static string properties, using an array "static_properties". Static properties (searchable or not) have the advantage, that they can be updated must faster than any other dynamic property. If the user resource from above had a static property "name", you could update that name quickly with

POST /user/{user_id}/name/{new_name}

It is only in rare occasions when you actually need this. In the regular case, properties of a resource should not need to be declared static, and property updates should be done with a standard PATCH request, returning the fully patched object.

There is one more application for static properties: Since they have their own underlying SQL column, they also enable easier SQL queries against generated tables for other services with direct acccess to the database. For example, we use a static property to store the provisioning_status for IoT devices.

Static properties can be made searchable by adding them to the "searchable_properties" array instead. This activates a filter in the collection get route with the name of the property. See the chapter on query parameters and pagination below.

Sorting and Timestamp

Collections of resources are sorted by the timestamp, with latest first. For additional flexibility, it is possible to overwrite the timestamp in a POST or PUT request. If you for example import workout activities of a user, you may choose to use the start time of each activity as timestamp.

Query Parameters and Pagination

The GET request on single resources - i.e. not on entire collections - can be customized with the "children" query parameter. It makes it possible to add child resources to the response, avoiding unnecessary rest calls. For example. if you want to retrieve a specific user, the user's profile and the user's devices, you can do all that with a single request to

GET /user?children=profile,devices

or

GET /user?children=profile&children=devices

By using the paramter nointercept=true, it is possible to supress any interceptors and return the latest version of the document stored.

The GET request on collections can be customized with any of the searchable properties, an external index, the ids of the resources or the first layer of properties of the json document as a filter. It is possible to search for equality of to search a pattern.

Searching and Filtering

Collections support two different operators for searching and filtering: search and filter. The operator "search" is guaranteed to be fast, it only works on external indices or explicitly marked searchable properties. If you try to search for resources by a different property it will flag a bad request error. The operator "filter" will try to use database indices when available, but it will also filter based on JSON properties.

Searching for equality: In our example, the resource "user" has an external index "identity", hence we can query all users for a specific identity with

GET /users?filter=identity=test@test.com

Searching pattern: Searching for pattern is done using the `~` character instead of `=`. Pattern are written using SQL LIKE format. % represents zero, one, or multiple characters _ represents one, single character

GET /users?filter=identity~%@test.com
returns all users with an email which ends with @test.com

If you specify multiple filters, they filter on top of each other (i.e. with logical AND).

Filters can be combined with the wildcard 'all' keyword. For instance, it is possible to get all the devices of a user by filtering on the user_id property

GET /users/all/devices?filter=user_id=f879572d-ac69-4020-b7f8-a9b3e628fd9d
	This is equivalent to using the following, but may be more convenient to write in some cases.
GET users/f879572d-ac69-4020-b7f8-a9b3e628fd9d/devices

The system supports pagination and filtering of responses by creation time:

?order=[asc|desc]  sets the sorting order to be descending (newest first, the default) or ascending (oldest first)
?limit=n  sets a page limit of n items
?page=n   selects page number n. The first page is page 1
?from=t   selects items created at or after the timestamp t
?until=t  selects items created up until and including the timestamp t. The default is "0001-01-01 00:00:00 +0000 UTC".
Timestamps must be formatted following RFC3339 (https://tools.ietf.org/html/rfc3339).

The response carries the following custom headers for pagination:

"Pagination-Limit"        the page limit
"Pagination-Total-Count"  the total number of items in the collection
"Pagination-Page-Count"   the total number of pages in the collection
"Pagination-Current-Page" the currently selected page
"Pagination-Until"	    the timestamp of the first item in the response

The maximum allowed limit is 100, which is also the default limit. Combining pagination with the until-filter avoids page drift. A well-behaving application would get the first page without any filter, and then use the timestamp reported in the "Pagination-Until" header as until-parameter for querying pages further down.

For collections it is possible to only retrieve meta data, by specifying the ?onlymeta=true query parameter. Meta data are all defining identifiers, the timestamp and each object's revision number.

Primary Resource Identifier

The primary resource identifier is not mandatory when creating resources. If the creation request (POST or PUT) contains no identifier or a null identifier, then the system creates a new unique UUID for it. Yet it is possible to specify a primary identifier in the request, which will be honored by the system. This feature - and the choice of UUID for primary identifiers - makes it possible to easily transfer data between different databases.

Notifications

The backend supports notifications through the Notifier interface specified at construction time.

Relations

The example demonstrated a relation between "user" and "device", which created two additional resources "user/device" and "device/user". Relations also work between different child resources, for example between "fleet/user" and "fleet/device", as long as both resources have a compatible base (in this case "fleet"). Furthermore relations are transient. Say you have actual resources "device" and "fleet", and a relation between them, which creates a virtual resource "fleet/device". In this case you can also have a relation between "fleet/user" and "fleet/device", leading to the two additional resources "fleet/user/device" and "fleet/device/user".

Relations support separate permits for the left and the right resource, called "left_permits" and "right_permits". "left_permits" applies to the left/right relations. "right_permits" applies to the right/left relations.

Examples:

• to read left/{left_id}/right/{right_id}, one needs to have the "read" permission on the left_permit.
• to read right/{right_id}/left/{left_id}, one needs to have the "read" permission on the right_permit.
• to create left/{left_id}/right/{right_id}, one needs to have the "create" permission on the left_permit and read permission on the right resource
• to create right/{right_id}/left/{left_id}, one needs to have the "create" permission on the right_permit and read permission on the left resource
• to list left/{left_id}/rights, one needs to have the "list" permission on the left_permit.
• to list right/{right_id}/lefts, one needs to have the "list" permission on the right_permit.
• to delete left/{left_id}/right/{right_id}, one needs to have the "delete" permission on the left_permit.
• to delete right/{right_id}/left/{left_id}, one needs to have the "delete" permission on the right_permit.
• the update permission is not used

For each relation, the number of related resources for one other resource is currently limited by 1000. In the above example, one fleet can have up to 1000 users and devices, and each user then can be assigned to 1000 devices max.

Relations support an extra query parameter "?idonly=true", which returns only the list of ids as opposed to full objects. If you furthermore specify "withtimestamp=true", you will receice both the ids and the timestamp when this relation was established.

Relations can also be given an explicit Resource name just like any other collection, which allows multiple different relations from the the same resource types. The resource name then becomes a prefix to access the relation.

Blobs

Blobs are collections of binary resources. They will be served to the client as-is. You can use blobs for example to manage a collection of images like this:

  	"blobs": [
	  {
		"resource": "image",
		"static_properties" : ["content_type"]
	  }
	]

This configuration creates the following routes:

GET /images  - returns meta data of all images as JSON
POST /images
GET /images/{image_id}
DELETE /images/{image_id}

All static properties, searchable properties and external indices of a blob are passed as canonical headers. The property "content_type" hence becomes a header "Content-Type". All other properties are transferred as the header "Kurbisio-Meta-Data".

Blobs are immutable by default, which means they can be optimally cached. If you need blobs that can be updated, for example a profile image, you get declare them mutable like this:

  	"blobs": [
	  {
		"resource": "image",
		"static_properties" : ["content_type"],
		"mutable": true,
		"max_age_cache": 3600
	  }
	]

This creates the extra route

PUT /images/{image_id}

In the example we have also set the "max_age_cache" property to 3600 seconds, which is one hour. The default for mutable blobs is no caching at all. Mutable blobs also support Etag and If-Non-Match out-of-the-box, which allows clients to check for updates quickly without re-downloading the entire blob. See section on If-None-Match and Etag below.

Authorization

If AuthorizationEnabled is set to true, the backend supports role based access control to its resources. By default, only the "admin" role has a permit to access resources. A permit object for each resource authorizes specific roles to execute specific operations. The different operations are: "create", "read", "update", "delete", "list" and "clear". The "list"-operation is the retrieval of the entire collection, "clear" deletes the entire collection.

"admin viewer" also has right to access all resources in read only mode. Only read and list operations are permitted.

For example, you want to declare a resource "picture", which is a child-resource of "user". Now you want to give each user permission to create, read and delete their own pictures, but only their own pictures. You declare a role for a user - in this case "userrole" - and specify the resource like this:

{
	"resource": "user/picture",
	"permits": [
		{
			"role": "userrole",
			"operations": [
				"create",
				"read",
				"update",
				"delete",
				"list",
				"clear"
			],
			"selectors": [
				"user"
			]
		}
	]
}

The selector basically states that the authorization object must contain a concrete user_id, and that any of the operations is only permitted for this user_id.

Now users want to be able to share links to their pictures. The public should be able to read them, but they should not be able to list all picture, nor to create new ones nor delete them. You can achieve this by issueing another permit for the "public" role:

"permits": [
	...
	{
		"role": "public",
		"operations": [
			"read"
		]
	}
]

There are three special roles in the system: The "admin" role who has permission to do everything. The "admin viewer" role has permission to read and list everything, but not modify or create. The "public" role, which is assumed by every non-authorized request. And finally the "everybody" role, which is a placeholder for any other role in the system but "public".

You can easily check the authorization state of any token, by doing a GET request to

/authorization

which will return the authorization state for the authenticated requester as JSON object.

Singletons conceptually always exist, i.e. they can be updated and patched with a permission for "update", even if there is no object in the database yet.

If-None-Match and Etag

All GET requests are served with Etag and obey the If-None-Match request. This allows clients to check whether new data is available without downloading and comparing the entire response. If a client puts the received Etag of a request into the If-None-Match header of a subsequent request, then the system will simply response to that subsequent with a 304 Not Modified in case the resource was not changed. In case the resource was changed, the request will be answered as usual.

Externally stored data

Collections allow to store a file with each individual collection item. Unlike blobs which should remain of reasonable size to preserve the performance of the database, a file associated to a collection item can be large since it is stored outside of the database. The storage backend can be either an AWS S3 bucket or a local file system. Selecting the storage backend is done at startup time using <TODO ADD DETAILS>

Local storage is most likely to be used only for testing purpose to avoid the need for an internet connection and an S3 bucket. Using local filesystem implementation is not intended to be used at scale since it has not been implemented with performance and scalability as requirements.

To enable file storage for a resource, set property "with_companion_file" to true in the resource configuration.

Example:

{
	"collections": [
	  {
		"resource": "release",
		"schema_id": "https://backend.com/schemas/release.json"
	  },
	  {
		"resource": "release/artefact",
		"with_companion_file": true,
		companion_presigned_url_validity: 3600
	  }
	],
	"singletons": [
	],
	"relations": [
	]
  }

In the above example, the release/artefact resource will have the possibility to store a file together with this resource. Accessing this collection will add to the returned object two string properties `companion_download_url`, `companion_upload_url`.

They will be populated based on the type of request: GET adds `download_url` POST adds `upload_url` PUT adds `upload_url` LIST no extra field. If flag `with_companion_urls=true` is set, `download_url` are provided for each item

As their name suggest, the `companion_download_url` and `companion_upload_url`allow to respectively download and upload data. As a result, uploading and downloading file is a two-steps operation. First the download URL is fetched, then the URL is used to fetch data. The URL are pre-signed URL with a validity time which is defined in the url itself.

It is possible to define the validity duration of the pre-signed URL in the configuration using the `companion_presigned_url_validity` key which defines the duration in seconds for which the URL will be valid

Deleting a resource also delete the associated companion file if it exist

Statistics

Statistics about the backend can be retrieved by doing a GET request to:

/statistics

This returns a JSON body like this:

{
	"resources": [
		{
			"name": "user"
			"type": "collection"
			"count": 123,
			"size_mb": 0.117,
			"average_size_b": 599
		},
		{
			"name": "device"
			"type": "collection"
			"count": 56483,
			"size_mb": 12,
			"average_size_b": 558
		}
	]
}

If you are only interested in certain resources, you can filter using the resource parameter like this:

/statistics?resource=user,device

Version

The Version of the software running can be obtain from a dedicated endpoint. The version can be set at compile time with the following parameter: -ldflags '-X github.com/relabs-tech/kurbisio/core/backend.Version="1.2.3"'

/version

This returns a Json body like this:

{
	"version": "1.2.3"
}

Index

• Constants
• Variables
• type Backend
  • func New(bb *Builder) *Backend
  • func (b *Backend) CancelEvent(ctx context.Context, event Event) (bool, error)
  • func (b *Backend) Config() Configuration
  • func (b *Backend) DefineRateLimitForEvent(event string, delta, maxAge time.Duration)
  • func (b *Backend) HandleEvent(event string, handler func(context.Context, Event) error)
  • func (b *Backend) HandleResourceNotification(resource string, handler func(context.Context, Notification) error, ...)
  • func (b *Backend) HandleResourceRequest(resource string, ...)
  • func (b *Backend) HasJobsToProcess() bool
  • func (b *Backend) Health(includeDetails bool) (Health, error)
  • func (b *Backend) HealthPurge() error
  • func (b *Backend) ProcessJobsAsync(heartbeat time.Duration)
  • func (b *Backend) ProcessJobsSync(max time.Duration) bool
  • func (b *Backend) ProcessJobsSyncWithTimeouts(max time.Duration, timeouts [3]time.Duration) bool
  • func (b *Backend) PublicURL() string
  • func (b *Backend) QueueEvent(ctx context.Context, event Event) error
  • func (b *Backend) RaiseEvent(ctx context.Context, event Event) error
  • func (b *Backend) RaiseEventIfNotExist(ctx context.Context, event Event) error
  • func (b *Backend) RetrieveEventSchedule(ctx context.Context, event Event) (*time.Time, error)
  • func (b *Backend) Router() *mux.Router
  • func (b *Backend) ScheduleEvent(ctx context.Context, event Event, scheduleAt time.Time) error
  • func (b *Backend) ScheduleEventIfNotExist(ctx context.Context, event Event, scheduleAt time.Time) error
  • func (b *Backend) TriggerJobs()
• type Builder
• type Configuration
• type Event
  • func (e Event) WithPayload(payload interface{}) Event
• type EventPriority
• type Health
• type HealthJobs
• type JobDetail
• type Notification
• type Request
• type ResourceStatistics
• type StatisticsDetails

Constants

const InternalDatabaseSchemaVersion = 3

InternalDatabaseSchemaVersion is a sequential versioning number of the database schema. If it increases, the backend will try to update the schema.

Variables

var ConfigSchemaJSON string

ConfigSchemaJSON contains the Json schemafor the backend's configuration file

var (
	// Version is the version of the current build
	Version = "unset"
)

Types

type Backend

type Backend struct {

	// Registry is the JSON object registry for this backend's schema
	Registry registry.Registry

	JsonValidator *schema.Validator
	KssDriver     kss.Driver
	// contains filtered or unexported fields
}

Backend is the generic rest backend

func New

func New(bb *Builder) *Backend

New realizes the actual backend. It creates the sql relations (if they do not exist) and adds actual routes to router

func (*Backend) CancelEvent

func (b *Backend) CancelEvent(ctx context.Context, event Event) (bool, error)

CancelEvent cancels a scheduled event of the same kind (event plus key) to the very same resource (resource + resourceID).

The payload of the passed event object is ignored.

The function returns true if an event was unscheduled, otherwise it returns false.

func (*Backend) Config

func (b *Backend) Config() Configuration

Config returns the backend configuration

func (*Backend) DefineRateLimitForEvent

func (b *Backend) DefineRateLimitForEvent(event string, delta, maxAge time.Duration)

DefineRateLimitForEvent defines a rate limit for the specified event. If the event is raised with RaiseEvent it will be scheduled at the next available time slot, leaving a duration of delta between all events of the same type. If at execution time the delta between the scheduled time and the actual time exceeds maxAge, then the event will be rescheduled (and an error will be written to the logs)

func (*Backend) HandleEvent

func (b *Backend) HandleEvent(event string, handler func(context.Context, Event) error)

HandleEvent installs a callback handler the specified event. Handlers are executed out-of-band. If a handler fails (i.e. it returns a non-nil error), it will be retried a few times with increasing timeout.

func (*Backend) HandleResourceNotification

func (b *Backend) HandleResourceNotification(resource string, handler func(context.Context, Notification) error, operations ...core.Operation)

HandleResourceNotification installs a callback handler for out-of-band notifications for a given resource and a set of mutable operations.

If no operations are specified, the handler will be installed for all mutable operations on single resources, i.e. create, update and delete.

Notification handlers only support mutable operations. They are executed reliably out-of-band when an object was modified, and retried a few times when they fail (i.e. return a non-nil error).

The payload of a Create, Update or Delete notification is the object itself. The only exception is a direct property update. In this case only the updated property is contained in the notification.

The payload for a Clear notification is a map[string]string of the query parameters (from,until,filter) and the collection's identifiers from the request URL.

If you need to intercept operations - including the immutable read and list operations -, then you can do that in-band with a request handler, see HandleResourceRequest()

func (*Backend) HandleResourceRequest

func (b *Backend) HandleResourceRequest(resource string,

	handler func(ctx context.Context, request Request, data []byte) ([]byte, error),
	operations ...core.Operation)

HandleResourceRequest installs an in-band interceptors for a given resource and a set of operations. If no operations are specified, the handler will be installed for the Read operation only.

Any returned non-nil error will abort the operation and result in a HTTP error status code. For write operations that would be 400 (bad request) and for read operations 500 (internal server error).

If the handler returns a non-nil []byte, this will replace the original data. In case of Read, the user will see the handler's version. In case of Create or Update, the handler's version will be written to the database and then be returned to the user. For the Delete operation, data will always be nil and the returned data is ignored.

Update property requests cannot be intercepted.

func (*Backend) HasJobsToProcess

func (b *Backend) HasJobsToProcess() bool

HasJobsToProcess returns true, if there are jobs to process. It then resets the process flag.

func (*Backend) Health

func (b *Backend) Health(includeDetails bool) (Health, error)

Health returns the backend's health status

func (*Backend) HealthPurge

func (b *Backend) HealthPurge() error

HealthPurge deletes old health data. Currently this is only failed jobs

func (*Backend) ProcessJobsAsync

func (b *Backend) ProcessJobsAsync(heartbeat time.Duration)

ProcessJobsAsync starts a job processing loop. It returns immediately. This function must only be called once.

If heartbeat is larger than 0, the function also starts a heartbeat timer for processing of scheduled events and notifications.

Left-over jobs in the database are processed right away.

func (*Backend) ProcessJobsSync

func (b *Backend) ProcessJobsSync(max time.Duration) bool

ProcessJobsSync commisions all pending jobs up to the specified maximum duration and then returns after the last commissioned job was fully processed. It returns true if it has maxed out and there are more jobs to process, otherwise it returns false. It you pass 0, it will process all pending jobs.

The function uses a 5 minute timeout for the first retry, 15 minutes for the 2nd and 45 minutes for the last

func (*Backend) ProcessJobsSyncWithTimeouts

func (b *Backend) ProcessJobsSyncWithTimeouts(max time.Duration, timeouts [3]time.Duration) bool

ProcessJobsSyncWithTimeouts commisions all pending jobs up to the specified maximum duration and then returns after the last commissioned job was fully processed. It returns true if it has maxed out and there are more jobs to process, otherwise it returns false. It you pass 0, it will process all pending jobs.

Jobs will be tried up to 3 times according to the timeouts specified.

func (*Backend) PublicURL

func (b *Backend) PublicURL() string

PublicURL returns this backend's deployments public URL

func (*Backend) QueueEvent

func (b *Backend) QueueEvent(ctx context.Context, event Event) error

QueueEvent adds the requested event to the queue. Payload can be nil, an object or a []byte. Callbacks registered with HandleEvent() will be called.

Queued events are always going to be delievered, there is no compression happening.

func (*Backend) RaiseEvent

func (b *Backend) RaiseEvent(ctx context.Context, event Event) error

RaiseEvent raises the requested event. Payload can be nil, an object or a []byte. Callbacks registered with HandleEvent() will be called.

Multiple events of the same kind (event plus key) to the very same resource (resource + resourceID) will be compressed, i.e. the newest payload will overwrite the previous payload. If you do not want any compression, use QueueEvent() instead.

If an event as a rate limit defined (see DefineRateLimitForEvent), then the event will be scheduled at the next available time slot.

Use ScheduleEvent if you want to schedule an event at a specific time.

func (*Backend) RaiseEventIfNotExist

func (b *Backend) RaiseEventIfNotExist(ctx context.Context, event Event) error

RaiseEventIfNotExist raises the requested event. Payload can be nil, an object or a []byte. Callbacks registered with HandleEvent() will be called.

If an event of the same kind(event plus key) to the very same resource (resource + resourceID) has already been raised, then the new event will be ignored completely.

Use RaiseEvent if you want to raise the event immediately. Use CancelEvent() to cancel a scheduled event.

func (*Backend) RetrieveEventSchedule

func (b *Backend) RetrieveEventSchedule(ctx context.Context, event Event) (*time.Time, error)

RetrieveEventSchedule exists for unit testing purposes only

func (*Backend) Router

func (b *Backend) Router() *mux.Router

Router returns the mux.Router for this backend

func (*Backend) ScheduleEvent

func (b *Backend) ScheduleEvent(ctx context.Context, event Event, scheduleAt time.Time) error

ScheduleEvent schedules the requested event at a specific point in time. Payload can be nil, an object or a []byte. Callbacks registered with HandleEvent() will be called.

Multiple events of the same kind (event plus key) to the very same resource (resource + resourceID) will be compressed, i.e. the newest payload will overwrite the previous payload. If you do not want any compression, use a unique key

Use RaiseEvent if you want to raise the event immediately. Use CancelEvent() to cancel a scheduled event.

func (*Backend) ScheduleEventIfNotExist

func (b *Backend) ScheduleEventIfNotExist(ctx context.Context, event Event, scheduleAt time.Time) error

ScheduleEventIfNotExist schedules the requested event at a specific point in time. Payload can be nil, an object or a []byte. Callbacks registered with HandleEvent() will be called.

If an event of the same kind(event plus key) to the very same resource (resource + resourceID) has already been scheduled, then the new event will be ignored completely.

Note that this is always case in an event handler reacting to the event, because the event exists until the handler terminated successfully.

Use RaiseEvent if you want to raise the event immediately. Use CancelEvent() to cancel a scheduled event.

func (*Backend) TriggerJobs

func (b *Backend) TriggerJobs()

TriggerJobs triggers pipeline processing.

type Builder

type Builder struct {
	// Config is the JSON description of all resources and relations. This is mandatory.
	Config string
	// DB is a postgres database. This is mandatory.
	DB *csql.DB
	// Router is a mux router. This is mandatory.
	Router *mux.Router
	// Optional public URL of the deployment
	PublicURL string
	// If AuthorizationEnabled is true, the backend requires auhorization for each route
	// in the request context, as specified in the configuration.
	AuthorizationEnabled bool

	// Number of concurrent pipeline executors. Default is 5.
	PipelineConcurrency int

	// JSONSchemasFS contains JSON schema files to be used by the json validator. It is exclusive with JSONSchemas and JSONSchemasRefs
	JSONSchemasFS *embed.FS

	// JSONSchemas is a list of top level JSON Schemas as strings. It is exclusive with the JSONSchemasFS
	JSONSchemas []string

	// JSONSchemasRefs is a list of references JSON Schemas as strings. It is exclusive with the JSONSchemasFS
	JSONSchemasRefs []string

	// If populated with a logger, the logger will be used. Otherwise a logger with LogLevel will be created (see InitLogger).
	Logger *logrus.Logger

	// The loglevel to be used by the logger if Logger is nil. Default is "info"
	LogLevel string

	// if true, always update the schema. Otherwise only update when the schema json has changed.
	UpdateSchema bool

	// Defines the configuration for the KSS service
	KssConfiguration kss.Configuration
}

Builder is a builder helper for the Backend

type Configuration

type Configuration struct {
	Collections []collectionConfiguration `json:"collections"`
	Singletons  []singletonConfiguration  `json:"singletons"`
	Blobs       []blobConfiguration       `json:"blobs"`
	Relations   []relationConfiguration   `json:"relations"`
	Shortcuts   []shortcutConfiguration   `json:"shortcuts"`
}

Configuration holds a complete backend configuration

type Event

type Event struct {
	Type        string
	Key         string
	Resource    string
	ResourceID  uuid.UUID
	Payload     []byte
	ScheduledAt *time.Time // only used for reporting in the event handler
	Priority    EventPriority
}

Event is a higher level event. Receive them with HandleEvent(), raise them with RaiseEvent(), schedule them with ScheduleEvent()

func (Event) WithPayload

func (e Event) WithPayload(payload interface{}) Event

WithPayload adds a payload to an event. Payload can be an object or a []byte

type EventPriority

type EventPriority int

EventPriority is the event priority

const (
	PriorityForeground EventPriority = iota
	PriorityBackground
)

type Health

type Health struct {
	Jobs           HealthJobs `json:"jobs"`
	BackgroundJobs HealthJobs `json:"background_jobs"`
}

Health contains the backend's health status

type HealthJobs

type HealthJobs struct {
	Scheduled int64       `json:"scheduled"`
	Failed    int64       `json:"failed"`
	Failing   int64       `json:"failing"`
	Overdue   int64       `json:"overdue"`
	Details   []JobDetail `json:"details,omitempty"`
}

type JobDetail

type JobDetail struct {
	Serial       int64      `json:"serial"`
	Job          string     `json:"job"`
	Type         string     `json:"type"`
	Key          string     `json:"key"`
	Resource     string     `json:"resource"`
	ResourceID   string     `json:"resource_id"`
	AttemptsLeft int64      `json:"attempts_left"`
	Timestamp    time.Time  `json:"timestamp"`
	ScheduledAt  *time.Time `json:"scheduled_at"`
}

JobDetail is detail on a job for the health endpoint

type Notification

type Notification struct {
	Resource   string
	ResourceID uuid.UUID
	Operation  core.Operation
	Payload    []byte
}

Notification is a database notification. Receive them with HandleResourceNotification()

type Request

type Request struct {
	// Resource for which this request is made
	Resource string
	// the primary ID for the resource, for singletons this is the parent ID, for
	// list requests this is a  null uuid.
	ResourceID uuid.UUID
	// Operation for this request
	Operation core.Operation
	// Selectors are the identifiers from the request URL, can be UUID or "all"
	Selectors map[string]string
	// Parameters are the query parameters from the request URL
	Parameters map[string]string
}

Request is a database request. Receive them with HandleResourceRequest()

type ResourceStatistics

type ResourceStatistics struct {
	Resource     string  `json:"resource"`
	Count        int64   `json:"count"`
	SizeMB       float64 `json:"size_mb"`
	AverageSizeB float64 `json:"average_size_b"`
}

ResourceStatistics represents information about a resource

type StatisticsDetails

type StatisticsDetails struct {
	Collections []ResourceStatistics `json:"collections"`
	Singletons  []ResourceStatistics `json:"singletons"`
	Relations   []ResourceStatistics `json:"relations"`
	Blobs       []ResourceStatistics `json:"blobs"`
}

StatisticsDetails represents information about the backend resources