datly

README

Datly - Modern flexible ORM for rapid development

This library is compatible with Go 1.11+

Please refer to CHANGELOG.md if you encounter breaking changes.

• Motivation
• Usage
• License

Motivation

The goal of this project is to simplify and speed up data layer prototyping and development. It can be used as golang ORM or purely rule based. This is achieved by utilising rules to govern data mapping and binding for all data interaction.

Introduction

Datly is a modern and flexible ORM and data management platform designed with three principles in mind: performance, productivity, and security. Datly is SQL-centric, where data comes first.

Productivity is achieved by using a higher 4th generation language called DSQL (Datly SQL dialect) to address common problems of manipulating data, allowing developers to focus on addressing business requirements. In addition, more complex cases can be easily delegated to pure Golang, where Datly intermediates with data access and modification. Higher abstraction languages promotes development consistency, offload developers from writing the same code over and over again, which includes routing, struct mapping, batching, security handling, common validation, pagination, dynamic field selection, dynamic criteria, data encoding: json,json-tabular, csv, caching, scaling, runtime/platform independence, sending notification vi universal message bus (sqs/sns/kafka/pubsub) and more.

Datly promotes data cohesion with grouping/batching operation. For example to boostrap your patch operation you would first analyze all inputs driving business logic, then define patch source generation SQL with only needed data points to generate initial patch rule, for example

SELECT  Products.* /* { "Cardinality": "One", "Field":"Entity" } */,
        ProductFlights.*,
        Vendor.*,
        Acl.*,
        Features.*
FROM (SELECT * FROM PRODUCTS) Products,
LEFT JOIN (SELECT * FROM PRODUCT_FLIGHTS) ProductFlights WHERE ProductFlights.PRODUCT_ID = Products.ID
LEFT JOIN (SELECT ID, 
                CURRENCY_ID,
                (SELECT ctz.IANA_TIMEZONE FROM TIME_ZONE ctz WHERE v.TIME_ZONE_ID = ctz.ID) AS IANA_TIMEZONE
            FROM Vendor v
) Vendor  ON Vendor.ID = Product.VENDOR_ID AND 1=1
LEFT JOIN (
    SELECT ID USER_ID,
           HasUserRole(ID, 'ROLE_READ_ONLY') AS IS_READ_ONLY,
           HasUserRole(ID, 'ADMIN') AS IS_ADMIN
    FROM (USERS)
) Acl ON Acl.USER_ID = Products.USER_ID AND 1=1
LEFT JOIN (SELECT
         ID USER_ID,
         HasFeatureEnabled(ID, 'EXPOSE_FEATURE_1') AS FEATURE_1,
        HasFeatureEnabled(ID, 'EXPOSE_FEATURE_2') AS FEATURE_2
        FROM (USERS)
) Features ON Features.USER_ID = Products.USER_ID AND 1=1

In the example above Products, Flights and Vendor represents previous state, Acl defines access-control list, and Features represents feature activator in the UI application.

While Datly in autonomous mode purely uses a meta-driven approach, custom Datly allows blending Go-developed code into rules. As opposed to the purely meta-driven approach, Datly allows both modes to be debugged and troubleshooted with traditional debuggers. Datly automatically generates openAPI documentation allowing any programing languages integrated seamlessly with Datly based micro/rest services. Datly is runtime agnostic, and it can be deployed as standalone, serverless (lambda, cloud function), or Dockerized. Datly is deployment time optimized, allowing rule and logic deployment with powerful Go plugins under seconds on Lambda and other serverless cloud platform.

Performance is achieved by utilizing Go with GoLang structs (never maps), while other frameworks manipulating data use Go reflection, which is around 100x slower than natively typed code, Datly uses xunsafe custom Go reflection, which is only around 5x slower than natively typed code. Datly has the ability to read and assemble data from various database vendors at once and provides powerful optimization techniques like seamless smart caching, driving both client performance and substantially reducing cost. Datly uses Velocity inspired velty templating language which is one of the fastest in the whole Go echo system. On average velty is 20x faster than go Text/template and 8-15x faster than JDK Apache Velocity

Datly can operate on both SQL and NoSQL databases. Large datasets (e.g., BigQuery) can be cached pre-warmed up without engineers writing a single line of code. Datly comes with powerful metrics that provide execution time breakdowns for each data access operation.

When it comes to data modification, Datly can leverage seamless batch and load operations, speeding up data ingestion by 25-50x compared to traditional insert techniques. Datly provides an easy way to build POST/PUT/DELETE and truly performant PATCH operations. Datly use modification marker to distinct input state, allowing handling user input effectively, ensuring data integrity, and improving the security of applications.

Security Datly is secure. It's resilient against SQL injection attacks. On top of that, it promotes secure secrets storage natively with all database/sql drivers. Finally, it's integrated with OAuth, which provides a convenient way for both controlling authentication and row and column based authorization.

See more Datly secutity

Datly use dql to auto generate struct or internal datly rule

dept.sql

SELECT 
    dept.* EXCEPT ORG_ID
    employee.* EXCEPT DEPT_ID, 
    organization.* 
FROM (SELECT * FROM DEPARMENT t) dept
JOIN (SELECT ID, NAME, DEPT_ID FROM EMP t) employee ON dept.ID = employee.DEPT_ID
JOIN ORG organization ON organization.ID = demp.ORG_ID AND 1=1

To test dql vi reset endpoint run the following command

datly translate -c='dev|mysql|root:dev@tcp(127.0.0.1:3306)/dev?parseTime=true' -s=dept.sql -P=8080
open http://127.0.0.1:8080/v1/api/dev/dept    

To persist rule and then run datly run the following

datly translate -c='mydb|mysql|myusser:mypass@tcp(127.0.0.1:3306)/mydb?parseTime=true' -s=dept.sql -r=reop/dev
datly run -c=proj/Datly/config.json

To see go struct generated for the view run the following

open http://127.0.0.1:8080/v1/api/meta/struct/dev/dept

To see go openapi for the view run the following

open http://127.0.0.1:8080/v1/api/meta/openapi/dev/dept

Usage

Managed mode

For reader usage, see: how to use reader

For executor usage, see: how to use executor

Autonomus mode

Contributing to datly

Datly is an open source project and contributors are welcome!

See TODO list

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Authors:

• Kamil Larysz
• Adrian Witas

Documentation

Index

• Variables
• func Handle(w http.ResponseWriter, r *http.Request)
• func LoadInput(ctx context.Context, aSession *session.Session, ...) error
• type OperateOption
  • func WithComponent(component *repository.Component) OperateOption
  • func WithInput(input interface{}) OperateOption
  • func WithOutput(output interface{}) OperateOption
  • func WithPath(aPath *contract.Path) OperateOption
  • func WithSession(session *session.Session) OperateOption
  • func WithSessionOptions(options ...SessionOption) OperateOption
  • func WithURI(URI string) OperateOption
• type Service
  • func New(ctx context.Context, options ...repository.Option) (*Service, error)
  • func (s *Service) AddComponent(ctx context.Context, component *repository.Component) error
  • func (s *Service) AddComponents(ctx context.Context, components *repository.Components) error
  • func (s *Service) AddConnector(ctx context.Context, name string, driver string, dsn string) (*view.Connector, error)
  • func (s *Service) AddConnectors(ctx context.Context, connectors ...*view.Connector) error
  • func (s *Service) AddHandler(ctx context.Context, aPath *contract.Path, handler xhandler.Handler, ...) (*repository.Component, error)
  • func (s *Service) AddMBusResources(ctx context.Context, resource ...*mbus.Resource) error
  • func (s *Service) AddResource(name string, resource *view.Resource)
  • func (s *Service) AddViews(ctx context.Context, views ...*view.View) (*repository.Component, error)
  • func (s *Service) BuildPredicates(ctx context.Context, expression string, input interface{}, baseView *view.View) (*codec.Criteria, error)
  • func (s *Service) Component(ctx context.Context, name string, opts ...repository.Option) (*repository.Component, error)
  • func (s *Service) Exec(ctx context.Context, viewId string, options ...executor.Option) error
  • func (s *Service) HTTPHandler(ctx context.Context, options ...gateway.Option) (http.Handler, error)
  • func (s *Service) HandlerSession(ctx context.Context, aComponent *repository.Component, ...) (xhandler.Session, error)
  • func (s *Service) LoadComponents(ctx context.Context, URL string, opts ...repository.Option) (*repository.Components, error)
  • func (s *Service) NewComponentSession(aComponent *repository.Component, opts ...SessionOption) *session.Session
  • func (s *Service) Operate(ctx context.Context, opts ...OperateOption) (interface{}, error)
  • func (s *Service) PopulateInput(ctx context.Context, aComponent *repository.Component, request *http.Request, ...) error
  • func (s *Service) Read(ctx context.Context, locator string, dest interface{}, option ...reader.Option) error
  • func (s *Service) Reconcile(from interface{}, to interface{}) error
  • func (s *Service) Resource() *view.Resource
  • func (s *Service) Resources() repository.Resources
  • func (s *Service) SignRequest(request *http.Request, claims *jwt.Claims) error
  • func (s *Service) View(ctx context.Context, name string) (*view.View, error)
• type SessionOption
  • func WithForm(form *hstate.Form) SessionOption
  • func WithRequest(request *http.Request) SessionOption
  • func WithStateResource(resource state.Resource) SessionOption

Variables

var Version string

Functions

func Handle

func Handle(w http.ResponseWriter, r *http.Request)

func LoadInput

func LoadInput(ctx context.Context, aSession *session.Session, aComponent *repository.Component, input interface{}) error

Types

type OperateOption

type OperateOption func(o *operateOptions)

func WithComponent

func WithComponent(component *repository.Component) OperateOption

func WithInput

func WithInput(input interface{}) OperateOption

func WithOutput

func WithOutput(output interface{}) OperateOption

func WithPath

func WithPath(aPath *contract.Path) OperateOption

WithPath returns

func WithSession

func WithSession(session *session.Session) OperateOption

func WithSessionOptions

func WithSessionOptions(options ...SessionOption) OperateOption

func WithURI

func WithURI(URI string) OperateOption

WithURI returns with URI option

type Service

type Service struct {
	// contains filtered or unexported fields
}

func New

func New(ctx context.Context, options ...repository.Option) (*Service, error)

New creates a datly service, repository allows you to bootstrap empty or existing yaml repository

func (*Service) AddComponent

func (s *Service) AddComponent(ctx context.Context, component *repository.Component) error

AddComponent adds components to repository

func (*Service) AddComponents

func (s *Service) AddComponents(ctx context.Context, components *repository.Components) error

AddComponents adds components to repository

func (*Service) AddConnector

func (s *Service) AddConnector(ctx context.Context, name string, driver string, dsn string) (*view.Connector, error)

AddConnector adds connector

func (*Service) AddConnectors

func (s *Service) AddConnectors(ctx context.Context, connectors ...*view.Connector) error

AddConnectors adds connectors

func (*Service) AddHandler

func (s *Service) AddHandler(ctx context.Context, aPath *contract.Path, handler xhandler.Handler, options ...repository.ComponentOption) (*repository.Component, error)

AddHandler adds handler component to repository

func (*Service) AddMBusResources

func (s *Service) AddMBusResources(ctx context.Context, resource ...*mbus.Resource) error

AddMBusResources adds message bus resources

func (*Service) AddResource

func (s *Service) AddResource(name string, resource *view.Resource)

AddResource adds named resource

func (*Service) AddViews

func (s *Service) AddViews(ctx context.Context, views ...*view.View) (*repository.Component, error)

AddViews adds views to the repository

func (*Service) BuildPredicates

func (s *Service) BuildPredicates(ctx context.Context, expression string, input interface{}, baseView *view.View) (*codec.Criteria, error)

BuildPredicates added build predicate method

func (*Service) Component

func (s *Service) Component(ctx context.Context, name string, opts ...repository.Option) (*repository.Component, error)

Component returns component matched by name, optionally you can use METHOD:component name notation

func (*Service) Exec

func (s *Service) Exec(ctx context.Context, viewId string, options ...executor.Option) error

Exec executes view template

func (*Service) HTTPHandler

func (s *Service) HTTPHandler(ctx context.Context, options ...gateway.Option) (http.Handler, error)

func (*Service) HandlerSession

func (s *Service) HandlerSession(ctx context.Context, aComponent *repository.Component, aSession *session.Session) (xhandler.Session, error)

HandlerSession returns handler session

func (*Service) LoadComponents

func (s *Service) LoadComponents(ctx context.Context, URL string, opts ...repository.Option) (*repository.Components, error)

LoadComponents loads components into registry, it returns loaded components

func (*Service) NewComponentSession

func (s *Service) NewComponentSession(aComponent *repository.Component, opts ...SessionOption) *session.Session

func (*Service) Operate

func (s *Service) Operate(ctx context.Context, opts ...OperateOption) (interface{}, error)

Operate performs respective operation on supplied component

func (*Service) PopulateInput

func (s *Service) PopulateInput(ctx context.Context, aComponent *repository.Component, request *http.Request, inputPtr interface{}) error

func (*Service) Read

func (s *Service) Read(ctx context.Context, locator string, dest interface{}, option ...reader.Option) error

Read reads data from a view

func (*Service) Reconcile

func (s *Service) Reconcile(from interface{}, to interface{}) error

Reconcile reconciles from with to

func (*Service) Resource

func (s *Service) Resource() *view.Resource

Resource returns resource

func (*Service) Resources

func (s *Service) Resources() repository.Resources

Resource returns resource

func (*Service) SignRequest

func (s *Service) SignRequest(request *http.Request, claims *jwt.Claims) error

SignRequest signes http request with the supplied claim

func (*Service) View

func (s *Service) View(ctx context.Context, name string) (*view.View, error)

View returns a view matched by name, optionally you can use METHOD:component name notation

type SessionOption

type SessionOption func(o *sessionOptions)

func WithForm

func WithForm(form *hstate.Form) SessionOption

func WithRequest

func WithRequest(request *http.Request) SessionOption

func WithStateResource

func WithStateResource(resource state.Resource) SessionOption