gonatus

README

Gonatus

Gonatus is an opensource Golang library for highly scalable environments for unstructured, structured, and relational data analysis.

Gonatus System

While Gonatus is meant to be Big Data framework for large scalable environments, we cover these areas:

• object system with functional serialization,
• reinvented file system,
• logging,
• network service over various protocols (HTTP, TCP, unix sockets, ...),
• services proxying user input into implementations based on the various protocols,
• collection definitions for fast column based index search and manipulation with driver definitions for various technology (solr, elastic, ...),
• cluster shared memory:
  • persistent object shared store,
  • unique identifier generation,
  • configuration store.
• dynamic scaling of gonatus nodes with scheduling.

Principles & Best Practices

We define Pseudo-Object system for Golang. Every structure implementing the Gobjecter interface is called a class. The structure can be private, then all interactions with it are made through an interface the structure implements. In this case, we speak about interface class.

Instead of self or this, we use ego as a receiver name (although uniform receiver names are not recommended in Golang).

Object system

To be a valid Gobject, the structure has to implement Gobjecter interface and to embed Gobject struct.

The Constructor for a class A should be in form of func NewA(AConf) *A or func NewA(AConf) A if A is an interface class. AConf is structure containing Golang basic types (numbers, strings, simple structures, slices and maps). Pointers and Gobjects are not allowed.

Serialization

Every Gonatus class has to be serializable into its own serialization structure, so that the following applies:

// let a be a Gobject and A its class
NewA(a.serialize()).serialize() == a.serialize()

For this purpose, every class A must have a Serialize() Conf method (required by Gobjecter interface) and a constructor NewA(AConf) A has to exist. The serialization method returns an universal Conf, so to use it in the constructor, it has to be asserted as AConf.

This serialization principle is defined as so for transparent instance transferring accross the whole cluster and its materialization elsewhere within the cluster.

Example

type AnimalConf struct {
	Weight int
	Size   int
}

type Animal struct {
	weight int
	size   int
}

func (ego *Animal) Breathe() {
	fmt.Println("Inhale...")
	fmt.Println("Exhale...")
}

type DogConf struct {
	AnimalConf
	Name string
	Age  int
}

type Dog struct {
	gonatus.Gobject
	Animal
	name        string
	yearOfBirth int
}

func NewDog(conf DogConf) *Dog {
	ego := new(Dog)
	ego.weight = conf.Weight
	ego.size = conf.Size
	ego.name = conf.Name
	ego.yearOfBirth = time.Now().Year() - conf.Age
	return ego
}

func (ego *Dog) Serialize() gonatus.Conf {
	return DogConf{
		AnimalConf: AnimalConf{
			Weight: ego.weight,
			Size:   ego.size,
		},
		Name: ego.name,
		Age:  time.Now().Year() - ego.yearOfBirth,
	}
}

func (ego *Dog) Bark() {
	fmt.Println("Woof")
}

Reinvented File System

Due to limitations of traditional file systems (max path legth, max name size, unicode problems, reserved characters, inode count, ...), we redefined the the architecture of a file system. The term "directory" does not exist, all records in the FS are called files. Each file may have content, topology (child records) or both (so files can also behave as directories). This way we can extract archives into the original file topology.

For more information, check fs/README.md.

Documentation

Index

• Constants
• type Conf
• type ConfWrapper
  • func WrapConf(conf Conf) ConfWrapper
• type DefaultService
  • func NewDefaultService[T RemoteObject](service Service, constructor func(Conf) (T, error)) *DefaultService[T]
  • func (ego *DefaultService[T]) Fetch(id GId) RemoteObject
  • func (ego *DefaultService[T]) New(conf Conf) (GId, error)
  • func (ego *DefaultService[T]) Remove(ids ...GId) error
• type DynamicConf
  • func NewConf(class string) DynamicConf
  • func (ego DynamicConf) Class() string
  • func (ego DynamicConf) Clone() DynamicConf
  • func (ego DynamicConf) Decode(ptr any) error
  • func (ego DynamicConf) Encode(ptr any) error
  • func (ego DynamicConf) Marshal() ([]byte, error)
  • func (ego DynamicConf) String() string
  • func (ego DynamicConf) Unmarshal(jsonBytes []byte) error
• type GId
• type Gobject
  • func (ego *Gobject) Log() *slog.Logger
  • func (ego *Gobject) SetLog(log *slog.Logger)
• type Gobjecter
• type GonatusKeeper
  • func (ego *GonatusKeeper) Fetch(id GId) Service
  • func (ego *GonatusKeeper) NewId() GId
  • func (ego *GonatusKeeper) Register(s Service)
  • func (ego *GonatusKeeper) Unregister(s Service) error
• type RemoteObject
• type Service

Constants

const ConfSuffix = "Conf" // Suffix of all configuration structures.

Types

type Conf

type Conf any

type ConfWrapper

type ConfWrapper struct {
	Class string
	Conf  Conf
}

func WrapConf

func WrapConf(conf Conf) ConfWrapper

type DefaultService

type DefaultService[T RemoteObject] struct {
	// contains filtered or unexported fields
}

func NewDefaultService

func NewDefaultService[T RemoteObject](service Service, constructor func(Conf) (T, error)) *DefaultService[T]

func (*DefaultService[T]) Fetch

func (ego *DefaultService[T]) Fetch(id GId) RemoteObject

func (*DefaultService[T]) New

func (ego *DefaultService[T]) New(conf Conf) (GId, error)

func (*DefaultService[T]) Remove

func (ego *DefaultService[T]) Remove(ids ...GId) error

type DynamicConf

type DynamicConf map[string]any

func NewConf

func NewConf(class string) DynamicConf

func (DynamicConf) Class

func (ego DynamicConf) Class() string

func (DynamicConf) Clone

func (ego DynamicConf) Clone() DynamicConf

func (DynamicConf) Decode

func (ego DynamicConf) Decode(ptr any) error

func (DynamicConf) Encode

func (ego DynamicConf) Encode(ptr any) error

func (DynamicConf) Marshal

func (ego DynamicConf) Marshal() ([]byte, error)

func (DynamicConf) String

func (ego DynamicConf) String() string

func (DynamicConf) Unmarshal

func (ego DynamicConf) Unmarshal(jsonBytes []byte) error

type GId

type GId uint64

type Gobject

type Gobject struct {
	// contains filtered or unexported fields
}

func (*Gobject) Log

func (ego *Gobject) Log() *slog.Logger

Log returns associated slog.Logger. If the logger is not initialized (nil), a DefaultLogger() is used instead

Returns:

• logger.

func (*Gobject) SetLog

func (ego *Gobject) SetLog(log *slog.Logger)

SetLog sets an associated slog.Logger of the Gobject.

Parameters: - logger.

type Gobjecter

type Gobjecter interface {
	Serialize() Conf
	SetLog(*slog.Logger)
	Log() *slog.Logger
}

type GonatusKeeper

type GonatusKeeper struct {
	Gobject
	// contains filtered or unexported fields
}

func (*GonatusKeeper) Fetch

func (ego *GonatusKeeper) Fetch(id GId) Service

func (*GonatusKeeper) NewId

func (ego *GonatusKeeper) NewId() GId

func (*GonatusKeeper) Register

func (ego *GonatusKeeper) Register(s Service)

func (*GonatusKeeper) Unregister

func (ego *GonatusKeeper) Unregister(s Service) error

type RemoteObject

type RemoteObject interface {
	Id() GId
	SetId(id GId)
}

type Service

type Service interface {
	RemoteObject
	New(Conf) (GId, error)
	Remove(ids ...GId) error
}