atomika

README

Atomika

---

Purpose

The goal of this project is to have a low-level, non-invasive framework which allows fast development of new software as well as refactoring old code.

Go is already fast as it is, but all the projects have a certain amount of redundancy which can cause a lot of frustrations. The part of creating the boilerplate is one of the most annoying and boring parts of the project.

Cases for motivation:

1. Multiple projects started by the same team but with different layouts, dependencies, and structures.
2. New developers unable to understand the original ideas and concepts behind a project do changes incompatible with the project structure or accidentally create jump or duplicate functionality many times unintentionally
3. Due to lack of consistency projects degrade very fast. Projects become unmaintainable even in under 6-8 months

I wanted to build this tool for this exact reason: to cover as many use cases as possible and make them look familiar despite their scopes.

The Suite

Macroscope Labs consists of two projects:

• Atomika - the underlying framework
• atomika - the CLI tool used to spin and generate applications build on top of Atomika.

The latest is completely optional, but its absence will reduce the speed of development.

Getting started

• Grab a copy of atomika CLI from the atomika Release page
• Unzip/Untar to a bin location such as /usr/local/bin or /home/user/go/bin

tar -C /aps/des/path -xzf /download/path/atomika_0.14.0_Linux-x86_64.tar.gz
mv /aps/des/path/atomika/atomika-linux-amd64 /usr/local/bin/atomika
sudo chown +x /usr/local/bin/atomika

• Test it atomika version

Atomika Version: 0.14.0

Because atomika is generating SDK code for various programming languages, run atomika doctor to make sure dependencies for a particular programming language are installed.

If all goes well you can run atomika or atomika --help to see all available commands

First project

1. Setup project folder and dependencies

atomika project create myproject

Select a place where the services for this project will be stored

? Project Services path [DEFAULT: ./] [hit enter]

At this point, you can hook the project to a preexisting repository. For this feature to work all git setup must be made. You need to have the git, and an ssh key properly installed into your git server account (Github, GitLab, bare).

? Repository path [DEFAULT: -] [hit enter]

Should see something like

➡ Create project folder: /home/user/my. OK
➡ Create project files. OK
➡ Checking for latest updates...
➡ Pulling latest updates...
	 Pulling: [gitlab.com/macroscope-lab/atomika@v0.12.5]
➡ Update finished. OK
⚠ git repo url not provided. Skipping

➡ Project successfully created. ID: my-iasuog!

All of these settings can be changed afterward.

cd into the project folder and run go run cmd/my.app/main.go

Is all went well you will get a simple log on screen

{"level":"info","app":"my.app","ver":"development","branch":"","compile":"","time":1680081640578,"message":"bootstrap"}

NOTE!

run and buildcommands are not available yet. Please refer to the Issues page to see when this features will be available.

2. Add your first service

From within the project folder run

atomika service create hello

Output

➡ Service successfully created. ID: hello-lnaopl!

A new folder should be added in the specified service folder or on root in none specified You can also check project.json from the root to make sure the service was set correctly.

{
 "id": "my-iasuog",
 "repo": "-",
 "servicesPath": "services",
 "services": [
  {
   "id": "hello-lnaopl",
   "path": "services/hello",
   "version": "0.0.1"
  }
 ]
}

The service can be used in three ways.

1. As dependency on another service
2. As an internal worker
3. Most frequently As a client-facing service with an http API access

3. Writing service business logic

From the begging, I'd like to mention that Macroscope Labs does not infer any style of writing code nor adds dependency through code in any way. All the following code is plain Go code with a high affinity for clean architecture design pattern.

Defining the service

In the service folder create another folder called def (from definition) In def create a file def.go (you can name it however you like)

Open the file and define the methods of your service

type GreetingService interface {
	Hi(HiReq) HiRes
}


type HiReq struct {
	Name string `json:"name"`
}

type HiRes struct {
	Message string `json:"message"`
}

The code above resembles a lot the protobuf .proto definitions.

Running atomika project gen will generate a file called server.go inside your service folder. The name is chosen to help hint to you that this service has client-facing support.

atomika project gen will go over the entire project and generate ALL your services dependency files.

If you want to generate/re-generate only one service you can opt for atomika service gen <service>

The contents of server.go follow a close pattern to the ones generated by protobuf.

In your service folder add a new file service.go. Again, this name is not enforced but is meant to hint the entry point of the service. In the case of debugging this is where it all starts. As guessed we need to implement the methods from def/def.go which not are copied into server.go

Implement service functionality methods

type greetingService struct {
    UnimplementedGreetingService
}

func NewGreetingService() GreetingService {
    return &greetingService{}
}

Same as protobuf you can include and UnimplementedGreetingService if skipping methods implementation is required

Registering the service

Include the service into the atomika service runner

In your cmd/my/main.go add and register your new service

func run() error {
	l := log.WithContext("app", App, "ver", Version, "branch", Branch, "compile", Compile)

	// Dependencies
	trans := transport.NewWithAutoConf()

	// Internal Services - Services with no transport definitions

	// Service
	greetingService := hello.NewGreetingService()

	// Register services to servers
	hello.RegisterGreetingServiceServer(trans, greetingService)

	// Bootstrap
	app := atomika.New()
	app.RegisterService([]atomika.Service{
		trans,
	})

	l.Info("bootstrap")

	return app.Boot()
}

Code explanation

• trans := transport.NewWithAutoConf() - Creates a new HTTP transporter and uses the default settings provided in .project/local.json

• greetingService := hello.NewGreetingService() - Instantiate the service. Feel free to alter NewGreetingService signature in case you want to pass in dependencies

• hello.RegisterGreetingServiceServer(trans, greetingService) - It's registering the service to a transportation layer

• The following snippet registers all transports (at present only HTTP) and boots the application

app := atomika.New()
app.RegisterService([]atomika.Service{
   trans,
})
app.Boot()

As probably one notice, if for whatever reason you need multiple HTTP transports you can have more (Separate sync calls from ws/async calls might be one example)

Finally, run the project go run cmd/my.app/main.go

Resounding success!

{"level":"info","app":"my.app","ver":"development","branch":"","compile":"","time":1680102110990,"message":"bootstrap"}
{"level":"info","port":":8080","time":1680102110990,"message":"HTTP Transport started"}

Open your favorite API test application and give it a go

>> curl -X POST http://localhost:8080/rpc/GreetingService/Hi --compressed -H "Content-Type: application/json"
{"error":{"code":6,"msg":"unable to unmarshal payload"}}

Notice: {"error":{"code":6,"msg":"unable to unmarshal payload"}} is because this framework is meant to be used with an SDK an empty payload should never be the case

Let's retry!

>> curl -X POST http://localhost:8080/rpc/GreetingService/Hi --compressed -H "Content-Type: application/json" -d "{}"
{"error":{"code":6,"msg":"method [Hi] is unimplemented"}}

As expected we have touched the UnimplementedGreetingService service Hi method

Add business logic

Open the service.go file and update it as follows

type greetingService struct {
	
}

func (greetingService) Hi(_ context.Context, req HiReq) (*HiRes, error) {
    return &HiRes{
        Message: fmt.Sprintf("Hello, %s", req.Name),
    }, nil
}

func NewGreetingService() GreetingService {
	return &greetingService{}
}

Notice we have removed the receiver variable and suppressed the context variable as they are not needed. We also removed UnimplementedGreetingService as we are implementing all the methods.

If the route is present and already registered a warning is issued and the new, same-name route will be skipped.

Restart the app and call:

>> curl -X POST http://localhost:8080/rpc/GreetingService/Hi --compressed -H "Content-Type: application/json" -d "{\"name\": \"user\"}"
{"message":"Hello, user"}

And scene!

4. Adding an SDK

The purpose of the SDK is to simplify integration by adding more concision and to provide at least some basic data validation.

Run

atomika project gen --help

generate project dependencies

Usage:
  atomika project gen [flags]

Flags:
  -h, --help     help for gen
      --sdk-go   generate go sdk
      --sdk-js   generate javascript sdk
      --sdk-ts   generate typescript sdk

For now, only js, ts and go are supported Let's generate the sdk for go

atomika project gen --sdk-go

or for single service

atomika service gen hello --sdk-go

Your SDKs are stored in the root folder sdk/<service name>. These SDKs are meant to be copied by the services willing to interact with this service.

For simplicity of this example create a new file in cmd/my.app/client/main.go

package main

import (
	"context"
	"fmt"
	"log"
	"my.app/sdk/hello"
)

func main() {

	// HTTP Client 
	client := hello.New("http://localhost:8080/rpc")

	// Create the service
	serviceClient := hello.NewGreetingServiceClient(client)

	res, err := serviceClient.Hi(context.TODO(), hello.HiReq{
		Name: "User",
	})

	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("RES: %+v", res)
}

>> go run cmd/my.app/client/main.go 
RES: &{Message:Hello, User Error:}

Again, same as a protobuf, we create a service client. The serviceClient can be used as dependency.

Operational files and folders

There are three types of such operational files and folders.

1. .project folder where you should store all the long-term configurations for the project. The underlying library is the well-known spf13/viper. For now, only JSON config file type is supported.
2. project.json is where the dependency nesting is stored (In the future will add the ability to migrate modules to another atomika project)
3. <service>/service.json the service description (In the future can be committed and versioned independently, as well as generate project architecture on the fly)

Naming convention

The rpc route is built using the interface name and method name e.g. GreetingService/Hi. In case of an error, you will know exactly where to go and where to look. These names are not validated in any way for now so you can choose the names you see fit.

Interceptors

Can be seen as middleware. They are prioritized from global to the method in this order.

A simple interceptor definition

type DummyInterceptor struct {
}

func (i DummyInterceptor) Intercept(next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // Do stuff here and 
        fmt.Println("DummyInterceptor executed. Trace key:", r.Context().Value(transport.TraceKey), "Time", time.Now().UTC().UnixMilli())
        time.Sleep(1 * time.Second)
		
        // either return and http response with an error or
        next.ServeHTTP(w, r)
    }
}

In your main.go or wherever you see fit you can create stacks of suck interceptors based on the problem you want to solve

TraceKey is a useful built-in feature to help grouping a set of calls

// main.go

func run() error {
	// ... 
    trans := transport.NewWithAutoConf()
    intercept(trans)
	
	// ...
}

func intercept(trans transport.Transport) {
    globalInterceptors := []transport.Interceptor{
        interceptors.DummyInterceptor{},
    }
    
    serviceInterceptors := []transport.Interceptor{
        interceptors.DummyInterceptor{},
    }
    
    methodInterceptors := []transport.Interceptor{
        interceptors.DummyInterceptor{},
    }
    
    // Will be executed on all calls
    trans.Use(transport.GlobalUse, globalInterceptors)
    
    // Will be called ONLY by `GreetingService`
    trans.Use("GreetingService", serviceInterceptors)
    
    // Will be called ONLY by `GreetingService/Hi`
    trans.Use("GreetingService/Hi", methodInterceptors)
}

Output as expected:

{"level":"info","app":"my.app","ver":"development","branch":"","compile":"","time":1680105658562,"message":"bootstrap"}
{"level":"info","port":":8080","time":1680105658562,"message":"HTTP Transport started"}
DummyInterceptor executed. Trace key: DZ9GNFKYcvDb0zep Time 1680105916443
DummyInterceptor executed. Trace key: DZ9GNFKYcvDb0zep Time 1680105917443
DummyInterceptor executed. Trace key: DZ9GNFKYcvDb0zep Time 1680105918443

Interceptors dependencies

Like the service, interceptors can receive dependencies.

For example

type DummyInterceptor struct {
	AppName string
}

Update the intercept function and simply pass the dependency

func run() error {
    // ... 
    trans := transport.NewWithAutoConf()
    intercept(trans, "my.app")
    
    // ...
}

func intercept(trans transport.Transport, appName string) {
    // Interceptors setup
    dummyInterceptor := interceptors.DummyInterceptor{
        AppName: appName,	
    },
	
    // Interceptors groups
    globalInterceptors := []transport.Interceptor{
        dummyInterceptor,
    }
	
    // ...
	
}

Services configuration

In real life, there are various ways through which you can receive configuration information based on various reasons: security, architecture, team agreements, etc. You can receive or get configuration data from local or remote files, env variables, or just plain static or default variables

In this regard, there is a runtime package to help exactly with this situations

With Options

Going to our my project open file hello/options.go

type Option func(o *Options)

type Options struct {
    AppName string
}

func WithAppName(v string) Option {
    return func(o *Options) {
        o.AppName = v
    }
}

func setOptions(opts ...Option) Options {
    defaults := defaultOptions()
    for _, opt := range opts {
        opt(&defaults)
    }

    return defaults
}

func defaultOptions() Options {
    return Options{
        AppName: "atomika.app",
    }
}

Conventions

• options.go - file name is to hint to the developer this package can be configured
• function name With+<Option propery name> to tell when used what variable is being set
• setOptions helper function for overwriting defaults
• defaultOptions helper function for setting default values

1. Alter your hello/service.go constructor function to be able to retrieve these options. I tend to have only calls and no logic in my public methods in case I want to change things later, to be able to do it without touching the method/function request/response signatures.

type greetingService struct {
    AppName string // Add variable here
}

func NewGreetingService(opts ...Option) GreetingService {
	return newGreetingService(setOptions(opts...))
}

func newGreetingService(opts Options) GreetingService {
    return &greetingService{
        AppName: opts.AppName,
    }
}

Because all options are optional if you run your project the options code will fix the options with defaults.

1. Update cmd/my/main.go

// Service
greetingService := hello.NewGreetingService(
    hello.WithAppName(App),
)

1. Update GreetingService/Hi method response to check it out

func (s greetingService) Hi(_ context.Context, req HiReq) (*HiRes, error) {
    return &HiRes{
        Message: fmt.Sprintf("Hello, %s from %s", req.Name, s.AppName),
    }, nil
}

1. Run you app and call GreetingService/Hi

{
    "message": "Hello, user T from my.app"
}

With Auto config

All is well with the above options IF you know these values beforehand. What if the AppName is provided through a file and another configuration value, let's say AppVersion, is provided via an env variable or a secret?

This is where autoconf pattern comes in handy (It's called pattern because is not enforced through code in any way, this is just our suggestion).

1. Create a new file in hello service called runtime.go

package hello

import (
	"errors"
	"gitlab.com/macroscope-lab/atomika/runtime"
)

type Runtime struct {
	AppName    string `mapstructure:"appName"`
	AppVersion string `mapstructure:"appVersion"`
}

func (c *Runtime) Configure(key ...string) error {
	return runtime.Get(c, key...)
}

func (c *Runtime) Bind() {
	runtime.BindKeyToEnv("appName", "my_APP_NAME")
	runtime.BindKeyToEnv("appVersion", "my_APP_VER")
}

func (c *Runtime) Validate() error {
	if c.AppName == "" {
		return errors.New("app name is required")
	}
	return nil
}

This file implements several interfaces:

type Validatable interface {
	Validate() error
}

type Bindable interface {
	Bind()
}

type Configurable interface {
	Configure(key ...string) error
}

All of these are being called automatically by the runtime package:

1. Configurable - To search into a raw byte array for data matching your configuration

2. Bindable - To map your data as an extra feature, to an env variable

3. Validatable - To validate the data according to your specifications

4. Create a new josn file in .project, my.json

{
  "appName": "DEMO APP"
}

1. Update your runtime.go file to add support for autoconf

func OptionsFromRuntime() (Options, error) {
	defaults := defaultOptions()
	rt := &Runtime{}

	if err := rt.Configure(); err != nil {
		return defaults, err
	}

	if err := rt.Validate(); err != nil {
		return defaults, err
	}

	// Overwrite each options
	withAppName := WithAppName(rt.AppName)
	withAppName(&defaults)

	withAppVersion := WithAppVersion(rt.AppVersion)
	withAppVersion(&defaults)

	return defaults, nil
}

Alternative runtime approach

---

Another flavor here could be to remove OptionsFromRuntime method altogether and put everything in main.go, but this approach might bloat the code a bit if not done carefully.

Edit the main.go as follows:

gsRuntime := &hello.Runtime{}
if err := gsRuntime.Configure("hello"); err != nil {
    return err
}

if err := gsRuntime.Validate(); err != nil {
    return err
}


// ...
greetingService, err := hello.NewGreetingService(
    hello.WithAppName(gsRuntime.AppName),
    hello.WithAppVersion(gsRuntime.AppVersion),
)

Notice the hello key in gsRuntime.Configure("hello") this allows you to specify a key in your JSON file where to look.

{
  "appName": "DEMO APP",
  "appVersion": "v1.0",
  "hello": {
    "appName": "DEMO APP V2"
  }
}

The first appName and appVersion will be ignored in favor of the contents of the hello key.

---

1. Update your hello/service.go file to add autoconf support

//...

func NewGreetingService(opts ...Option) (GreetingService, error) {
    return newGreetingService(setOptions(opts...))
}

func NewGreetServiceWithAutoConf() (GreetingService, error) {
    opts, err := OptionsFromRuntime()
    if err != nil {
        return nil, err
    }
    return newGreetingService(opts)
}

func newGreetingService(opts Options) (GreetingService, error) {
// ...

Now your service has two ways of setup. If you think either of them is not being used feel free to remove it later on.

1. Update the main.go and let the service be set up via the new NewGreetServiceWithAutoConf

// Service
greetingService, err := hello.NewGreetServiceWithAutoConf()
if err != nil {
    return err
}

1. Update GreetingService/Hi method to illustrate better what is going on

type greetingService struct {
	AppName    string
	AppVersion string
}

// ...
func (s greetingService) Hi(_ context.Context, req HiReq) (*HiRes, error) {
    return &HiRes{
        Message: fmt.Sprintf("Hello, %s from %s:Ver:%s", req.Name, s.AppName, s.AppVersion),
    }, nil
}

// newGreetingService
// Don't forget to include the var on constructor
func newGreetingService(opts Options) (GreetingService, error) {
    return &greetingService{
        AppName:    opts.AppName,
        AppVersion: opts.AppVersion,
    }, nil
}

1. Run the app

{
    "message": "Hello, user T from DEMO APP:Ver:"
}

Notice the version is correctly not set as we opted in default to an empty string.

1. Let's add the default to v1.0 in our .project/my.json file

{
    "appName": "DEMO APP",
    "appVersion": "v1.0"
}

If we run the app we will get:

{
    "message": "Hello, user T from DEMO APP:Ver:v1.0"
}

1. Let's assume we need to overwrite this with an environment variable.

export my_APP_VER=v2.0
go run cmd/my/main.go

And the output would be:

{
    "message": "Hello, user T from DEMO APP:Ver:v2.0"
}

Config values priority goes from lowest to highest

1. defaultValue/variable
2. config file
3. env variable

Credits and Recognition

Massive thanks to all!

For inspiration and ideas

• oto - Tone of inspiration for code generation

Dependency libraries in this project:

• spf13/viper
• stretchr
• go-playground

And last but not least, two open source projects which need no introduction

• Redis
• Postgres

Documentation

Index

• type Atomika
  • func New() Atomika
• type Service

Types

type Atomika

type Atomika interface {
	RegisterService([]Service)
	Boot() error
}

func New

func New() Atomika

type Service

type Service interface {
	Run(ctx context.Context) error
}