README ¶
Example gotro/W2 Project
How to use this template?
- install Go 1.16+ and clone this repo with
--depth 1
flag - copy this
example1
directory to another folder (rename toprojectName
) go mod init projectName
- replace all word
github.com/kokizzu/gotro/W2/internal/example1
andexample1
withprojectName
How to develop?
- modify or create new
model/m*/*_tables.go
, then runmake gen-orm
(will generate ORM), you may only add column/field at the end of model. - create a new
*_In
,*_Out
,*_Url
, and the business logic methods insidedomain
, then runmake gen-route
(will generate routes and API docs). - create an integration/unit test to make sure that your code is correct
How to release?
- change
production/
configuration values - setup the server, ssh to the production/staging server and run
setup_server.sh
- cd
production
, run./sync_service.sh
- run
./deploy_prod.sh
How to do multi server?
- replace id64 with string, for example1 lexid or standard
uuid
- add an environment variable for SERVER_ID, and init it as
lexid.ServerId
- before running deployment script, make sure to append environment variable SERVER_ID that are must unique per server
How to do multi database?
- see this blog post
Directory Structure
3rdparty
- all third party wrapper should be here as a subfolderconf
- all configuration constantsdomain
- contains your business logic, these are the one that should be integration/unit testedmodel
- contains your domains' data storem[Domain]
- contains data store that should be grouped inside that domainrq[Domain]
- read query (R from CQRS), you can add a new file here to extend the default ORMsa[Domain]
- statistics analytics (event source), you can add a new file here to extend the default ORMwc[Domain]
- write command (C from CQRS), you can add a new file here to extend the default ORM*_table.go
- the schema file for that domain, to generate the ORM and as an input for migration
production
- scripts and env for deploying to productionsvelte
- frontend (can be replaced with any framework)
outer files:
main_*.GEN.go
- will be generated per transport/presentation/adapter (eg. gRPC, REST, WebSocket, CLI, etc)
Setup
# install tools required for codegen
make setup-deps
# install reverse proxy
make setup-webserver
# install dependencies for web frontend (Svelte with ESBuild): localhost:5500
make webclient
# start dependencies (Tarantool, Clickhouse, Mailhog): localhost:3301, localhost:9000, localhost:1025
make compose
# run api server (Go with Air auto-recompile): localhost:9090
make apiserver
# run reverse proxy (Caddy): localhost:80
make reverseproxy
Usage
# connect to OLTP database
tarantoolctl connect 3301
# connect to OLAP database
clickhouse-client
# generate ORM (after add new table or columns on models/m*/*_tables.go)
make gen-orm
# generate route (after add new _In+_Out struct, _Url const and business logic method on domain/*.go)
make gen-route
Every new *.svelte
file will automatically generate corresponding *.html
file, also will automatically generate index.GEN.go
, so you can create proper handler to inject into _layout.html
or json into the generated *.html
file. For now, please rerun make webclient
after adding new *.svelte
to generate new route.
Gotchas
- Calling direct assignment (
=
) instead ofwc*.Set*()
before callingwc*.DoUpdateBy*()
will do nothing, as direct assignment does not append mutation property
# proper way to update
x := mAuth.NewUsersMutator(s.Taran)
x.Id = ...
if !x.FindById() {
// not found
return // or x.DoInsert() then continue with update
}
x.SetBla(..)
x.SetFoo(..)
x.SetBar(..)
x.SetBaz(..)
x.SetBaz(..) // calling twice on the same column will cause DoUpdate to fail
if !x.DoUpdateById() {
// failed to update
}
# but if you need only insert or replace, you can use = directly
x := mAuth.NewUsersMutator(s.Taran)
x.Bla = ..
x.Foo = ..
x.Bar = ..
x.Baz = ..
x.DoInsert() or x.DoUpsert() // calling DoUpdateBy*() will do nothing, since mutation property only set when calling .Set*() method
- Clickhouse inserts are buffered using chTimedBuffer, so you must wait ~1s to ensure it's flushed
- Clickhouse have eventual consistency, so you must use
FINAL
query to make sure it's force-committed - You cannot change Tarantool's datatype
- You cannot change Clickhouse's ordering keys datatype
- Currently migration only allowed for adding columns/fields at the end (you cannot insert new column in the middle/begginging)
- All Tarantool's columns always set not null after migration (I hate null values XD)
- Tarantool does not support client side transaction (so you must use Lua or 2PC or split into SAGAs)
- Current parser/codegen does not allow calling SetError with more than 1 concatenation or complex expression or non constant left-hand-side, eg.
d.SetError(500, "error on" + Bla(bar) + Yay(baz))
, you must repharase the error detail into something like this:d.SetError(500, "error on " + msg)
TODOs
- Add SEO pre-render: Rendora
- Add search-engine: TypeSense or MeiliSearch for single node
- Add more persisted cache option: IceFireDB or Aerospike
- Add external storage upload example1 (minio? wasabi?)
- Replace LightStep with SigNoz, tutorial, NewRelic and/or datav or just dump it to Clickhouse directly and MQ like RedPanda as fallback
- Add Analytics: Jitsu or Materialize
- Add example1 deployment script with LXC/LXD share for single server multi-tenant or Docker Compose or Docker Swarm
- Add NBIO codegen for websocket presentation/transport layer
File Upload Example
the schema (/model/m[Something]/[something]_tables.go
), after creating this, run make gen-orm
:
const (
TableMediaUploads Tt.TableName = `mediaUploads`
Id = `id`
CreatedBy = `createdBy`
CreatedAt = `createdAt`
UpdatedBy = `updatedBy`
UpdatedAt = `updatedAt`
DeletedBy = `deletedBy`
DeletedAt = `deletedAt`
IsDeleted = `isDeleted`
RestoredBy = `restoredBy`
RestoredAt = `restoredAt`
SizeByte = `sizeByte`
FilePath = `filePath`
ContentType = `contentType`
OrigName = `origName`
)
var TarantoolTables = map[Tt.TableName]*Tt.TableProp{
TableMediaUploads: {
Fields: []Tt.Field{
{Id, Tt.Unsigned},
{CreatedAt, Tt.Integer},
{CreatedBy, Tt.Unsigned},
{UpdatedAt, Tt.Integer},
{UpdatedBy, Tt.Unsigned},
{DeletedAt, Tt.Integer},
{DeletedBy, Tt.Unsigned},
{IsDeleted, Tt.Boolean},
{RestoredAt, Tt.Integer},
{RestoredBy, Tt.Unsigned},
{SizeByte, Tt.Unsigned},
{FilePath, Tt.String},
{ContentType, Tt.String},
{OrigName, Tt.String},
},
Unique1: Id,
Unique2: FilePath,
},
}
func GenerateORM() {
Tt.GenerateOrm(TarantoolTables)
}
// don't forget to add migration on model.go:
// m.Taran.MigrateTables(mSomething.TarantoolTables)
the code for domain/business logic /domain/media.go
, after creating this, run make gen-route
:
type (
MediaUpload_In struct {
RequestCommon
UploadId uint64 `json:"uploadId,string" form:"uploadId" query:"uploadId" long:"uploadId" msg:"uploadId"`
FileBinary string `json:"fileBinary" form:"fileBinary" query:"fileBinary" long:"fileBinary" msg:"fileBinary"`
}
MediaUpload_Out struct {
ResponseCommon
MediaUpload *rqSomething.MediaUploads `json:"mediaUpload" form:"mediaUpload" query:"mediaUpload" long:"mediaUpload" msg:"mediaUpload"`
}
)
const MediaUpload_Url = `/MediaUpload`
func (d *Domain) MediaUpload(in *MediaUpload_In) (out MediaUpload_Out) {
sess := d.mustAdmin(in.SessionToken, in.UserAgent, &out.ResponseCommon)
if sess == nil {
return
}
if len(in.Uploads) == 0 {
out.SetError(400, `missing fileBinary, enctype not multipart/form-data?`)
return
}
up := wcSomething.NewMediaUploadsMutator(d.Taran)
up.Id = in.UploadId
if in.UploadId > 0 {
if !up.FindById() {
out.SetError(404, `upload not found, wrong env?`)
return
}
}
if up.CreatedAt == 0 {
up.Id = id64.UID()
up.CreatedAt = in.UnixNow()
up.CreatedBy = sess.UserId
}
up.UpdatedAt = in.UnixNow()
up.UpdatedBy = sess.UserId
for fileName, tmpFile := range in.Uploads {
up.OrigName = fileName
oldPath := up.FilePath
uriPath := conf.UPLOAD_URI + conf.MEDIA_SUBDIR
dir := conf.UPLOAD_DIR + conf.MEDIA_SUBDIR
idStr := I.UToS(up.Id)
if S.StartsWith(oldPath, uriPath) {
oldPath = dir + S.RightOf(oldPath, uriPath)
} else if oldPath != `` {
L.Print(`ERROR weird name format to be replaced for mediaUpload.id: ` + idStr + `:` + oldPath)
}
mtype, err := mimetype.DetectFile(tmpFile)
if L.IsError(err, `cannot detect file type: `+tmpFile) {
out.SetError(500, `cannot detect file type: `+up.OrigName)
return
}
err = os.MkdirAll(dir, 0755)
if L.IsError(err, `failed to create upload directory: `+dir) {
out.SetError(500, `cannot create upload directory`)
return
}
ext := S.ToLower(filepath.Ext(fileName))
newName := idStr + ext
newPath := dir + newName
err = os.Rename(tmpFile, newPath)
if L.IsError(err, `failed to rename `+tmpFile+` to `+newPath) {
out.SetError(500, `failed moving uploaded file`)
return
}
in.Uploads[fileName] = oldPath // delete old file later
up.FilePath = uriPath + newName
stat, err := os.Stat(newPath)
if L.IsError(err, `failed to stat moved file: `+newPath) {
out.SetError(500, `failed to stat moved file`)
return
}
up.SizeByte = uint64(stat.Size())
up.ContentType = mtype.String()
// ignore if upload more than one
break
}
//if in.DoDelete {
// up.IsDeleted = true
// up.DeletedAt = in.UnixNow()
// up.DeletedBy = sess.UserId
//}
//if in.DoRestore {
// up.IsDeleted = false
// up.RestoredAt = in.UnixNow()
// up.RestoredBy = sess.UserId
//}
if !up.DoUpsert() {
out.SetError(500, `cannot upsert media`)
return
}
out.MediaUpload = &up.MediaUploads
return
}
Testing with CURL
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
time curl -X POST -H 'content-type: application/json' -d '{"email":"root@gmail.com","password":"123"}' http://localhost:9090/api/UserLogin
time curl -X POST -H 'content-type: application/json' -d '{"email":"root@gmail.com","password":"123","userName":"kokizzu"}' http://localhost:9090/api/UserRegister
cd; go install github.com/rakyll/hey@latest
hey -c 50 -n 200 http://localhost:9090/api/health
hey -c 1000 -n 100000 http://localhost:9090/api/UserLogin\?email\=root@gmail.com
Testing GraphQL
After starting docker-compose up
and make apiserver
, open localhost:9090/graphql and run these query:
mutation _ {
UserLogin(email: "root@localhost", password: "test123", debug: true) {
ResponseCommon {
debug
error
sessionToken
statusCode
}
}
}
TODOs:
- add more tests (eg. multiple cart)
- generate graphql schema with relationship to other model (and max nested and rate limiter)
Documentation ¶
There is no documentation for this package.