tag_api

package module
v0.0.0-...-3576469 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 14, 2019 License: Apache-2.0 Imports: 21 Imported by: 0

README

tag_api

API using Go-lang struct tags to load SQL data and implement JSON endpoints.

Goal

Demonstrate how simple it is to prototype and modify an API in Go.

By simply adjusting or adding a field to a Go struct, you automatically update both how the server loads from the database, as well as what it outputs for the API endpoint.

  • This project builds a MySQL database Docker image, initialized with sample data
  • The sample data uses photos from https://clients3.google.com/cast/chromecast/home
  • It also builds an api-server application in Go, which it installs on a Docker image

It uses govvv to provide the Github version string in the application. Install as follows:

go get "github.com/ahmetb/govvv"

Then clone the tag_api project:

go get "github.com/DavidSantia/tag_api"
cd $GOPATH/src/github.com/DavidSantia/tag_api

Building and Running the System

First install govvv, clone this project, and start in the tag_api directory as shown above.

Next, run the build.sh script to compiles the apps and images.

cd docker
./build.sh

Then type docker images, and you should see the following images:

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
tagdemo/api-server              latest              0fc6d6e09ab2        About an hour ago   8.98MB
tagdemo/mysql                   latest              9d795daac22c        About an hour ago   255MB

There is also a clean.sh script to remove containers and images from your previous builds.

./clean.sh

Finally, start the database, NATS server, and api-server as follows:

docker-compose up

The NATS server handles communication between the services. In this Basic example, the API server server subscribes to NATS, but it doesn't do anything further.

Accessing the API

Once you have the API server up and running, use your browser to authenticate and access data. The following endpoints are defined:

router.Handle("GET", "/authenticate", handleAuthTestpage)
router.Handle("POST", "/authenticate", makeHandleAuthenticate(cs))
router.Handle("GET", "/keepalive", makeHandleAuthKeepAlive(cs))
router.Handle("GET", "/image", makeHandleAllImages(cs))
router.Handle("GET", "/image/:Id", makeHandleImage(cs))
router.Handle("GET", "/user", makeHandleUser(cs))

By browsing to localhost:8080/authenticate, you will see a test page with two buttons.

Each button authenticates you as a particular user from the sample database, either in the Basic or Premium group. Once authenticated, your browser will have a Session cookie to allow you to continue using the API.

You can then browse to

Monitoring

The API server and the MySQL container both have integrations with New Relic. If you have a license key for this, you can enable monitoring simply by setting a docker environment variable.

cd docker
cp newrelic.template.env newrelic.env

Then edit newrelic.env and set the license key on the first line:

NEW_RELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE

Running docker-compose will automatically use this key, and you should see additional lines in the logs like so:

api-server      |  INFO: 2019/02/24 01:18:32 New Relic monitor started

and

tagdemo-mysql   | INFO: New Relic monitor started

The build.sh script creates an empty stub of newrelic.env if you are not using monitoring.

Developing

For developing, you can run this server locally without Docker.

Build and Run the Database

Build the database manually as follows:

cd docker
docker build -t tagdemo/mysql ./mysql

You can then start the MySQL container as follows:

docker run --name tagdemo-mysql --rm -p 3306:3306 tagdemo/mysql

As shown above, we are mapping the MySQL default port 3306 from the container, to 3306 on localhost.

  • If this conflicts with a local installations of MySQL server, specify a different port
  • If you change the MySQL port, also specify -dbport in the api-server/Dockerfile entrypoint.

The database will be ready after you see the message:

[Entrypoint] MySQL init process done. Ready for start up.

If you need to stop the MySQL container, use

docker stop tagdemo-mysql
Build and Run the API Server

In a separate terminal, build the API server as follows:

cd apps/api-server
govvv build

Use the help option to get command-line usage

./api-server -h
Usage of ./api-server:
  -bolt string
    	Specify BoltDB filename (default "./content.db")
  -dbhost string
    	Specify DB host
  -dbport string
    	Specify DB port (default "3306")
  -debug
    	Debug logging
  -host string
    	Specify Api host
  -log string
    	Specify logging filename
  -nhost string
    	Specify NATS host
  -nport string
    	Specify NATS port (default "4222")
  -port string
    	Specify Api port (default "8080")

Then run the server as follows:

./api-server -dbhost 127.0.0.1 -dbload -debug

How it works

The database loader uses the Go reflect package to auto-generate SELECT statements from the struct tags.

Example Struct
type Image struct {
	Id           int64   `json:"id" db:"id"`
	Width        int64   `json:"width" db:"width"`
	Height       int64   `json:"height" db:"height"`
	Url          string  `json:"url" db:"url"`
	Title        *string `json:"title" db:"title"`
	Artist       *string `json:"artist" db:"artist"`
	Gallery      *string `json:"gallery" db:"gallery"`
	Organization *string `json:"organization" db:"organization"`
}

Tags shown above are interpreted as follows:

  • json: field name returned in API
  • db: field name in SQL
  • sql: optional SQL for SELECT

The sql tag is useful when

  • using joined statements with otherwise ambiguous field names
  • you want to insert an IFNULL or other logic
func (data *ApiData) makeQuery
func (data *ApiData) MakeQuery(dt interface{}, query string, v ...interface{}) (finalq string)

This takes two inputs:

  • dt: the struct you are loading data into
  • query: the FROM and WHERE part of a query

It can also take optional v parameters. If using these, include a format 'verb' (see the Go fmt package) in your query for each parameter.

It returns one output, the generated query.

Example Code
query = makeQuery(Image{},"FROM images i WHERE i.media IS NOT NULL"),

This combines the fields in the Image() struct, along with the FROM clause provided, as follows:

SELECT id, width, height, url, title, artist, gallery, organization
FROM images i WHERE i.media IS NOT NULL

The api-server -debug flag reveals the generated SQL queries.

DEBUG: 2019/05/14 16:31:59 GroupQuery: SELECT id, name, sess_seconds
FROM groups g
 INFO: 2019/05/14 16:31:59 Load Groups: 3 entries total
DEBUG: 2019/05/14 16:31:59 ImageQuery: SELECT id, width, height, url, title, artist, gallery, organization
FROM images i WHERE i.media IS NOT NULL
 INFO: 2019/05/14 16:31:59 Load Images: 12 entries total

Using the sqlx package, the code loads each object with a single rows.StructScan() step as shown:

for rows.Next() {
	err = rows.StructScan(&image)
	if err != nil {
		fmt.Printf("Load Images: %v\n", err)
		continue
	}
	bs.ImageMap[image.Id] = image
}

Above we see all the parameters from makeQuery() are loaded into an image object.

Documentation

Index

Constants

View Source 
const (
	// Retries to wait for docker DB instance
	DbConnectRetries = 5

	// NATS server
	NHost = "nats://localhost:4222"
	NSub  = "update"
)

Folders and credentials

Variables

View Source 
var GroupSegment = newrelic.DatastoreSegment{
	Collection:         "groups",
	Operation:          "SELECT",
	ParameterizedQuery: makeQuery(Group{}, "FROM groups g"),
}
View Source 
var ImageGroupSegment = newrelic.DatastoreSegment{
	Collection:         "images_groups",
	Operation:          "SELECT",
	ParameterizedQuery: makeQuery(ImagesGroups{}, "FROM images_groups ig"),
}
View Source 
var ImageSegment = newrelic.DatastoreSegment{
	Collection:         "images",
	Operation:          "SELECT",
	ParameterizedQuery: makeQuery(Image{}, "FROM images i WHERE i.media IS NOT NULL"),
}
View Source 
var JwtKey = []byte{194, 164, 235, 6, 138, 248, 171, 239, 24, 216, 11, 22, 137, 199, 215, 133}

16-byte JSON Web Token encryption key

View Source 
var SessionKey = []byte("something-very-secret")

Session key

View Source 
var UserSegment = newrelic.DatastoreSegment{
	Collection:         "users",
	Operation:          "SELECT",
	ParameterizedQuery: makeQuery(User{}, "FROM users u WHERE u.id = ?"),
}

Functions

func GetGroupIdFromSession 

func GetGroupIdFromSession(r *http.Request) (gid int64, err error)

func HandleError 

func HandleError(w http.ResponseWriter, status int, uri string, err error)

func HandleReply 

func HandleReply(w http.ResponseWriter, status int, j string)

func Index 

func Index(w http.ResponseWriter, r *http.Request, _ httprouter.Params)

func NewLog 

func NewLog(level Level, logfile string)

func WrapRouterHandle 

func WrapRouterHandle(app newrelic.Application, handle httprouter.Handle) httprouter.Handle

Types

type ApiData 

type ApiData struct {
	// contains filtered or unexported fields
}

Local data - most functions are methods of this

func NewData 

func NewData(host, port string) (data *ApiData)

func (*ApiData) InitSessions 

func (data *ApiData) InitSessions()

func (*ApiData) NewRouter 

func (data *ApiData) NewRouter(cs ContentService, ds *DbService, app newrelic.Application) (router *httprouter.Router)

func (*ApiData) StartServer 

func (data *ApiData) StartServer()

type BoltService 

type BoltService struct {
	GroupMap GroupMap
	ImageMap ImageMap
	// contains filtered or unexported fields
}

func (*BoltService) CloseNATS 

func (bs *BoltService) CloseNATS()

func (*BoltService) ConfigureNATS 

func (bs *BoltService) ConfigureNATS(host, port, channel string)

func (*BoltService) ConnectNATS 

func (bs *BoltService) ConnectNATS() (err error)

func (*BoltService) EnableLoadAll 

func (bs *BoltService) EnableLoadAll()

func (*BoltService) GetGroup 

func (bs *BoltService) GetGroup(id int64) (group Group, ok bool)

func (*BoltService) GetImage 

func (bs *BoltService) GetImage(id int64) (image Image, ok bool)

func (*BoltService) ListenForUpdates 

func (bs *BoltService) ListenForUpdates()

func (*BoltService) LoadCacheUpdates 

func (bs *BoltService) LoadCacheUpdates() (err error)

func (*BoltService) LoadFromDb 

func (bs *BoltService) LoadFromDb(ds *DbService, txn newrelic.Transaction) (err error)

func (*BoltService) PublishUpdate 

func (bs *BoltService) PublishUpdate() (err error)

func (*BoltService) ShowUpdates 

func (bs *BoltService) ShowUpdates()

func (*BoltService) StoreDbUpdates 

func (bs *BoltService) StoreDbUpdates()

func (*BoltService) UpdateFromCache 

func (bs *BoltService) UpdateFromCache()

type ContentService 

type ContentService interface {
	ConfigureNATS(host, port, channel string)
	ConnectNATS() (err error)
	CloseNATS()
	EnableLoadAll()
	GetGroup(id int64) (group Group, ok bool)
	GetImage(id int64) (image Image, ok bool)
	ListenForUpdates()
	LoadCacheUpdates() (err error)
	LoadFromDb(ds *DbService, txn newrelic.Transaction) (err error)
	PublishUpdate() (err error)
	ShowUpdates()
	StoreDbUpdates()
	UpdateFromCache()
}

func NewContentService 

func NewContentService(boltFile, boltBucket string) ContentService

type ContentSettings 

type ContentSettings struct {
	// contains filtered or unexported fields
}

type DbService 

type DbService struct {
	// contains filtered or unexported fields
}

func NewDbService 

func NewDbService(user, pass, name, host, port string) *DbService

func (*DbService) Close 

func (ds *DbService) Close()

func (*DbService) Connect 

func (ds *DbService) Connect() (err error)

func (*DbService) GetUser 

func (ds *DbService) GetUser(id int64, txn newrelic.Transaction) (user User, ok bool)

func (*DbService) Queryx 

func (ds *DbService) Queryx(query string, v ...interface{}) (rows *sqlx.Rows, err error)

type DbSettings 

type DbSettings struct {
	// contains filtered or unexported fields
}

type Group 

type Group struct {
	Id              int64           `json:"id" db:"id"`
	Name            string          `json:"name" db:"name"`
	SessionSeconds  *int64          `json:"sess_seconds" db:"sess_seconds"`
	ImagesGroupsMap ImagesGroupsMap `json:"-"`
}

type GroupMap 

type GroupMap map[int64]Group

type Image 

type Image struct {
	Id           int64   `json:"id" db:"id"`
	Width        int     `json:"width" db:"width"`
	Height       int     `json:"height" db:"height"`
	Url          string  `json:"url" db:"url"`
	Title        *string `json:"title" db:"title"`
	Artist       *string `json:"artist" db:"artist"`
	Gallery      *string `json:"gallery" db:"gallery"`
	Organization *string `json:"organization" db:"organization"`
}

Pointers to int/string to allow for 'null' value in JSON

type ImageMap 

type ImageMap map[int64]Image

type ImagesGroups 

type ImagesGroups struct {
	GroupId int64 `json:"group_id" db:"group_id"`
	ImageId int64 `json:"image_id" db:"image_id"`
}

type ImagesGroupsMap 

type ImagesGroupsMap map[int64]bool

type JwtPayload 

type JwtPayload struct {
	UserId int64  `json:"id"`
	Guid   string `json:"guid"`
}

type Level 

type Level uint
const (
	LogNONE Level = iota
	LogERROR
	LogWARN
	LogINFO
	LogDEBUG
)

type Logging 

type Logging struct {
	Error *log.Logger
	Warn  *log.Logger
	Info  *log.Logger
	Debug *log.Logger
}
var Log Logging

type QueueMessage 

type QueueMessage struct {
	Command string `json:"command"`
}

type ResponseMessage 

type ResponseMessage struct {
	Message string `json:"message"`
	Status  string `json:"status"`
}

type User 

type User struct {
	Id         int64  `json:"id" db:"id"`
	GroupId    int64  `json:"group_id" db:"group_id"`
	Guid       string `json:"guid" db:"guid"`
	FirstName  string `json:"first_name" db:"first_name"`
	MiddleInit string `json:"middle_init" db:"middle_init"`
	LastName   string `json:"last_name" db:"last_name"`
	Email      string `json:"email" db:"email"`
	Addr       string `json:"addr" db:"addr"`
	City       string `json:"city" db:"city"`
	State      string `json:"state" db:"state"`
	Zip        string `json:"zip" db:"zip"`
	Gender     string `json:"gender" db:"gender"`
	Status     bool   `json:"status" db:"status"`
}

func GetUserFromSession 

func GetUserFromSession(ds *DbService, r *http.Request) (user User, err error)

type UserMap 

type UserMap map[int64]User

type UserMessage 

type UserMessage struct {
	Command   string `json:"command"`
	Id        int64  `json:"id"`
	GroupId   int64  `json:"group_id"`
	Guid      string `json:"guid"`
	FirstName string `json:"first_name"`
	LastName  string `json:"last_name"`
}

Source Files

View all Source files

Directories

Path Synopsis
apps
api-server

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.