mq-to-db (message queue to database)
This is a Golang (go) program to read a specific JSON Payload message from a Message Queue System and Store into Database using concurrency
This is a close image of how it works:
Consumers supported
Storage supported
Characteristics
- The number of queue consumers could be different from the numbers of storage workers, see config-sample.yaml
- The process (job) of consuming one message from the queue and store into the database is synchronous because every message needs to be acknowledged (confirmed as stored).
- Golang
pprof
enabled via --profile
command line when starting the service
- Prometheus metrics for consumers, storage workers, go statistics, and database
- Grafana dashboard for Prometheus.io metrics
- Dockerfile multi-stage build
- Makefile to facilitate the project builds
- docker-compose file and configuration to testing all elements
- docker images at docker-hub and Github Packages
- CI/CD Github Action pipeline workflow
How to execute
There are many ways to do it, but always is necessary PostgreSQL
and RabbitMQ
dependencies, the easy way to see how it works is using containers.
docker-compose
The program and all dependencies and visibility systems at once
Up
docker-compose up --build
Down
docker-compose down -v
Available links
After docker-compose start all the services, you have the following links ready to be curious, I prepared a simple grafana dashboard
to show you part of the prometheus metrics
implemented
Profiling
# terminal 1, for mq-to-db-01 inside the docker-compose-file
go tool pprof http://127.0.0.1:8080/debug/pprof/goroutine
# terminal 2, for mq-to-db-02 inside the docker-compose-file
go tool pprof http://127.0.0.1:8081/debug/pprof/goroutine
# once you are into tool pprof, execute the command web
(pprof) web
See the Logs
docker-compose logs mq-to-db-01
docker-compose logs mq-to-db-02
Manually
First install dependencies
RabbitMQ
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management
NOTES:
PostgreSQL
docker run --rm --name postgresql -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres
# logs
docker logs postgresql -f
# remember to stop and remove (--rm in docker run do it for you)
docker stop postgresql
Using source code
git clone https://github.com/christiangda/mq-to-db.git
cd mq-to-db/
go run -race ./cmd/mq-to-db/main.go --help
# and then
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yaml
NOTE: the parameter -race
is to check race conditions because we are using Go Concurrency
How to build
compiling for your ARCH and OS
# make executable first
make
# check the available options
./build/mq-to-db --help
# execute
./build/mq-to-db --configFile config-sample.yaml
Cross-compiling
make build-dist
# cross-compiling files
ls -l ./dist/
NOTES related to make
-
- This create a cross-compiling binaries and also Docker Image (linux 64bits)
-
- Check the Makefile to see
The make available targets options
-
- Remember to start dependencies first
Using docker image
Here I use latest tag
, but you can see all releases here
# pull the image first
docker pull christiangda/mq-to-db:latest
# see available option
docker run --rm --name mq-to-db christiangda/mq-to-db:latest - --help
# run with a config file mapped and with profile option
docker run --rm -v <path to config file>:/etc/mq-to-db/config.yaml --name mq-to-db christiangda/mq-to-db:latest - --profile
NOTES:
- Remember to start dependencies first
Available endpoints
The application expose different endpoints via http server
References
Internals
Externals
License
This module is released under the GNU General Public License Version 3: