Werify: PoC with a cheesy name
Werify is a Go stdlib-only PoC attempt at distributed host checks.
Setup
Installation
With a correctly configured Go installation:
go get -u github.com/disq/werify/cmd/werifyctl
go get -u github.com/disq/werify/cmd/werifyd
This will install werifyctl
and werifyd
in your $GOPATH/bin
directory.
Build (Developers)
git clone https://github.com/disq/werify
cd werify
make
This will build werifyctl
and werifyd
.
Concepts
werifyd
is the server, listening on TCP/30035
.
- Each
werifyd
stores a list of servers (like itself) in memory.
werifyctl
is used to talk to one of the servers, usually the local one.
- In the course of the
werifyctl
run, the destination server becomes a coordinator node. Server list is edited using werifyctl
.
- One or more operations can be ran on the server list of the coordinator node.
werifyctl
is a dummy RPC client.
- Separate
werifyd
instances can have separate lists of servers -- but none other than the coordinator node needs to have the list.
- The coordinator node will hand out work (to run and/or forward to servers) in the same env tag, collect results, and pass them to the calling
werifyctl
.
Caveats
- If the coordinator node is one of the destination hosts, it should also be added to the server list (
./werifyctl add 127.0.0.1
)
- Each host is automatically identified by its first-referred
ip[:port]
pair. Adding a single host to multiple coordinators using different referral schemes (internal vs. external ip, forwarded port, etc.) is not supported.
Persistent Server List
Persistent server list is not implemented directly. But after launching werifyd
, werifyctl
can be used to populate the list using a commands-file:
cat examples/init.werifyd | ./werifyctl -
Invocation
werifyd is the server, and werifyctl is the client. Work is always done on the server.
Server
Usage of ./werifyd:
-env string
Env tag (default "dev")
-port int
Listen on port (default 30035)
-w int
Number of workers per operation (default runtime.NumCPU)
env
is the environment tag. It should match exactly on all werifyd
/werifyctl
instances and it is enforced on every RPC call.
- Number of workers (
-w
) applies to every worker-pool related event. The daemon utilizes multiple worker pools.
Client
Usage: ./werifyctl [OPTION]... [COMMAND [PARAMS...]]
Available options:
-connect string
Connect to werifyd (default "localhost:30035")
-env string
Env tag (default "dev")
-timeout duration
Connect timeout (default 10s)
Available commands:
add Adds a host to werifyd
del Removes a host from werifyd
list Lists hosts in werifyd
listactive Lists active hosts in werifyd
listinactive Lists inactive hosts in werifyd
operation Runs operations from file on werifyd
get Get status of operation with handle
refresh Start health check on all hosts
Commands can also be specified from stdin using "-".
connect
parameter can be set with the environment variable WERIFY_CONNECT
.
env
parameter can be set with the environment variable WERIFY_ENV
.
Host checks/operation file is a JSON file. The first-level object keys are user specified. There isn't any imposed limit on the number of checks.
{
"check_name": {
"type": "type of check",
"path": "value for the path parameter for check_name",
"check": "value for the check parameter for check_name"
},
"name_for_another_check": {
"type": "type of check",
"path": "value for the path parameter for name_for_another_check",
"check": "value for the check parameter for name_for_another_check"
},
...
}
In the response, each check will be referred to by its key name and the Host's first-referred identifier. (See Caveats)
A sample ops.json
file is provided in the examples directory.
Types of Host Checks
File exists
Checks if the file or directory exists on the host system.
Parameters:
type
: Should be set to file_exists
path
: Full path to the file to check
File contains
A type of line-by-line grep.
Parameters:
type
: Should be set to file_contains
path
: Full path to the file to check
check
: Text/contents to check
Running Process
Checks if the given process is running on the host system. Linux (/proc
filesystem) only.
Parameters:
type
: Should be set to process_running
path
: Full path to the process to check, ie. /bin/bash
check
: Basename of the process to check, ie. bash
At least one of path
or check
should be supplied.
Example Run
Launch your daemon: (or multiple daemons on multiple hosts)
./werifyd
Choose one of the hosts (well...) as the coordinator node. This node will have a list of all the other nodes. For the sake of brevity, let's choose the local one.
Add each ip address, like so:
./werifyctl -connect 127.0.0.1 add 10.42.0.3
./werifyctl -connect 127.0.0.1 add 10.42.0.4
./werifyctl -connect 127.0.0.1 add 127.0.0.1 # Let the coordinator node run the checks as the others
You can also pipe the commands through stdin:
cat examples/init.werifyd | ./werifyctl -
You can now check the list, using the werifyctl list
command:
Active hosts (3)
10.42.0.3:30035
10.42.0.4:30035
127.0.0.1:30035
Inactive hosts (0)
End of list
To run a host check operation, prepare an operations file. Then use the werifyctl operation
command:
./werifyctl operation examples/ops.json
The output will be:
Operation submitted. To check progress, run: ./werifyctl get asv1
(asv1
is a unique handle for the submitted operation)
You can then check the progress using werifyctl get
:
./werifyctl get asv1
Host:10.42.0.3:30035 Operation:check_virus_file_exists Success:false
Host:10.42.0.3:30035 Operation:check_etc_hosts_has_4488 Success:false
Host:10.42.0.4:30035 Operation:check_virus_file_exists Success:false
Host:10.42.0.4:30035 Operation:check_etc_hosts_has_4488 Success:false
Host:127.0.0.1:30035 Operation:check_virus_file_exists Success:false
Host:127.0.0.1:30035 Operation:check_etc_hosts_has_4488 Success:false
Operation ended, took 3.445244ms