MapleJuice - MP4
Please checkout to the branch failMapleJuice
Distributed File System - MP3
Group members:
- Ruochen Shen rs20
- Jun Luo junluo2
Setup
modify any code file and then run:
cd
to project root folder
./scripts/deploy_nodes.sh
we used systemctl so you will need root privalege to do it.
Command
In any of our vms, Use dcli
to see available commands:
exec "<command>"
- execute command on all servers
dump
- dump local host membership list
ls <sdfsfilename>
- list all machine addresses where this file is currently being stored
store
- list all files currently being stored at this machine
put <localfilename> <sdfsfilename>
- Insert or update a local file to the distributed file system
put <localdirname>
- Insert or update all local files in a directory
get <sdfsfilename> <localfilename>
- Get the file from the distributed file system, and store it to
delete <sdfsfilename>
- Delete a file from the distributed file system`
Distributed Node System - MP2
Group members:
- Ruochen Shen rs20
- Jun Luo junluo2
Setup
modify any code file and then run:
cd
to project root folder
./scripts/deploy_nodes.sh
Demo
- start a node as introducer: at any machine:
sudo systemctl start dnode
- start all other machine:
./scripts/start_all_nodes.sh
Monitor
- use
dcli --dump
to see memberlist of current node
- use
dcli "tail /apps/logs/node.log"
to see all logs from other machine
- use grep for above command for specific target
Leave
- to tell a node to leave. We login into that machine and type command
kill -2 <PID>
which sends a SIGINT
- the can be found by checking
systemctl status dnode
Docker
# build
docker build -f Dockerfile-dnode -t luojl/dnode_starter .
# run
docker run -d -v /apps/logs:/apps/logs -v /tmp:/tmp luojl/dnode_starter
Distributed Log Querier - MP1
Group members:
- Ruochen Shen rs20
- Jun Luo junluo2
Setup
This section is already done. Safely skip it.
If servers need to be updated: we will use the following method to deploy new servers
cd
to project root folder
- set user to be root for all host in ~/.ssh/config
- run
./scripts/deploy_servers.sh
if server.go is modified and need to be rebuilt, it will restart log_querier as a service
to start a service, we must have root privilege, so it is requires you to type root password every time,
We put our own pubkey in the root directory to make it possible, so the deploy_servers script can not be used by others.
Demo
Log files are under /usr/logs
and we should use the absolute path when grep
log files.
Usage:
$ log_client "grep <pattern> -c /usr/logs/<your log pattern>"
Example
$ log_client "grep -HcE '^[0-9]*[a-z]{5}' /usr/logs/vm*" |sort | awk -F '/' '{print $4}'
vm1.log:4102
vm2.log:4012
vm3.log:4154
vm4.log:4246
vm5.log:4130
vm6.log:4165
vm7.log:4083
vm8.log:4211
vm9.log:4069
vm10.log:4075
Start a server
Login a remote vm and type sudo systemctl start log_querier
Kill a server
We can kill a server by login a remote vm and type sudo systemctl stop log_querier
alternatively, we can find out the PID by sudo systemctl status log_querier
or ps -aux |grep log_querier/server
then sudo kill <PID>
Scripts
Scripts under scripts
folder
Build
[distributed_log_querier]$ ./scripts/build.sh
Build server and client, you will get two bin, server
and client
, under project root folder
Test
- To send logs to 10 vms,
/usr/logs/
, run
[distributed_log_querier]$ sh scripts/deploy_test_log.sh
- To run test,
[distributed_log_querier]$ go test -v ./.../test
the test function included 6 tests:
- grep Rare/Frequent/SomewhatFrequent word is efficient
- grep pattern that appears in only one file, some files, or all files, determine if result is expeted